last char in line is a quote

SyntaxEditor for WPF Forum

The latest build of this product (v2020.1.1) was released 2 months ago, which was before this thread was created.
Posted 2 months ago by Martin de Jong - Blaise - Statistics Netherlands
Version: 20.1.1
Avatar

There is a little hickup. When i remove the last character on a line and this a quote or apostrophe, the next lines are not lexed. I use a Mergable Lexer implemention. If i add an 'normal' character, the next lines are lexed. Some thoughts on this?

Greetz Martin

Version 19.1

[Modified 2 months ago]

Comments (5)

Posted 2 months ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hi Martin,

It's tough to say without looking at and debugging your language definition, since we don't know how you defined the lexical states and patterns.  We'd need a lot more information to help.  But if they are defined in a way to handle missing end quotes, everything should work.


Actipro Software Support

Posted 2 months ago by Martin de Jong - Blaise - Statistics Netherlands
Avatar

We use a Antlr lexer

Adding a "or a ' do lex the lines after, but deletion of the last non-space character does not (if quote).

    Survey ENG "Survey" : STRING[30] '
    tMainSurveyID: STRING[36]  //GUID of survey. Could be a WAVE
    Survey ENG "Survey" : STRING[30] 'sssss
    tMainSurveyID: STRING[36]  //GUID of survey. Could be a WAVE

So the the deletion of the ' in the second snippet works.

I can solve it with the following code in the mergablelexer:

        protected override TextSnapshotRange GetIncrementalParseRange(TextSnapshotRange snapshotRange)
        {
            TextSnapshotRange? result = null;
            if (snapshotRange.StartPosition.DisplayLine+1 == snapshotRange.EndPosition.DisplayLine &&
                snapshotRange.StartPosition.Character==0 && snapshotRange.EndPosition.Character == 0)
            {
                int line = Math.Min(snapshotRange.Snapshot.Lines.Count - 1, snapshotRange.StartPosition.Line + 2);
                result = new TextSnapshotRange(snapshotRange.Snapshot, new TextRange(snapshotRange.Snapshot.Lines[snapshotRange.StartPosition.Line].StartOffset, snapshotRange.Snapshot.Lines[line].StartOffset));
                System.Diagnostics.Debug.WriteLine($"GetIncrementalParseRange: {result.Value.StartPosition} {result.Value.EndPosition}");
            }
            if (!result.HasValue)
                result = base.GetIncrementalParseRange(snapshotRange);

            return result.Value;
        }

But it seems to me the default method has a little hickup here:)

Greetz Martin

Posted 2 months ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hi Martin,

Let me provide some more information.  GetIncrementalParseRange will return the snapshot range of affected lines, including their line terminators (\n).  When you have a "'" character on your line (is that a single-line comment start in your language?), your lexer might be marking that "'" character's offset through the end of the line terminator as a comment token.  Thus when you delete the "'" character, it's reverting all of those characters (including the line terminator) back to other tokens.  Whereas deleting another non-"-" character wouldn't affect the tokenization of the line terminator.

The lexer only tries to lex as far as it needs to and I believe it stores incremental context information at line ends.  So if the line end's token changes, that would trigger the next line to lex as well.  That all is by design.  

Do you think that's what's going on here?


Actipro Software Support

Posted 2 months ago by Martin de Jong - Blaise - Statistics Netherlands
Avatar

Both ' and " are string delimiters. In our languages (called Blaise and Manipula) strings are multi-line and therefore can contain line terminators. So removing a ' or " will be influencing both lines back as next lines.  So we have tokens across lines. 

Changing the snapshotrange is a bit overdone probably, but i dont see another way yet. Maybe added a property HasMultiLineTokens to MergableLexerBase and make the GetIncrementalParseRange react on that;) My solution is a little bit brute force. Anyway, no show stopper.

Posted 2 months ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hi Martin,

Our SQL syntax language that comes with SyntaxEditor also has single ' character delimiters for multiline strings.  When I test a similar scenario there with deleting a single quote at the end of a line, it seems to refresh everything fine.  That's why I wonder if something in your lexer definition might need to be tweaked instead.


Actipro Software Support

Add Comment

Please log in to a validated account to post comments.