last char in line is a quote

Hi Martin,

It's tough to say without looking at and debugging your language definition, since we don't know how you defined the lexical states and patterns.  We'd need a lot more information to help.  But if they are defined in a way to handle missing end quotes, everything should work.

We use a Antlr lexer

Adding a "or a ' do lex the lines after, but deletion of the last non-space character does not (if quote).

    Survey ENG "Survey" : STRING[30] '
    tMainSurveyID: STRING[36]  //GUID of survey. Could be a WAVE

    Survey ENG "Survey" : STRING[30] 'sssss
    tMainSurveyID: STRING[36]  //GUID of survey. Could be a WAVE

So the the deletion of the ' in the second snippet works.

I can solve it with the following code in the mergablelexer:

        protected override TextSnapshotRange GetIncrementalParseRange(TextSnapshotRange snapshotRange)
        {
            TextSnapshotRange? result = null;
            if (snapshotRange.StartPosition.DisplayLine+1 == snapshotRange.EndPosition.DisplayLine &&
                snapshotRange.StartPosition.Character==0 && snapshotRange.EndPosition.Character == 0)
            {
                int line = Math.Min(snapshotRange.Snapshot.Lines.Count - 1, snapshotRange.StartPosition.Line + 2);
                result = new TextSnapshotRange(snapshotRange.Snapshot, new TextRange(snapshotRange.Snapshot.Lines[snapshotRange.StartPosition.Line].StartOffset, snapshotRange.Snapshot.Lines[line].StartOffset));
                System.Diagnostics.Debug.WriteLine($"GetIncrementalParseRange: {result.Value.StartPosition} {result.Value.EndPosition}");
            }
            if (!result.HasValue)
                result = base.GetIncrementalParseRange(snapshotRange);

            return result.Value;
        }

But it seems to me the default method has a little hickup here:)

Greetz Martin

Hi Martin,

Let me provide some more information.  GetIncrementalParseRange will return the snapshot range of affected lines, including their line terminators (\n).  When you have a "'" character on your line (is that a single-line comment start in your language?), your lexer might be marking that "'" character's offset through the end of the line terminator as a comment token.  Thus when you delete the "'" character, it's reverting all of those characters (including the line terminator) back to other tokens.  Whereas deleting another non-"-" character wouldn't affect the tokenization of the line terminator.

The lexer only tries to lex as far as it needs to and I believe it stores incremental context information at line ends.  So if the line end's token changes, that would trigger the next line to lex as well.  That all is by design.  

Do you think that's what's going on here?

Both ' and " are string delimiters. In our languages (called Blaise and Manipula) strings are multi-line and therefore can contain line terminators. So removing a ' or " will be influencing both lines back as next lines.  So we have tokens across lines. 

Changing the snapshotrange is a bit overdone probably, but i dont see another way yet. Maybe added a property HasMultiLineTokens to MergableLexerBase and make the GetIncrementalParseRange react on that;) My solution is a little bit brute force. Anyway, no show stopper.

Hi Martin,

Our SQL syntax language that comes with SyntaxEditor also has single ' character delimiters for multiline strings.  When I test a similar scenario there with deleting a single quote at the end of a line, it seems to refresh everything fine.  That's why I wonder if something in your lexer definition might need to be tweaked instead.