Need help with SyntaxLangauge.PerformLexicalParse

SyntaxEditor for Windows Forms Forum

Posted 12 years ago by Bob Puckett
I am working on a non-mergable SyntaxLangauge. I am using the sample code for PerformLexicalParse with some modifications. In PerformLexicalParse, when a space is added in the editor, I get called with the TextRange representing the space character. Example: if my text is


and I edit it to

Hello World

the invalid token for the space gets created and PerformLexicalParse gets called. If I parse just the one token, the preceding and following tokens are not necessarily correct. I assume that I actually need to reparse both the preceding and the following token as well, so I set the beginning offset to be the front of the preceding and the edning offset to be the end of the following and then I reparse resulting in 3 tokens. I call parseTarget.OnTokenParsed on each one.

OnTokenParsed returns false after the second token becuase it thinks I don't need to go any farther, but I ignore that becuase I am convinced that I do. Here is the problem - the tokens are not changed in the document. The token representing the space still has a type of Invalid even after the next edit. I have verified that it had the correct type when OnTokenParsed is called, but it never gets reflected in the document.
public override TextRange PerformLexicalParse(Document document, TextRange parseTextRange, ILexicalParseTarget parseTarget)
            // Get the offset at which to start and end parsing
            // TODO: Implement this differently by backing up a certain number of tokens or document lines
            //       for the start and moving forward a certain number of tokens or lines for the end

            int parseStartOffset = parseTextRange.StartOffset;
            int parseThroughOffset = parseTextRange.EndOffset;

            LimeToken t = document.Tokens.GetTokenAtOffset(parseStartOffset - 1) as LimeToken;
            parseStartOffset = t.StartOffset;
            t = document.Tokens.GetTokenAtOffset(parseThroughOffset + 1) as LimeToken;
            parseThroughOffset = t.EndOffset;

            // Get the ID of the lexical state at the parse start offset
            // TODO: Implement this differently by examining the existing token at the parse start offset
            int lexicalStateID = 0;

            // Initialize the modified start/end offsets
            int modifiedStartOffset = parseStartOffset;
            int modifiedEndOffset = parseThroughOffset;

            // Create a text buffer reader
            ITextBufferReader reader = new StringBuilderTextBufferReader(document.GetCoreTextBuffer(), 0, parseStartOffset);

            // Notify the parse target that parsing is starting
            // Loop and generate all the tokens... this is optimized for incremental parsing
            LimeToken token;
            while (!reader.IsAtEnd)
                // Get the next token
                token = myLexicalParser.GetNextToken(reader, lexicalStateID);
                lexicalStateID = token.LexicalStateID;

                // Update the parse target.  OnTokenParsed will return false when it thinks parsing should be done
                // but we want to go at least one token farther.

                parseTarget.OnTokenParsed(token, reader.Offset - token.StartOffset);
                if (token.StartOffset < modifiedStartOffset)
                    modifiedStartOffset = token.StartOffset;
                if (reader.Offset > modifiedEndOffset)
                    modifiedEndOffset = reader.Offset;

                // Quit the loop if we have already parsed all changes
                if (reader.Offset >= parseThroughOffset)

            // Notify the parse target that parsing is ending

            return new TextRange(modifiedStartOffset, modifiedEndOffset);
Please help me understand what is going on. Thanks.

Comments (2)

Posted 12 years ago by Bob Puckett
OK, I found my problem. Token.ParseDataEquals was not implemented correctly. I fixed my implementation and now it works fine. Thanks.
Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA
Yes it's always very important for you to handle ParseDataEquals properly. Otherwise the lexer won't recognize where a token with a different meaning is found.

Actipro Software Support

The latest build of this product (v2018.1 build 0341) was released 3 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.