Need help with SyntaxLangauge.PerformLexicalParse

SyntaxEditor for Windows Forms Forum

Posted 14 years ago by Bob Puckett
I am working on a non-mergable SyntaxLangauge. I am using the sample code for PerformLexicalParse with some modifications. In PerformLexicalParse, when a space is added in the editor, I get called with the TextRange representing the space character. Example: if my text is


and I edit it to

Hello World

the invalid token for the space gets created and PerformLexicalParse gets called. If I parse just the one token, the preceding and following tokens are not necessarily correct. I assume that I actually need to reparse both the preceding and the following token as well, so I set the beginning offset to be the front of the preceding and the edning offset to be the end of the following and then I reparse resulting in 3 tokens. I call parseTarget.OnTokenParsed on each one.

OnTokenParsed returns false after the second token becuase it thinks I don't need to go any farther, but I ignore that becuase I am convinced that I do. Here is the problem - the tokens are not changed in the document. The token representing the space still has a type of Invalid even after the next edit. I have verified that it had the correct type when OnTokenParsed is called, but it never gets reflected in the document.
public override TextRange PerformLexicalParse(Document document, TextRange parseTextRange, ILexicalParseTarget parseTarget)
            // Get the offset at which to start and end parsing
            // TODO: Implement this differently by backing up a certain number of tokens or document lines
            //       for the start and moving forward a certain number of tokens or lines for the end

            int parseStartOffset = parseTextRange.StartOffset;
            int parseThroughOffset = parseTextRange.EndOffset;

            LimeToken t = document.Tokens.GetTokenAtOffset(parseStartOffset - 1) as LimeToken;
            parseStartOffset = t.StartOffset;
            t = document.Tokens.GetTokenAtOffset(parseThroughOffset + 1) as LimeToken;
            parseThroughOffset = t.EndOffset;

            // Get the ID of the lexical state at the parse start offset
            // TODO: Implement this differently by examining the existing token at the parse start offset
            int lexicalStateID = 0;

            // Initialize the modified start/end offsets
            int modifiedStartOffset = parseStartOffset;
            int modifiedEndOffset = parseThroughOffset;

            // Create a text buffer reader
            ITextBufferReader reader = new StringBuilderTextBufferReader(document.GetCoreTextBuffer(), 0, parseStartOffset);

            // Notify the parse target that parsing is starting
            // Loop and generate all the tokens... this is optimized for incremental parsing
            LimeToken token;
            while (!reader.IsAtEnd)
                // Get the next token
                token = myLexicalParser.GetNextToken(reader, lexicalStateID);
                lexicalStateID = token.LexicalStateID;

                // Update the parse target.  OnTokenParsed will return false when it thinks parsing should be done
                // but we want to go at least one token farther.

                parseTarget.OnTokenParsed(token, reader.Offset - token.StartOffset);
                if (token.StartOffset < modifiedStartOffset)
                    modifiedStartOffset = token.StartOffset;
                if (reader.Offset > modifiedEndOffset)
                    modifiedEndOffset = reader.Offset;

                // Quit the loop if we have already parsed all changes
                if (reader.Offset >= parseThroughOffset)

            // Notify the parse target that parsing is ending

            return new TextRange(modifiedStartOffset, modifiedEndOffset);
Please help me understand what is going on. Thanks.

Comments (2)

Posted 14 years ago by Bob Puckett
OK, I found my problem. Token.ParseDataEquals was not implemented correctly. I fixed my implementation and now it works fine. Thanks.
Posted 14 years ago by Actipro Software Support - Cleveland, OH, USA
Yes it's always very important for you to handle ParseDataEquals properly. Otherwise the lexer won't recognize where a token with a different meaning is found.

Actipro Software Support

The latest build of this product (v21.1.0) was released 18 days ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.