Posted 12 years ago
by Bob Puckett
I am working on a non-mergable SyntaxLangauge. I am using the sample code for PerformLexicalParse with some modifications. In PerformLexicalParse, when a space is added in the editor, I get called with the TextRange representing the space character. Example: if my text is
HelloWorld
and I edit it to
Hello World
the invalid token for the space gets created and PerformLexicalParse gets called. If I parse just the one token, the preceding and following tokens are not necessarily correct. I assume that I actually need to reparse both the preceding and the following token as well, so I set the beginning offset to be the front of the preceding and the edning offset to be the end of the following and then I reparse resulting in 3 tokens. I call parseTarget.OnTokenParsed on each one.
OnTokenParsed returns false after the second token becuase it thinks I don't need to go any farther, but I ignore that becuase I am convinced that I do. Here is the problem - the tokens are not changed in the document. The token representing the space still has a type of Invalid even after the next edit. I have verified that it had the correct type when OnTokenParsed is called, but it never gets reflected in the document.Please help me understand what is going on. Thanks.
HelloWorld
and I edit it to
Hello World
the invalid token for the space gets created and PerformLexicalParse gets called. If I parse just the one token, the preceding and following tokens are not necessarily correct. I assume that I actually need to reparse both the preceding and the following token as well, so I set the beginning offset to be the front of the preceding and the edning offset to be the end of the following and then I reparse resulting in 3 tokens. I call parseTarget.OnTokenParsed on each one.
OnTokenParsed returns false after the second token becuase it thinks I don't need to go any farther, but I ignore that becuase I am convinced that I do. Here is the problem - the tokens are not changed in the document. The token representing the space still has a type of Invalid even after the next edit. I have verified that it had the correct type when OnTokenParsed is called, but it never gets reflected in the document.
public override TextRange PerformLexicalParse(Document document, TextRange parseTextRange, ILexicalParseTarget parseTarget)
{
// Get the offset at which to start and end parsing
// TODO: Implement this differently by backing up a certain number of tokens or document lines
// for the start and moving forward a certain number of tokens or lines for the end
int parseStartOffset = parseTextRange.StartOffset;
int parseThroughOffset = parseTextRange.EndOffset;
LimeToken t = document.Tokens.GetTokenAtOffset(parseStartOffset - 1) as LimeToken;
parseStartOffset = t.StartOffset;
t = document.Tokens.GetTokenAtOffset(parseThroughOffset + 1) as LimeToken;
parseThroughOffset = t.EndOffset;
// Get the ID of the lexical state at the parse start offset
// TODO: Implement this differently by examining the existing token at the parse start offset
int lexicalStateID = 0;
// Initialize the modified start/end offsets
int modifiedStartOffset = parseStartOffset;
int modifiedEndOffset = parseThroughOffset;
// Create a text buffer reader
ITextBufferReader reader = new StringBuilderTextBufferReader(document.GetCoreTextBuffer(), 0, parseStartOffset);
// Notify the parse target that parsing is starting
parseTarget.OnPreParse(parseStartOffset);
// Loop and generate all the tokens... this is optimized for incremental parsing
LimeToken token;
while (!reader.IsAtEnd)
{
// Get the next token
token = myLexicalParser.GetNextToken(reader, lexicalStateID);
lexicalStateID = token.LexicalStateID;
// Update the parse target. OnTokenParsed will return false when it thinks parsing should be done
// but we want to go at least one token farther.
parseTarget.OnTokenParsed(token, reader.Offset - token.StartOffset);
if (token.StartOffset < modifiedStartOffset)
modifiedStartOffset = token.StartOffset;
if (reader.Offset > modifiedEndOffset)
modifiedEndOffset = reader.Offset;
// Quit the loop if we have already parsed all changes
if (reader.Offset >= parseThroughOffset)
break;
}
// Notify the parse target that parsing is ending
parseTarget.OnPostParse();
return new TextRange(modifiedStartOffset, modifiedEndOffset);
}
