At the risk of answering my own question, I think I got it. Because I needed to "bridge" the dynamic lexer to a lexer that derives from MergableRecursiveDescentParser, I needed to implement GetNextTokenCore.
In my previous implementation (copied from http://www.actiprosoftware.com/support/forums/viewforumtopic.aspx?forumtopicid=1940), the next token from the manager was always returned.
I modified the code to check the .IsWhiteSpace property, and to consume that token as such:
protected override IToken GetNextTokenCore()
{
int startOffset = this.TextBufferReader.Offset;
while (!this.IsAtEnd)
{
//consume whitespace tokens
IToken token = this.Manager.GetNextToken();
if (!token.IsWhitespace)
return token;
}
// Return an end of document token
if (this.Token != null)
return this.Language.CreateDocumentEndToken(startOffset, this.Token.LexicalState);
else
return this.Language.CreateDocumentEndToken(startOffset, this.Language.DefaultLexicalState);
}
