Parsing with a Non-Mergable Programmatic Lexer

Hi John,

We usually just make the token reader and pass in the reader that was passed and the lexer instance.  Like this:

public override ITokenReader CreateTokenReader(ITextBufferReader reader) {
	return new CSharpTokenReader(reader, new CSharpLexer());
}

For a NonMergableTokenReader implementation, you'd probably want to inherit TokenReaderBase and override the remaining members needed to call some other public method on your lexer instance.  So that you call this other method instead of the ILexer.Parse one you have.  Perhaps something like a GetNextToken method, etc.  You'd make the lexer.GetNextToken method call from the token reader's GetNextToken method override.

ok, this makes sense. Thank you.

My tokens inherit from TokenBase as follows:

    internal class TokenString : TokenBase
    {
        public TokenString(int startOffset, int length, TextPosition startPosition, TextPosition endPosition)
            : base(startOffset, length, startPosition, endPosition)
        {
        }

        public override int Id
        {
            get { return (int)SimpleTokenId.String; }
        }
    }

 The startOffset and length can be pulled from the reader and calculated respectively. How do I provide the TextPosition values. Should I calculate the line and character myself?

I think the ITextBufferReader has a Position property, so it tracks it for you.  You'd need to track the TextPosition before you read a token (for token start position), then use reader.Position for the token end position.