Parsing with a Non-Mergable Programmatic Lexer

SyntaxEditor for WPF Forum

Posted 7 years ago by John Youren
Version: 12.2.0570
Avatar

I have a lexer class like so

public class Lexer : ILexer {

...

public TextRange Parse(TextSnapshotRange snapshotRange, ILexerTarget parseTarget) { ... }

...

}

 which works correctly, syntax highlighting works fine.

I also have a token reader class for parsing.

public sealed class SimpleTokenReader : TokenReaderBase {

...

public SimpleTokenReader(ITextBufferReader reader) : base(reader) { ... }

...

}

 

My question is, how can I construct a TextSnapshotRange and ILexerTarget from the variables I have to pass into the Parse method? Alternatively, what do I need to do to be able to populate my TokenBase classes with the correct data (so I can draw red squiggly lines under syntax errors etc).

 

Thanks

Comments (3)

Answer - Posted 7 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hi John,

We usually just make the token reader and pass in the reader that was passed and the lexer instance.  Like this:

public override ITokenReader CreateTokenReader(ITextBufferReader reader) {
	return new CSharpTokenReader(reader, new CSharpLexer());
}

For a NonMergableTokenReader implementation, you'd probably want to inherit TokenReaderBase and override the remaining members needed to call some other public method on your lexer instance.  So that you call this other method instead of the ILexer.Parse one you have.  Perhaps something like a GetNextToken method, etc.  You'd make the lexer.GetNextToken method call from the token reader's GetNextToken method override.


Actipro Software Support

Posted 7 years ago by John Youren
Avatar

ok, this makes sense. Thank you.

 

My tokens inherit from TokenBase as follows:

    internal class TokenString : TokenBase
    {
        public TokenString(int startOffset, int length, TextPosition startPosition, TextPosition endPosition)
            : base(startOffset, length, startPosition, endPosition)
        {
        }

        public override int Id
        {
            get { return (int)SimpleTokenId.String; }
        }
    }

 The startOffset and length can be pulled from the reader and calculated respectively. How do I provide the TextPosition values. Should I calculate the line and character myself?

Posted 7 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar

I think the ITextBufferReader has a Position property, so it tracks it for you.  You'd need to track the TextPosition before you read a token (for token start position), then use reader.Position for the token end position.


Actipro Software Support

The latest build of this product (v2019.1 build 0682) was released 27 days ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.