How to get the context in the GetNextToken method ?

SyntaxEditor for WPF Forum

Posted 7 years ago by Peter Luo - AIMNEXT
Version: 12.1.0560
Avatar

Hi,

In the override GetNextToken method of  MergableLexerBase class, how to get the previous tokens or context ?

I'm writting a C language lexer which Inherits the MergableLexerBase, in the GetNextToken method, after read the '<', I want to determine whether it is the start of path-specific(#include <...>) or a operator, and give the different token ids.(Also between #include "..." and string literals)

I think after scan the preprocessor keywords, I should do something, but how ?

Thanks a lot!

Comments (3)

Answer - Posted 7 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hi Peter,

You have access to a ITextBufferReader, which is a low-level plain text reader.  You won't have access to previous tokens since the lexers are creating the tokens as the GetNextToken method is called.  For pre-processor directives, you might want to have them transition to a new lexical state.  That way when you get to your '<' you'll know you're in a pre-processor directive.  Then exit the state on the '\n' character.

Alternatively, you can use the ITextBufferReader to scan backwards through the raw text characters and do your determination with that.  Just make sure you return its Offset to where it should be following the token creation.


Actipro Software Support

Posted 7 years ago by Peter Luo - AIMNEXT
Avatar

Hi,

Thank you for that.

I defined two lexical states, and it works fine, but another question:

Because the preprocessor directives can be used in everywhere(just like comment), so I skipped them in TokenReader class, but also skipped the parser, if I want to parse them in grammar class, how to do ? Two SyntaxLanguage classes and two Grammar classes with CreateChangeBatch calling ?

 

Thanks again!

Posted 7 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hi Peter,

That can be a tricky scenario.  You may want to make a custom IParser that does two rounds of parsing.  The first would just look for pre-processor directives and store info for those.  The second would be more like your normal one now.  Then merge the IParseData results together into a new IParseData and return that from IParser.Parse.


Actipro Software Support

The latest build of this product (v2019.1 build 0683) was released 1 month ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.