Getting token list for Semantic parser?

SyntaxEditor for Windows Forms Forum

Posted 12 years ago by Bob Puckett
Version: 4.0.0257
Avatar
I am writing a custom parser. The Lexical parser is passed a text range and determines the tokens and adds them to the token stream. The Semantic parser is passed a textReader and it appears that it is expected to create a lexical Parser and re-parse the text from the beginning. Is this correct or is the previously parsed token list available to the semantic parser through some property?

Thanks.

Comments (1)

Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Right now the lexer does create a secondary set of tokens for semantic parsing so that they don't interfere with the Document tokens. This is done only when using the semantic parser service since work is done in a secondary thread. However there is a difference.

The Document.Tokens are updated in the main thread and are all loaded upon lexical parsing completion. They stay in Document.Tokens. The semantic parser in a worker thread only calls the lexer one step at a time when reading a token and any old tokens that it has already read are discarded. So it dynamically parses tokens again using the lexer. When working in a separate thread you can't currently access the main Document.Tokens.


Actipro Software Support

The latest build of this product (v2018.1 build 0341) was released 10 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.