Change to Programmatic Mergable Lexer - need Grammar anyway?

SyntaxEditor for WPF Forum

Posted 8 years ago by Christel
Version: 13.1.0580


I first use a dynamic lexer and the LLparser framework.
I created a complex Grammar class and the TokenReader, created by the parser, has some additional functions or some special tokens, given by the dynamic lexer.

But due to some performance issues, I decided now to change to a programmatic mergable lexer.

That works fine,I get ExctMatches and NoMatches and some valid tokens.

Everything perfect as long I use simple grammar definitions with Terminals.As soon as a NonTerminal is recognized, I get some syntax errors, that have not been there using the dynamic lexer. It seems, as if the special GrammarClass or better the LLParser still recognize on base of the Grammar a wrong token and error occurs, that I can't debug and can't understand.

When I switched to the programmatic lexer I hoped to be able to neglect the complex grammar class in the future and do all the verifying of the tokens in the programmatic lexer and TokenReader of the syntaxLanguage.

My question is, can I use my own customer programmatic lexer together with a normal Parser, inheriting from ParserBase, instead of the LLParser?
Could I get syntax highlighting, error messages, red squiggle lines anyway, although using a very simple parser implementation without Grammar?

I realized that I don't really need an AST tree. Therefore I would like to keep the rest as simple as possible

The validation of the tokens now in the TokenReader of the LL framework, created by the LLParser, where can it be done - in the Lexer or in the Parser class inheriting vom ParserBase class? Or can I use the tokenReader together wîth the ParserBase class?

Intellisense, Completion list, Error list, can all that be used without using a LlParser with a grammar class?

Thnaks for answers!



Comments (1)

Answer - Posted 8 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Christel,

As long as the lexer output is the same and you've updated your token reader to use the new lexer, the grammar-based parser should give you the same results as before.  You could always use the LL Parser Debugger to help sort out what tokens are coming through.

But to answer your question, yes, everything in SyntaxEditor is interfaces and pluggable.  So by implementing IParser you can inject any custom parser.  Parsers can return anything in their IParseData result. If your IParseData implements IParseErrorProvider then syntax errors will be supported.  And if you register a ParseErrorTagger provider service on your language, you'll see the squiggle lines.  All of this can be done without using our grammar parser.

For validation of tokens, that is a parser job.  Lexers just tokenize text and that's it.

Yes, IntelliPrompt is independent of parsing.  You typically examine tokens using a snapshot reader and combine that info with the IParseData that is stored in document.ParseData to determine what to populate IntelliPrompt with.

Actipro Software Support

The latest build of this product (v2020.1.1) was released 11 days ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.