Compare Regexp or wildcards with DynamicalLexicalPatternGroup Explicit Patterns - SyntaxEditor for WPF Forum

Posted 12 years ago by Christel

Version: 12.1.0561

Hi,

in the Lexer I defined a DynamicalLexicalPatternGroup with Explicit Patterns:

DynamicLexicalPatternGroup lexicalPatternGroup = new DynamicLexicalPatternGroup(DynamicLexicalPatternType.Explicit, "GopMenge", classificationTypeProvider.GOP);
lexicalPatternGroup.TokenId = MyTokenId.MySpecialToken;
lexicalPatternGroup.LookAheadPattern = "[,|+]|{LineTerminator}";
lexicalPatternGroup.Patterns.Add(new DynamicLexicalPattern("11301"));
lexicalPatternGroup.Patterns.Add(new DynamicLexicalPattern("22301"));
lexicalPatternGroup.Patterns.Add(new DynamicLexicalPattern("35301"));
lexicalPatternGroup.Patterns.Add(new DynamicLexicalPattern("35301A"));
lexicalPatternGroup.Patterns.Add(new DynamicLexicalPattern("35301B"));
lexicalState.LexicalPatternGroups.Add(lexicalPatternGroup);

Text in the editor:

"11301, 22301": all tokens are recognized perfectly

Requirement: Token should be recognized as well with using regexp or wildcard in the editor

Text in editor:

"..301", 35301*: both should be reognized as a valid "MySpecialToken"

Each '.' is a placeholder for a digit, and '*' is a placeholder for all letters after the number.

I use the LLParser Framework, with GrammarClass providing AST Nodes.

My question:

Where in this framework with parser, lexer, tokenreader, etc. can I implement a special "IsValid"-method

perhaps before or after the lexer, parser or tokenreader compare the strings with the explicit patterns and states a "does not fit"?

Looking forward to your ideas!

Comments (3)

Answer - Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Christel,

If you are using dynamic lexers, it all needs to be done via patterns. So you'd have to do a special regex pattern like "\.* [0-9]+ \*?" or whatever is valid for your scenario. Then perhaps in your token reader, if you come across a token of that type, you could validate it and return the MySpecialToken ID if it's valid. Otherwise, return something else like an error token or skip over it completely so the parser never sees it.

The alternative is to use a programmatic lexer (hand coded) instead. Since in that case you can do any logic you want to tokenize text. Then all of it would be handled in the lexer.

Actipro Software Support

Posted 12 years ago by Christel

Hi,

I decided to do it in my token reader.

I validate the token, and if it is not valid, I try to return an error token, so that the parser can recognize an error.

But I'm struggling how to creating an error token.

Either it can be a new MergableToken, with same Offset, Length... as the token, which is invalid.

Something like a copy but with another ID or classification type.

But calling new MergableToken(...)I don't know how to create a IMergableTokenLexerData.

Or what I would like more, is to change the classification type of the vaildated token, so that it gets just another colour (red for an error)

But I don't know how to do this, as I can only get the classification type of this token via the lexer, but I can't change it.

Or is there any better idea to show the parser, that this token needs either be "red-highlighted" or "red line squiggled"?

Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Christel,

Take a look at the Adornment Features / Squiggle Lines documentation topic. Basically if the IParseData that comes back from the parser implements IParseErrorProvider, via the IParseErrorProvider interface you can flag ranges that should get red line squiggles.

If you are using our LL(*) Parser Framework then the LLParseData it returns does implement that interface.

Actipro Software Support

The latest build of this product (v24.1.2) was released 2 days ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.