Lexer and SyntaxLanguage (nonmergeable) very close... but.

SyntaxEditor for Windows Forms Forum

Posted 17 years ago by Kelly Leahy - Software Architect, Milliman
Version: 4.0.0257
Avatar
I'm very close to having my generic lexer/filter/parser/AST combination working that will be used for my syntax-language derived implementation of my language (and my command line compiler).

However, I'm puzzled about where the DocumentEnd token should come from. Is this normally to be generated by my lexer / filter, or is this somehow "magically" generated by the syntax editor? Is there a harm in me returning a DocumentEnd token at the end of my document, should the lexer's GetNextToken be called past the last token in the string? I seem to remember that your example implementation didn't return a DocumentEnd token, if I remember correctly.

The thing is, I'd like to use my lexer/filter/parser inside and outside of the syntax editor, and in order to do so, I need to return a documentEnd token, or I need to NOT rely on it in my parser. Of course, if I don't rely on it in my parser, it's hard to ensure that there isn't the "subset" problem where my parser will recognize (as valid) a subprogram without noticing that there is invalid text after the end of the program text. Some languages consider this acceptable (Pascal comes to mind), but in my case it would be wholly unacceptable.

Thanks,

Kelly Leahy Software Architect Milliman, USA

Comments (2)

Posted 17 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Hi Kelly,

The lexer should call your language's CreateDocumentEndToken method when the token after the last is retrieved. That's generally where you return a token with an ID of DocumentEnd. The DocumentEnd can really be any ID value so long as it is identifiable to you.

The Simple language defines a DocumentEnd token ID and overrides CreateDocumentEndToken in SimpleSyntaxLanguage to return a token with that ID.


Actipro Software Support

Posted 17 years ago by Kelly Leahy - Software Architect, Milliman
Avatar
Wow! Thanks for the late reply...

I've changed my lexer to return my end of document token when reader.IsAtEnd is true. That seems to be doing the trick for my code, we'll see what happens when I hook my code up to the syntax editor and throw some tests together there (I'm still building my unit tests for the "outside" case right now).

I'll let you know if I have any other problems. Thanks again for the "after hours" reply - you really have great support here - I hope your customers all know it!

Thanks,

Kelly Leahy Software Architect Milliman, USA

The latest build of this product (v24.1.0) was released 1 month ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.