Trouble implementing PerformLexicalParse - SyntaxEditor for Windows Forms Forum

Posted 17 years ago by Sten Linnarsson

Version: 4.0.0249

I'm trying to implement a custom language inheriting from SyntaxLanguage. I've got lexical parsing and syntax coloring almost working, with the following glitch:

In PerformLexicalParse(), I correctly parse all the tokens in the document (as verified by printing all tokens to the console) and call

parseTarget.OnTokenParsed(token, token.Length)

on them. However, when I get into semantical parsing, the list of tokens is missing the last token and instead has one InvalidToken and one EndOfStream token.

Inspecting PerformLexicalParse, I noticed that parseTarget contains these two tokens already when the PerformLexicalParse() method is entered, with InvalidToken spanning the whole document. As tokens are added by OnTokenParsed, the TextRange of InvalidToken shrinks until it is just one character long (1481-1482). Then when the last token (a single-character curly brace, with StartOffset=1481 and Length=1) is sent to OnTokenParsed, it seems to be simpy thrown away.

Obviously I'm missing something, but I can't figure it out. Any suggestions?

btw, thanks for an excellent piece of software!

Comments (5)

Posted 17 years ago by Kelly Leahy - Software Architect, Milliman

By the way, this is the same problem I was having with my attempt at implementing a non-mergeable parser.

Kelly Leahy Software Architect Milliman, USA

Posted 17 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Sten,

Would you be able to throw together a tiny sample project that parses a couple characters and shows this happening? I don't believe this is happening in the languages we wrote so I'd like to see what the difference is by debugging it. Thanks!

Actipro Software Support

Posted 17 years ago by Sten Linnarsson

I've given up trying to implement SyntaxLanguage directly, and instead I'm inheriting MergableSyntaxLanguage. This works, and I now have syntax coloring and even semantic parsing up and running. No outlining yet, but I haven't digged into it.

/Sten

Posted 17 years ago by Actipro Software Support - Cleveland, OH, USA

Ok, or if Kelly you could email over a sample that would help. Sten, glad you're making good progress.

Actipro Software Support

Posted 17 years ago by Actipro Software Support - Cleveland, OH, USA

Thanks for the sample Kelly. Found the problem... when using a mergable token, this is taken care of for you. However in a non-mergable situation, you are tracking parse data (token ID and lexical state) on your own. Therefore you need to override the ParseDataEquals method in your custom token class.

That method should return whether the current token is equal in terms of lexical state/ID to another token. In your sample's case, we were always returning true from the base TokenBase implementation.

What I'm doing for the next maintenance release is changing TokenBase.ParseDataEquals to be abstract so that will force you to implement it and it won't be one of those things that we can forget about.

Actipro Software Support

The latest build of this product (v24.1.0) was released 2 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.