Confusion about Token ID - SyntaxEditor for Windows Forms Forum

Posted 19 years ago by Greg Shaffer - Boulder

Hi,

I use the Token.Key field a lot during my semantic parse, which means I do a lot of string compares. It appears that the ID field is meant to allow me do to numeric compares.

Do I need to add an "TokenID" xml attribute to my "...PatternGroup" tags in the language XML file?

In the Readme, it talks about the "LexicalPatternGroup.TokenID" being used to set the Token.ID. Does the ID field have to be unique for each PatternGroup within that state?

Thanks,
Greg

Comments (2)

Posted 19 years ago by Actipro Software Support - Cleveland, OH, USA

Yes we added numeric IDs in 3.0 so you can do faster compares than string IDs.

You can assign your numeric values in the TokenID attribute, however we did some really cool stuff to automate its setting. You can make a class that defines bunch of public constants named the same as your TokenKeys.

So like if you have a CommentToken and a ReservedWordToken, you could have a class like this:

public class TokenID {
public const int Comment = 1;
public const int ReservedWord = 2;
}

Then in the SyntaxLanguage's TokenIDTypeName attribute, you give a full type name (type and assembly) of that type. The XML language definition loading code will match the name of the TokenKey property of each pattern group with the name constant in the specified class, by using reflection. It's pretty slick stuff.

In the documentation, there was a topic added to the Parsing Guide section that describes a lot of this.

Also, if you use CodeSmith, we included a CodeSmith template that automatically loads an XML language definition file and generates the class of constants for you!

Actipro Software Support

Posted 19 years ago by Greg Shaffer - Boulder

Cool. Thanks!

I can't wait to use it!

-Greg

The latest build of this product (v24.1.1) was released 1 month ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.