Dynamically Adding Token types at parse time - SyntaxEditor for Windows Forms Forum

Posted 7 years ago by David Conkey

Version: 13.1.0330

I have an application that uses the Windows Forms Actipro syntax editor for having the user write script in our own custom language. Our langauge is a compiled, scripting language which allows the creation of a hard-coded list of object types. We are currently adding in support for defining structures in the scripting language, which is essentially the same as supporting user-defined objects in script.

A sample of what the script looks like for this is below:

Struct MyStructure;

   Variable v1;

End;

MyStructure m;

m.v1 = 3;

In this, I want MyStructure to be identified as an IDENTIFIER token type on the "Struct MyStructure;" line, and an OBJECT_TYPE_IDENTIFIER on the "MyStructure m;" line.

My initial approach was to, when parsing the "Struct MyStructure" line, add "MyStructure" as a new LexicalPattern to the OBJECT_TYPE_IDENTIFIER LexicalPatternGroup. However, it seems like language updates need to be done on the main UI thread, and the overall affect is to the force a redraw / parse that then makes the MyStructure token on the "Struct MyStructure;" line a OBJECT_TYPE_IDENTIFIER (when I want it as a IDENTIFIER).

We are using a xml language definition, and so with that, is it possible to do what I want here, or do I need to wrtite a custom lexer? Also, can you point to any good samples / examples that I can reference?

Comments (2)

Posted 7 years ago by Tobias Lingemann - Software Devolpment Engineer, Vector Informatik GmbH

You have to distinguish between the lexer (lexical parser) and the semantic parser. The lexer cannot identify types. This is the job of the parser.

First off you will need to wirte a custom lexer with your own token class. Then the parser can run as a separate step and identify which identifier is a type declaration and which is actually a type usage. There you can set a property of your custom token class like IsType that is used to change the highlighting style.

You can take a look at this topic to get the idea. But be aware that the parsing process is relatively slow compared to the lexer, so you probably want to run it in the background thread. However this requires some additional synchronization and handling to refresh the highlighting styles.

Best regards, Tobias Lingemann.

Posted 7 years ago by Actipro Software Support - Cleveland, OH, USA

Hi David,

What Tobias said is spot on. That kind of thing is tricky to do, especially in the WinForms version. In the WPF version, you can leave the lexer to tokenize like normal but then could classify certain ranges as a special identifier (following an asynchronous parse) so they colorize differently, overridding the token-based coloring. But back to the WinForms version... yes all lexer changes would need to be made in the UI thread and you probably can't really modify things for a dynamic language in the middle of a lex, since any modifications to a lexer will likely invalidate the whole document again.

Actipro Software Support

The latest build of this product (v24.1.1) was released 7 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.