Changing token highlighting in semantic phase

SyntaxEditor for Windows Forms Forum

Posted 12 years ago by Paul Fuller
Version: 4.0.0239
Avatar
I am implementing a non-dynamic language based on the CSharpSyntaxLanguage example.

The lexical parser and semantic parser are both working pretty well. I'm developing the AST etc now.

One limitation that I have found is that the semantic parsing phase can't assign a custom HighlightingStyle to a token. The lexer may have determined that there is an identifier token and can even say that it is a keyword. It does not know the context however. It might be a valid keyword but in the wrong place. I'd like to flag this with a custom style but only the semantic parser will be able to determine the full context.

The documentation under "Assigning a Custom HighlightingStyle to a Specific Token" informs me that "This feature is for dynamic languages only".

Also I note that using the parsing service does not allow access to the tokens since it runs in a separate thread. I'm running in the main UI thread now but want to move it to the service later on.

I see that Adam Dickinson in his post "Advanced Syntax Highlighting Problem" seems to have had a similar problem and has a found a workaround with Solution #1 and your toggle ReadOnly trick.

Is there a better way to approach this now? If not would you please consider this as something to be enhanced. Pretty much any advanced language would want to possibly rework the tokens in the semantic phase.

Other than that I'm really enjoying the new V4 product ! So much more powerful to write and faster to execute than the dynamic approach. More of a learning curve but well worth it.

Comments (4)

Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Hi Paul,

Yes the reason it only works for dynamic languages right now is that their tokens store an extra reference for a custom highlighting style and they are also single-threaded.

We certainly would like to support this feature in a multi-threaded way as well but really aren't sure of the best way to go about it. Of course we're open to all suggestions for implementation.

When you look at Microsoft's implementation in VS, they have a slight delay before the types are highlighted, meaning that it must be done in a separate thread. And they are somehow knowing when a type's name is a type's name and not a regular identifier with the same text. Therefore they must be examining AST information and perhaps doing another pass to update tokens. If you'd like to brainstorm via e-mail on this, feel free to send over ideas.


Actipro Software Support

Posted 12 years ago by Paul Fuller
Avatar
I'll think about it and send something over if it has legs. I'm just getting into the AST part of things in order to support folding, drop down lists etc so it is early stages and I'm quite a newbie.

Do you handle this in your CSharp add-on ? The example you sent does not include the semantic parser so I don't know whether you have an approach. Your lexical parser does a bit more than the basics by looking ahead at tokens in some special cases. It is doing a bit of simple semantic style parsing in the lexer. This is tactical and works in limited cases but can't extend to general use IMHO.

Off the cuff, could the semantic parsing service be given a clone of the token stream. It applies CustomTokenId, CustomLexicalState and/or CustomHighlightingStyle to its copy.

When the service completes it raises an event against the syntax editor to scan through the current and service copies of the streams and performs any updates. If the tokens don't match (by Id, Start, Length) then it is because the source has been modified. The service tokens are thrown away and it is told to re-parse on a fresh clone of the updated token stream.

The service keeps going until the streams match and the changes are accepted.

Cloning the token stream would be the most significant process to occur in the main thread since it has to do the full stream rather than just the changed range that the lexer processes.

Applying changes again has to run the full stream but only a linear compare and update. Then whatever UI updates result.
Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Hi Paul,

That approach is somewhat similar to one of the two suggestions on this topic that we posted in this thread:
http://www.actiprosoftware.com/Support/Forums/ViewForumTopic.aspx?ForumTopicID=1905

We already have a second copy of the tokens for semantic parsing, so the trick is creating a new loop to go after semantic parsing is complete and update the tokens in the document. You can probably do this with the existing codebase since you do know the range that was lexically parsed. You just need a way to store the information and retrieve it later and update it in the main UI thread.


Actipro Software Support

Posted 12 years ago by Paul Fuller
Avatar
Ok, I'm working on some other bits right now. When that is settled I'll see if something along those lines is within my capability.

Perhaps somebody will come up with an approach that can migrate its way back into the core product. Always better if somebody takes care of the difficult bits.

Cheers.
The latest build of this product (v2018.1 build 0341) was released 6 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.