changing a highlightingstyle during semantic parsing

If you are using a dynamic language, then their default tokens (DynamicToken) have a CustomHighlightingStyle property. Make one special highlighting style instance for your underline variation and assigned that as needed to CustomHighlightingStyle. That way you don't modify the "real" highlighting style. Again, reuse the HighlightingStyle you create instead of making a new style for each token. This will save on memory and performance.

I still can't seem to get this working. I am using a dynamic language but the tokens that are being generated by the lexical parser aren't DynamicTokens. I do have my own RecursiveLexicalParser class that inherits from MergableRecursiveDescentLexicalParser but I can't figure out how to convert IToken to DynamicToken.

I also tried to cast the Token whose Style I am changing to a DynamicToken. In my custom semantic parser class that inherits from RecursiveDescentSemanticParser, I declared the following class member: In the class constructor: And at the point during the parsing stage where I want to change the Token's style, I use the following piece of code: Obviously I'm missing some basic concept that is keeping this from working correctly. Do you see anything obvious? How do I ensure that the lexical parser is generating DynamicTokens instead of ITokens?

Thanks again in advance.

Hi Hugh,

If you are using a plain dynamic language then they will be DynamicTokens. What type is your token (call GetType on it to see)? Did you write your own core programmatic lexical parser or did you override the token creating code in the language to make a custom token?

I called GetType() on my tokens and they are in fact DynamicTokens. However, in order to "see" the CustomHighlightingStyle property, I have to cast the token to a DynamicToken. That part of the code is in my previous reply. It is definitely not reflecting the customhighlighting style in the document. The style doesn't change at all.

One thing that may be causing this:
Using the SimpleRecursiveDescentLexicalParser class as an example, I created my own RecursiveLexicalParser class that is derived from MergableRecursiveDescentLexicalParser. I then overrode the GetNextTokenCore() method. That method, which I assume is automagically called by SyntaxEditor behind the scenes since it definitely IS being called, returns an IToken object. I'm overriding the method so that I can intercept certain tokens and not pass them along to the semantic parser. I think since this overridden method returns an IToken, then the token appears to be just a plain ol' IToken in my RecursiveDescentSemanticParser class. Is this why the CustomHighlightingStyle is not being reflected as the highlighting style in the document? Does the token need to be a 'pure' DynamicToken (in other words it doesn't need to be recast) in order for that property to be the property that determines the highlighting style? I don't see a way to intercept a DynamicToken in my SimpleRecursiveDescentLexicalParser derived class.

Thanks again for any help you can provide. I'm stumped.

I see, what is happening here is that you are getting a clone of the token in your semantic parser. Let me explain.

When the semantic parser starts, it starts at some arbitrary time after it was requested and it is running in a separate thread. Between request time and parse time (and even during parsing), the tokens in the Document might be changing. So what we do is take a snapshot of the document text right before parsing starts and create tokens upon request by the semantic parser. The tokens it gets are not the token references in the Document, they are clones.

Therefore if your custom style stuff is happening in the secondary thread, what you'd probably have to do is keep track of the text offset ranges for your style and pass those back to the main thread via your compilation unit. Then in the main thread (in DocumentSemanticParseDataUpdated, run through the appropriate tokens and update them.

The issue here is that the tokens might already be out of sync. So to recover, do you quit if the offsets/IDs don't line up? This is an issue we've struggled with because we'd like to do type highlighting in our C# add-on. We're open to any ideas for making this easier in future versions too.

If what you are saying is the case, wouldn't changing the Token's HighlightingStyle also result in no change to the styles in the document? If I change the token's HighlightingStyle in the semantic parser (instead of CustomHighlightingStyle), all the words' styles in the document that correspond to that Token ID get updated. Is the behavior for HighlightingStyle different than the behavior for CustomHighlightingStyle?

Thanks again.

No the HighlightingStyles and other language elements are shared amongst all the tokens so changing a style in one place will affect everywhere that uses that style.

Setting a CustomHighlightingStyle is setting a custom style for a specific token. However by setting that property in the multi-threaded scenario, you are not updating the actual Document's copy of the token because the one you are updating is really a cloned token that is created only for semantic parsing and then is disposed of.

Got it. I'll just handle it in DocumentSemanticParseDataChanged(...) like you suggested. I'll keep my mind open during my design to hopefully offer some suggestions on a 'better' way to handle something like this at the parser level.

Thanks again!