SematicParseData is removed after each semantic parse - SyntaxEditor for Windows Forms Forum

Posted 18 years ago by Marianne

Version: 4.0.0236

If you assign some data to the SemanticParseData property of a DynamicToken object, it is removed sometime before the next parse takes place. To repro in the sample app, do the following:

-In the DocumentTextChanged event of the SDI sample app add the following code:

            TextStream ts = e.Document.GetTextStream(
                Math.Max(e.Modification.InsertionEndOffset - 1, 0));
            DynamicToken token = ts.Token as DynamicToken;

            if (token == null) return;
            if (token.Key == "CommentWordToken")
            {
                string parseData = token.SemanticParseData == null ?
                    "" : token.SemanticParseData.ToString();
                this.WriteLine("Comment Changed, Parse Data: " + 
                    parseData);
                token.SemanticParseData = "my parse data";
            }

-Launch the SDI sample app and select the c# dynamic language
-Go to the first comment of the sample and modify a comment word
-Modify the same comment word again, you should see the parse data "my parse data" in the output but it is null every time.

If getting/setting SemanticParseData works differently in 4.0 then just let me know how I can set and retrieve data from dynamic tokens via the SemanticParseData property. Thanks.

------------------------------- Marianne

Comments (1)

Posted 18 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Marianne,

This is a side effect of design changes for v4.0. One of the major new features of v4.0 is that any custom type of token can be used. This is ideal for situations where you want to make your own token that perhaps uses minimal memory for the information you need to store. Therefore, any token implements the new IToken interface, whereas in v3.1 every token was simply a Token class.

So the change comes into play here. Now that it's up to the language to create the token objects, in v4.0 they are replacing the old tokens.

Take this scenario:
* You set semantic parse data in a token on parsing pass #1
* A document modification occurs that covers the range of that token
* The token is now moved to the DeletedTokens collection (if the related Document property for tracking that is enabled), and a new token created by the language is moved in its place

The replace has to occur because the token types are unknown and may not be compatible.

Other notes, the semantic parse data should still be there in the TextChanging event since that occurs before the lexical parsing phase.

I hope that explains it correctly. It came down to a trade-off decision as far as implementing tokens goes. However with the v4.0 style, it really enables you to create tokens that use minimal memory if desired.

Actipro Software Support

The latest build of this product (v25.1.0) was released 12 days ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.