How to get the collection of tokens in a document

Posted 12 years ago by Nassim Farhat

Version: 11.2.0551

Hello,

Back in the days with the old WinForm SyntaxEditor we could enumerate through the document's token collection and retrieve all the tokens that we wanted along with their textrange, the pattern group they belonged to etc....
The command used to be the following:

syntaxEditor.Document.Tokens

Q1-
I'm looking to obtain the same thing using the WPF SyntaxEditor but am unable to. Can you point to me the new namespace where to obtain the active document tokens or/else if the system changed, can you please explain me how to go about it?

Q2-
Also, once we get our tokens, how can i obtain their text property? I can't find any method in the IToken object that returns the text of the Token?

Regards
Nassim

P.S. I have not create my own AST or Lexer/Parser.I'm using the Dynamic Language definition file to define my language.

[Modified at 12/22/2011 10:21 AM]

Comments (3)

Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Nassim,

In WPF, we have immutable snapshots of a document where there is one snapshot for each edit of the document. It allows you to examine the snapshot without the fear of the snapshot changing, even if the user is editing the document while you examine it.

So you can do something like:
editor.Document.CurrentSnapshot.GetReader(0);

That will give you an ITextSnapshotReader, which you can use to iterate over tokens.

I believe it also has a TokenText property that tells you the text of the current token. Or you can call snapshot.GetSubstring over the token's range to get the text for other tokens.

Actipro Software Support

Posted 12 years ago by Nassim Farhat

thank you for your reply... I actually found out how to do it by simply looking through the doc and the samples. But, since i started this discussion let me ask you something else please:

Q1. When I obtain my IToken, how can i check what DynamicLexicalPatternGroup it belongs too? I haven't seen any property within the IToken object that gives me that info.

Q2.I'm also trying to brows through my collection of PatternGroups and acquire a specific DynamicLexicalPatternGroup so i can remove the patterns and add new ones in real-time, but how can i access that collection? Is it through the CurrentSnapshot again?

Thank you
Nassim

Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Nassim,

1) Sorry, but the tokens don't track the pattern group that created them. You should use things like the token ID, Key, or ClassificationType to identify various tokens.

2) You can do something like this...

var lexer = language.GetLexer() as DynamicLexer;
if (lexer != null) {
    using (IDisposable batch = lexer.CreateChangeBatch()) {
        var patternGroup = lexer.DefaultLexicalState.LexicalPatternGroups["yourkey"];
        // Make changes here (assuming pattern group is in your default lexical state)
    }
}

Actipro Software Support

The latest build of this product (v24.1.2) was released 2 days ago, which was after the last post in this thread.

Comments (3)

Add Comment