Registration of TokenTaggerProvider - SyntaxEditor for WPF Forum

Posted 12 years ago by Ian Davis

Version: 12.2.0571

In the documentation, it states that to register a TokenTaggerProvider, we should use a line similar to

language.RegisterService(new TokenTaggerProvider<EcmaScriptTokenTagger>());

We register the service inside of the XxxLanguageWithParser class

    public class XxxSyntaxLanguageWithParser : XxxSyntaxLanguage
    {
        public XxxSyntaxLanguageWithParser()
        {
            RegisterService<IParser>(new XxxParser());
            RegisterService<ITokenIdProvider>(new XxxTokenId());
            // ...
            RegisterService(new TokenTaggerProvider<IdentifierHighlightingClassificationTagger>());
        }
    }

When we run this code, our token tagger is never instantiated. If we update the code to include removal of the generated TokenTaggerProvider which is automatically registered in the XxxSyntaxLangage ctor, our implementation is used.

XxxSyntaxLanguage:

    [System.CodeDom.Compiler.GeneratedCodeAttribute("LanguageDesigner", "12.2.573.0")]
    public partial class XxxSyntaxLanguage : SyntaxLanguage {
        
        /// <summary>
        /// Initializes a new instance of the <c>XxxSyntaxLanguage</c> class.
        /// </summary>
        public XxxSyntaxLanguage () : 
                base("Yyy Xxx Language") {

            // Create a classification type provider and register its classification types
            XxxClassificationTypeProvider classificationTypeProvider = new XxxClassificationTypeProvider();
            classificationTypeProvider.RegisterAll();

            // Register an ILexer service that can tokenize text
            this.RegisterService<ILexer>(new XxxLexer(classificationTypeProvider));

            // Register an ICodeDocumentTaggerProvider service that creates a token tagger for
            //   each document using the language
            this.RegisterService(new XxxTokenTaggerProvider(classificationTypeProvider));

            // Register an IExampleTextProvider service that provides example text
            this.RegisterService<IExampleTextProvider>(new XxxExampleTextProvider());
        }
    }

XxxSyntaxLanguageWithParser:

    public class XxxSyntaxLanguageWithParser : XxxSyntaxLanguage
    {
        public XxxSyntaxLanguageWithParser()
        {
            UnregisterService<XxxTokenTaggerProvider>();

            RegisterService<IParser>(new XxxParser());
            RegisterService<ITokenIdProvider>(new XxxTokenId());
            // ...
            RegisterService(new TokenTaggerProvider<IdentifierHighlightingClassificationTagger>());
        }
    }

The XxxTokenTaggerProvider does not derive from TokenTaggerProvider{T}.

Why do we have to remove the underlying provider to use our own when they are different implementations?
Are we supposed to implement all of our token tagging in the supplied XxxClassificationTypeProvider via a partial class?
As far as we can see, we are doing what the documentation says for registration. Are we doing something wrong?

Comments (3)

Answer - Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Ian,

The problem here is that only one token tagger can really be installed for a language. Since that is what gets used to tokenize text.

That being said, you can add as many classification taggers as you want. So perhaps you need to change your IdentifierHighlightingClassificationTagger into a classification tagger (ITagger<IClassificationTag>) instead.

Out of curiosity, does that particular highlight user type names in identifiers?

Actipro Software Support

Posted 12 years ago by Kelly Leahy - Software Architect, Milliman

I can speak for Ian on this one... The tagger actually tags a number of things in our UI, its name isn't really good.

Basically, we have a number of types of identifiers and keywords in our language that are "special" and can only have their type / classification for highlighting identified after the parse is complete.

Can you describe the difference between the main token tagger and a classification tagger for us? I think we're a bit confused as to why there are two things like this. Also, maybe it's just the name of the method (ClassifyToken) in TokenTagger that's confusing us. For us, the reason we wanted to use the TokenTagger was that it seemed much easier to implement - the ClassifyToken method is EXACTLY what we want to use (looking at its signature at least), but we didn't see this method when we were working on a classification tagger - did we just miss it, or is it not thtere?

Thanks,

Kelly

Kelly Leahy Software Architect Milliman, USA

Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Kelly,

A language can only have one token tagger, since it is basically meant to be the output of the lexer assigned to the language. If we allowed more than one, then they would be fighting over each other when doing things like getting ITextSnapshotReader.Token.

That being said, the token tagger should be your higher level one, providing tokens for the things you know about. If you have a word that you don't know is something or another until after a parse, your token tagger should just mark it as an identifier.

Then a classification tagger can come in later to classify the text range as something else, like a user type. Taggers are all effectively callbacks (via GetTags) that need to examine a range and return any special classifications over that range. You can do them on-demand, as in the AdornmentsColorPreview QuickStart, or you could use a CollectionTagger<T> base class to set them up ahead of time for the whole document (like in AdornmentsIntraTextNotes QuickStart) and in that case, we handle when to return results in GetTags.

Using CollectionTagger<T> is best if you know everything upfront (such as from your parse results) and don't have a lot of tags. Using the more on-demand method is better in cases where there are a lot of tags since then you can store your core data however you want (ideally optimized for quick lookup) and you only return what is necessary to render the visible view lines.

As for how to get a ClassifyToken kind of approach, in an on-demand GetTags implementation, you could get a reader like:

var reader = snapshotRange.Snapshot.GetReader(snapshotRange.StartOffset);

Then call ReadToken until you reach the end of the snapshotRange being asked for.

Actipro Software Support

The latest build of this product (v24.1.5) was released 2 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.