Tokens in Completion Provider - ITextSnapshotReader

Posted 5 years ago by Avatar Martin de Jong - Statistics Netherlands
In my completion provider i am trying to get a token of the ITextReader:
            // Get a snapshot reader and use it to see which token we're at
                ITextSnapshotReader reader = view.GetReader();
                IToken token = reader.Token;

                // Quit if we are in a comment
                if ((token != null && ( token.Id == Mediator.SL_COMMENT || token.Id == Mediator.ML_COMMENT)) )
                    return false;

                if (token == null) {  // TODO : doesnt work ever
                    token = reader.Token;
The reader is filled with text, but i cant get ANY token from it. I am using a mergable lexer to get my tokens:
            BlaiseLexer lexer = new Meta.Parsing.BlaiseLexer();
            string text =  reader.GetSubstring(reader.Offset , reader.Length-reader.Offset);
            Antlr.Runtime.IToken t = null;
                lexer.LoadText(text, "");
                t = lexer.NextToken();
                if (t.Text != null) {
                    reader.Offset += t.Text.Length;
            int tokenId = t.Type;
            if (tokenId != IDE.Language.Mediator.Invalid) {
                LexicalStateTokenData lsd = new LexicalStateTokenData(lexicalState, tokenId);
                return new MergableLexerResult(ActiproSoftware.Text.RegularExpressions.MatchType.ExactMatch, lsd);
            } else
                return MergableLexerResult.NoMatch;
It uses our Antlr lexer:)

Syntax highlighting works fine, even i made a squiggle. ( i have implemented a parser too showing errors)

I thought the Mergable Lexer would create the tokens and store them somewhere. Maybe i should store them myself? Or should i attach the lexer to the reader somehow?


[Modified at 02/03/2010 07:14 AM]

Comments (2)

Posted 5 years ago by Actipro Software Support - Cleveland, OH, USA
Hi Martin,

If you have syntax highlighting working then it must be tokenizing your text. Can you make a simple sample project that has a SyntaxEditor on a Window and shows the issue and email that over so we can take a look? Please do not include any .exe files in your ZIP or else the mail will bounce.

Actipro Software Support
Posted 5 years ago by Actipro Software Support - Cleveland, OH, USA
Hi Martin,

I think you may have modified something in the tagger provider output that prevented your token tagger from being found:

Your BlaiseTokenTaggerProvider.GetTagger<T> method had this incorrect line in it:
return ((ITagger<T>)(document.Properties.GetOrCreateSingleton(typeof(BlaiseTokenTagger), new ActiproSoftware.Text.Utility.PropertyDictionary.Creator<BlaiseTokenTagger>(factory.CreateTagger))));
The Language Designer generated output should have given you this instead (at least in 514 it does):
return ((ITagger<T>)(document.Properties.GetOrCreateSingleton(typeof(ITagger<ITokenTag>), new ActiproSoftware.Text.Utility.PropertyDictionary.Creator<BlaiseTokenTagger>(factory.CreateTagger))));
We are registering the token tagger that is created with the document properties under the key "typeof(ITagger<ITokenTag>)" here. Our ITextSnapshotReader looks for a document property with that exact key so that it can get tokens. In your case it wasn't found because you registered the property with key "typeof(BlaiseTokenTagger)" instead. So if you change that line it should work.

Actipro Software Support
Information The latest build of this product (2014.2 build 0610) was released 12 days ago, which was after the last post in this thread.

Add a Comment

Please log in to a validated account to post comments.