Identifying tokens with DataflowLexer

Posted 15 years ago by Phil Devaney - Senior Software Engineer, Serck Controls Ltd

Version: 9.1.0503

I have a SyntaxLanguage that uses a DataflowLexer, and I am currently trying to implement Completion Lists. I need to know a bit of context to determine what to display in the completion list, so I call:

var token = view.GetReader().ReadTokenReverse();

however token.Key is null, and I have no way of tying token.ID to a particular token type.

I found Bill's post on MSDN about token descriptions in MGrammer (http://social.msdn.microsoft.com/Forums/en-US/oslo/thread/1d740ec6-e3d9-493f-9152-064525d1aa9d) and came up with this method:


private static string GetTokenDesc( ActiproSoftware.Text.Lexing.IToken token )
{
    foreach ( object o in Expression.Parser.GetTokenInfo( token.Id ) )
    {
        var descAttr = System.Dataflow.GraphNode.LookupGraph( "Description", o );
        if ( descAttr != null )
            return System.Dataflow.GraphNode.LookupGraphArgument( descAttr ) as string;
    }

    return null;
}

This seems to work if I add Description attributes to the tokens in my grammar definition, but its a bit convoluted. Do you have any plans/guidance for how to handle this? It would be nice to use integer comparisons instead of string comparisons - perhaps a tool that generates a class derived from DynamicParser tied to a particular parser, with public constants for each token type.

Comments (8)

Posted 15 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Phil,

Unfortunately it seems that unless we add some attributes to the MGrammar grammar file, there is no possible way for us to identify a "key" for a token. We'd be happy to discuss maybe some sort of best practice for this. Couple initial ideas...

1) Microsoft said any text attributes could be defined. Perhaps we make our add-on look for one called "TokenKey" and we pull in that value into the token if it's available?

2) It would also be nice if we could generate an ITokenIdProvider based on the DynamicParser too, possibly from our Language Desinger app, also using this "TokenKey" attribute. The only thing about that is that we'd need some plugin functionality to our Language Designer app because many people may not have MGrammar installed and we can't distribute those assemblies. The Language Designer is very much work in progress and far from complete, but ideally long term it will be a real helpful tool for working with anything related to languages.

What do you think?

By the way we haven't announced it yet but the next build has some awesome new features for the MGrammar add-on. With two more lines of code on your end, you can enable multi-threaded parsing of your document on text changes. So as you type it will call the MGrammar parser in a background thread, and asynchronously return parse data (which includes the AST and syntax error list) back to the document. We'll have a blog post on this soon.

Actipro Software Support

Posted 15 years ago by Phil Devaney - Senior Software Engineer, Serck Controls Ltd

I've had a look at ITokenProvider and it doesn't seem too hard to implement GetDescription and GetKey to pull the necessary information out of a DynamicParser. You would still need to do some codegen to get the constants needed for integer comparisons, I think generating a TokenidProvider class is better than generating a class derived from DynamicParser.

As a general note on the language designer, have you considered providing custom build actions, so you can add a language definition file to your project (be it MGrammar or your own custom XML format) and have it generate the code files within Visual Studio.

Posted 15 years ago by Actipro Software Support - Cleveland, OH, USA

Right, I meant generate a TokenIdProvider class based on the data from a given DynamicParser.

The custom build action is a great idea. I'll mark it down on our TODO list.

Actipro Software Support

Posted 15 years ago by Phil Devaney - Senior Software Engineer, Serck Controls Ltd

I've finally been able to look at this again after working on something completely unrelated for the past month.

I've implemented a DataflowTokenIdProvider class that iterates round the tokens of a DynamicParser and extracts a TokenKey attribute (using same the method as my first post). I then assigned an instance of the class to DataflowLexer.TokenIdProvider, but none of the methods on my TokenIdProvider seem to get called, and when I extract tokens, Token.Key is still null.

My class looks like this:


public class DataflowTokenIdProvider : ActiproSoftware.Text.Lexing.ITokenIdProvider
{
  private readonly string[] tokenKeys; 

  public DataflowTokenIdProvider( System.Dataflow.DynamicParser parser )
  {
    if ( parser == null )
      throw new ArgumentNullException( "parser" );

    this.tokenKeys = Enumerable.Range( 0, parser.TokenCount )
                     .Select( i => GetTokenKey( i, parser ) )
                     .ToArray();
  }

  public bool ContainsId( int id )
  {
    return id >= this.MinId && id <= this.MaxId;
  }

  public string GetDescription( int id )
  {
    throw new NotImplementedException();
  }

  public string GetKey( int id )
  {
    if ( id < this.MinId || id > this.MaxId )
      throw new ArgumentOutOfRangeException( "TODO", "id" );

    return this.tokenKeys[id];
  }

  public int MaxId
  {
    get { return this.tokenKeys.Length - 1; }
  }

  public int MinId
  {
    get { return 0; }
  }

  private static string GetTokenKey( int tokenId, System.Dataflow.DynamicParser parser )
  {
    foreach ( object o in parser.GetTokenInfo( tokenId ) )
    {
      var keyNode = System.Dataflow.GraphNode.LookupGraph( "TokenKey", o );
      if ( keyNode != null )
        return System.Dataflow.GraphNode.LookupGraphArgument( keyNode ) as string;
    }

    return null;
  }
}

Any ideas?

Posted 15 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Phil,

The ITokenIdProvider isn't used by our code directly yet. To get a key, you'd have to pass the ITokenIdProvider.GetTokenKey method your token's id value. We'll keep this post in mind for if we can enhance things in the future to make token.Key more automated based on an ITokenIdProvider. I don't believe our DataflowToken stores the lexer reference right now but maybe we could add that.

If you'd like to send over a simple sample project that shows what you're doing and how you'd like it to work, that would help us and maybe we could get something in place for you.

Actipro Software Support

Posted 15 years ago by Phil Devaney - Senior Software Engineer, Serck Controls Ltd

Sorry, I assumed that the purpose of ITokenIdProvider was to automatically fill out IToken.Key. If that isn't the case then what is it used for?

I don't actually need this right now as I've generated a TokenIds class of constants, as discussed earlier, so I can use token.Id instead of token.Key. This is good enough for me to get the logic of building a completion list right, and I can wait for a more robust solution.

Posted 15 years ago by Actipro Software Support - Cleveland, OH, USA

Phil,

The main reason right now is to provide a known place to go when looking for token ID constants. We'd like to use them elsewhere too (possibly in scenarios like yours) but are still working out the best way to harness their features.

Like I said, it would be most helpful if you could email over a simple sample project over showing an editor with your language and we can use that to help test some enhancements that we could possibly work in for the next maintenance release to help you with IToken.Key.

Actipro Software Support

Posted 15 years ago by Actipro Software Support - Cleveland, OH, USA

Thanks for the sample Phil. We've updated our add-on so the next build will populate IToken.Key based on the "TokenKey" custom attribute applied in your MGrammar source.

We've also updated our MGrammar Integration sample to show a quick info provider (new feature in next build) that displays a quick info tip of the token you are over, indicating its range and key.

Actipro Software Support

The latest build of this product (v24.1.2) was released 2 days ago, which was after the last post in this thread.

Comments (8)

Add Comment