How to get tokens in Ast

SyntaxEditor for WPF Forum

Posted 8 years ago by SvenG - VIPA
Version: 11.2.0554
Avatar

Hi guys,

I have successfully built a test application where I can input some text and via a self built grammar i get an Ast. 

That was quite easy the documentation is very good and guided me straightforward !

Now I want to do some sort of code visualization where I want to do additional after analysis of the Ast.
In order to do that properly I need to know the token associated to an Ast node value. At the moment I only get the value and the offset which is not sufficient for my task.

Is it possible somehow to have the original Token in the SyntaxTree ? I haven't found anything appropriate in the documentation.

Thanks in advance!

Sven

Comments (4)

Posted 8 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hi Sven,

Unfortunately no.  The tokens are transient and we generate them on-demand.  Once consumed by the token reader used by the grammar, they disappear and nothing will hold onto them.

What sort of information are you looking to obtain from the token?  Perhaps we can suggest a way to gather that info.


Actipro Software Support

Posted 8 years ago by SvenG - VIPA
Avatar

Hi ActiproTeam,

thank you so much for your quick response! Let me elaborate more on that issue:
I have an Assembler like grammar that consists of lines of instructions and each instruction has an Operator and an Operand.

Something like this:

LOAD Variable1
ADD Variable2
SAVE Variable3

 

My custom built Ast looks something like this:

AstInstructionList[
  AstInstruction[
    AstOperator[
      "LOAD"
    ]
    AstOperand[
      "VARIABLE1"
    ]
  ]
  AstInstruction[
    AstOperator[
      "ADD"
    ]
    AstOperand[
      "VARIABLE2"
    ]
  ]
  AstInstruction[
    AstOperator[
      "SAVE"
    ]
    AstOperand[
      "VARIABLE3"
    ]
  ]
]

 

In a next step I want to visualize my code that displays combinations of lines in visual blocks. eg. above statement could be displayed as a visual box that has Variable 1 and Variable2 as Input parameters and Variable3
as an output parameter. The box is called "ADDER".


That means, when I traverse my Ast and find the operators LOAD, ADD and SAVE directly after another that results in a "ADDER" visual representation that I will output on a canvas.

If I could access the Tokens/TokenIDs that would be a fairly easy task and I wouldn't have to parse the value of the Ast node again.

[Modified 8 years ago]

Answer - Posted 8 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar

What about if you used type-specific AST nodes instead of DefaultAstNode?  With type-specific AST nodes, you can have our Language Designer code generate dedicated AST node classes for you.  So you could have an Instruction class that had Operator and Operand properties on it.  The Operator one could be an enumeration of possilble operators.

If you had that, then you could just cycle through your Instruction AST nodes and look at their properties.  That would be pretty straightforward.

Just FYI, the Getting Started 4d sample shows how to build and consume type-specific AST nodes.


Actipro Software Support

Posted 8 years ago by SvenG - VIPA
Avatar

Hi ActiproTeam,

I had to install/upgrade to the latest version (11.2) to use type-specific ASTNodes.
I tried your suggestion and it works like a charm.
Thank you very much for your help! 

Regards

Sven 

The latest build of this product (v2019.1 build 0683) was released 1 month ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.