When to Set the TokenIDType

SyntaxEditor for Windows Forms Forum

Posted 14 years ago by Markus Foser - Developer, PROCOS AG
Avatar
I generated the Tokens and the Sematic states class with Code Smith
But somehow the the token.id isn't set to an other value than 0
Do I have to set special properties or make a special call?


The code used to test


Document document = new Document();
Stream stream = SyntaxEditorHelper.GetLanguageStream(Language.CSharp);
// load language definition of CSharp
document.LoadLanguageFromXml(stream, 0);
document.Language.TokenIDType= typeof(CSharp.Tokens);
document.Language.LexicalStateIDType = typeof(CSharp.States);
// Set programming text
document.Text = Content;
foreach (Token token in document.Tokens)
{
Debug.WriteLine(token.ID);
Always prints out 0
}

namespace Test
{
public class CSharp
{
public class Tokens
{
public const int Default = 1;
public const int Whitespace = 2;
public const int LineTerminator = 3;
public const int OpenParenthesis = 4;
public const int CloseParenthesis = 5;
public const int OpenCurlyBrace = 6;
public const int CloseCurlyBrace = 7;
public const int OpenSquareBrace = 8;
public const int CloseSquareBrace = 9;
public const int Punctuation = 10;
public const int Null = 11;
public const int NativeType = 12;
public const int Boolean = 13;
public const int ReservedWord = 14;
public const int Identifier = 15;
public const int Operator = 16;
public const int RealNumber = 17;
public const int IntegerNumber = 18;
public const int HexIntegerNumber = 19;
public const int CharacterDefault = 20;
public const int CharacterEscapedCharacter = 21;
public const int CharacterWhitespace = 22;
public const int CharacterWord = 23;
public const int CharacterStart = 24;
public const int CharacterEnd = 25;
public const int StringDefault = 26;
public const int StringEscapedCharacter = 27;
public const int StringWhitespace = 28;
public const int StringWord = 29;
public const int StringStart = 30;
public const int StringEnd = 31;
public const int VerbatimStringDefault = 32;
public const int VerbatimStringEscapedQuote = 33;
public const int VerbatimStringWhitespace = 34;
public const int VerbatimStringWord = 35;
public const int VerbatimStringStart = 36;
public const int VerbatimStringEnd = 37;
public const int CommentDefault = 38;
public const int CommentDelimiter = 39;
public const int CommentWhitespace = 40;
public const int CommentURL = 41;
public const int CommentLineTerminator = 42;
public const int CommentWord = 43;
public const int CommentStart = 44;
public const int CommentEnd = 45;
public const int MultiLineCommentDefault = 46;
public const int MultiLineCommentWhitespace = 47;
public const int MultiLineCommentLineTerminator = 48;
public const int MultiLineCommentURL = 49;
public const int MultiLineCommentWord = 50;
public const int MultiLineCommentStart = 51;
public const int MultiLineCommentEnd = 52;
public const int XMLCommentDefault = 53;
public const int XMLCommentDelimiter = 54;
public const int XMLCommentStartTag = 55;
public const int XMLCommentEndTag = 56;
public const int XMLCommentWhitespace = 57;
public const int XMLCommentLineTerminator = 58;
public const int XMLCommentURL = 59;
public const int XMLCommentWord = 60;
public const int XMLCommentStart = 61;
public const int XMLCommentEnd = 62;
public const int PreProcessorDirectiveDefault = 63;
public const int PreProcessorDirectiveWhitespace = 64;
public const int PreProcessorDirectiveWord = 65;
public const int RegionPreProcessorDirectiveStart = 66;
public const int PreProcessorDirectiveEnd = 67;
public const int EndRegionPreProcessorDirectiveStart = 68;
public const int DefinePreProcessorDirectiveStart = 69;
public const int UndefPreProcessorDirectiveStart = 70;
public const int IfPreProcessorDirectiveIfStart = 71;
public const int IfPreProcessorDirectiveElIfStart = 72;
public const int IfPreProcessorDirectiveElseStart = 73;
public const int IfPreProcessorDirectiveEndIfStart = 74;
public const int LinePreProcessorDirectiveStart = 75;
public const int WarningPreProcessorDirectiveStart = 76;
public const int ErrorPreProcessorDirectiveStart = 77;
}

public class States
{
public const int DefaultState = 1;
public const int CharacterState = 2;
public const int StringState = 3;
public const int VerbatimStringState = 4;
public const int CommentState = 5;
public const int MultiLineCommentState = 6;
public const int XMLCommentState = 7;
public const int PreProcessorDirectiveState = 8;
public const int PreProcessorDirectiveCommentState = 9;
}
}
}

Comments (5)

Posted 14 years ago by Markus Foser - Developer, PROCOS AG
Avatar
I found a Solution with code.

Document document = new Document();
Stream stream = SyntaxEditorHelper.GetLanguageStream(Language.CSharp);
// load language definition of CSharp
document.LoadLanguageFromXml(stream, 0);
document.Language.TokenIDType = typeof(CSharp.Tokens);
document.Language.LexicalStateIDType = typeof(CSharp.States);
MemoryStream memStream = new MemoryStream();
document.Language.SaveToXml(memStream,1);
memStream.Position=0;
document.LoadLanguageFromXml(memStream,1);
document.Text = Content;
foreach (Token token in document.Tokens)
{
Debug.WriteLine(token.ID);
Always prints out 0
}

Or I can set the fully qualified TokenIDTypeName and StateIDTypeName attributes in the SyntaxEditorLanguage Element.

<SyntaxLanguage Key="C#" LanguageDefinitionVersion="3.0" Secure="True" xmlns="http://ActiproSoftware/SyntaxEditor/3.0/LanguageDefinition" TokenIDTypeName="PROCOS.WinControls.CodeEditor.Engine.CSharp+Tokens, PROCOS.WinControls.CodeEditor.Engine, Version=7.0.0.0, Culture=neutral, PublicKeyToken=f36351223a00f7d0" StateIDTypeName="PROCOS.WinControls.CodeEditor.Engine.CSharp+States, PROCOS.WinControls.CodeEditor.Engine, Version=7.0.0.0, Culture=neutral, PublicKeyToken=f36351223a00f7d0" >
Posted 14 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Exactly, you have to tell it which Types have the consts in the XML or it can't translate the names when loading the definition.

The document.Language.TokenIDType and document.Language.LexicalStateIDType properties are only useful when saving a language back out to XML. When you do that it will attempt to examine those Types to find a const with each TokenID value.

Also FYI, shortening this:
TokenIDTypeName="PROCOS.WinControls.CodeEditor.Engine.CSharp+Tokens, PROCOS.WinControls.CodeEditor.Engine, 
Version=7.0.0.0, Culture=neutral, PublicKeyToken=f36351223a00f7d0"
to this will work:
TokenIDTypeName="PROCOS.WinControls.CodeEditor.Engine.CSharp+Tokens, PROCOS.WinControls.CodeEditor.Engine"
That prevents you from having to upgrade version numbers any time you change your assemblies. However it is less specific and could pick up the wrong one if you have multiple assembly versions on your system.


Actipro Software Support

Posted 14 years ago by tobias weltner
Avatar
Unfortunately, I seem to be unable to get this to work.

How do I find the fully qualified name of my types in a vb.net project? Could you may be provide a small example?

I tried to find the name by using GetType which resulted in something like "AppName.mCommons+tokenID" but LoadLanguageFromXML complains about not being able to load it.

I put the definitions as enum in a module. Do they need to go elsewhere?
Posted 14 years ago by tobias weltner
Avatar
Ah, I begin to understand. From what I see, ALL token names need to be defined in the enum. Is that right?
I would much rather have defined IDs only for those tokens that I really need to care about.
So if I am correct, all I need to do is define ALL token names with appropriate IDs, right?
Posted 14 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
When you specify the type name, you have to at a minimum include the assembly name as well or .NET won't resolve it.

Yes all token names need to be defined. That's why we include a CodeSmith template so that if you use CodeSmith you can auto-generate the ID class.


Actipro Software Support

The latest build of this product (v2020.1 build 0400) was released 5 days ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.