
Hi,
I'm attempting to create a langauage (lexical and semantic parser) based heavily on your Simple example and have some newbie questions.
1) Tokens are created in SimpleLexicalParser.GetNextTokenLexicalParseData. Later in the semantic phase I will want to use the content that the tokens represent. Examples - for an Identifier token I want to get the identifier name, for a Number token the numeric value (or the original string). Is this the purpose of the IToken.Key ? If so, I find them all null even after the parsing has occurred and I walk through the Document.Tokens collection. I guess that I can go back and substring the document given the token range but that seems wrong. How should I store or access the token values ?
2) The constructor for LexicalStateAndIDTokenLexicalParseData(lexicalState, (byte)tokenID) casts tokenID to a byte. Isn't this a severe limit on the number of token types, especially when you are using one per keyword ? Elsewhere TokenID is an int.
3) In general I don't understand the purpose of ITokenLexicalParseData, ILexicalScope, ILexicalState etc. The documentation for these is quite bare. For example there isn't much if anything beyond a one line description of the interfaces and each property. ITokenLexicalParseData.TokenKey says "Gets the token key assigned to the token, if it is known". Yes but when is it known and when isn't it? How do I get it to be known? What about an example?
Sorry but I'm struggling to get my mind around even this 'Simple' example.
Paul
I'm attempting to create a langauage (lexical and semantic parser) based heavily on your Simple example and have some newbie questions.
1) Tokens are created in SimpleLexicalParser.GetNextTokenLexicalParseData. Later in the semantic phase I will want to use the content that the tokens represent. Examples - for an Identifier token I want to get the identifier name, for a Number token the numeric value (or the original string). Is this the purpose of the IToken.Key ? If so, I find them all null even after the parsing has occurred and I walk through the Document.Tokens collection. I guess that I can go back and substring the document given the token range but that seems wrong. How should I store or access the token values ?
2) The constructor for LexicalStateAndIDTokenLexicalParseData(lexicalState, (byte)tokenID) casts tokenID to a byte. Isn't this a severe limit on the number of token types, especially when you are using one per keyword ? Elsewhere TokenID is an int.
3) In general I don't understand the purpose of ITokenLexicalParseData, ILexicalScope, ILexicalState etc. The documentation for these is quite bare. For example there isn't much if anything beyond a one line description of the interfaces and each property. ITokenLexicalParseData.TokenKey says "Gets the token key assigned to the token, if it is known". Yes but when is it known and when isn't it? How do I get it to be known? What about an example?
Sorry but I'm struggling to get my mind around even this 'Simple' example.
Paul