How do I extract a specific pattern from a TokenId to create a terminal?

Hello,

Since the grammar-based parser works off of token IDs, you would have to assign the semi-colon its own unique token ID and then "classify" it the same as the other punctuation for the purposes of syntax highlighting.

Okay. Is it possible to go from a EBNF file to a grammar classs file? Would you recommend any way to do this if we already have our custom syntax EBNF rules?

There are so many forms of EBNF notation, so there's nothing to do an automated conversion.  You would need to hand write your EBNF rules in our C#-based EBNF-like syntax based on the documentation and samples.

Okay. Could you maybe provide an example of how the C# Operator token was converted into terminals? From the CSharp.langproj I can see you guys did this (this also applies to othe TokenIds like NativeType or ReservedWord):

https://i.imgur.com/fcz41lU.png

How was the terminals created for the different operators?

Hello,

Oh that CSharp.langdef was just a basic definition for a lexer to be used with syntax highlighting.  In that scenario, it doesn't matter if multiple keywords/operators use the same token ID.

However in our .NET Languages Add-on where we have a grammar-based parser, we use a programmatic lexer that defines a separate token ID for each keyword, operator, etc.  That's the only way a parser can identify each keyword/operator.