Language addition

There is a SemanticParseData property on Tokens. You can use that to store any custom data that you'd like.

This is not what I want.
I don't want to store information in individual tokens but rather add general grouping info in the language definition.

For example, a commentEnd and a LineFeed and a DocumentEnd all share the same meaning: they terminate a line, so in this simple example, I'd like to group them all with one category id.

There may be other token groups that for example allow to call help on them, so I'd like to add those to a group as well. Did I explain the idea sufficiently?

There may be more than one such logical grouping so I'd need a couple of categories or at least a bitmask.

This is not however about individual tokens or semanticparsedata. It's something about the quality/richness of the language definition.

Oh so you mean you want something on a lexical pattern group level, right?

Yes exactly, it would be something to add to the XML definition!

For lexical pattern groups, there is a Tag property that you can use to store whatever data you want. You'd have to assign it at run-time but that will work for you.