Adding tokens directly

I'm curious. Is what you want here really custom tokens? Or is it the ability to colorize the SE text according to the results of an external parser?

Bill and I were just having a discussion about this today and talking about the possibility of seperating out style from tokens. Possibly not even storing tokens with the document any more and just lexing the document, creating style spans from that process and then passing the tokens off to the semantic parser, but not storing them in memory with the document.

Hi Tobias,

Been working on a new system for the updated parsing model (prototyping with WPF) that lets you review the tokens parsed by the language and either modify them or classify various text ranges in different ways. It's really a nice model so far and lets you even add multiple layers of classifications for text before the editor ever receives anything.

So the gist is that the language lexer classifies the text via tokens. That info is stored in one classification layer. You tie classification types to highlighting styles. So "keywords" classification types can be designated as blue, etc. But then you could add another layer of classification results that classifies a token as something else as well and if that higher level classification has a style associated with it, it will merge into the more base classification layer, allowing you to override coloring, etc.

This should accommodate your request.