Adding tokens directly

SyntaxEditor Brainstorming Forum

Posted 16 years ago by tobias weltner
Currently, SyntaxEditor is creating all tokens so it is not possible to directly add tokens to a document. I can append a document text and have the parser generate the tokens according to the language implementation.

I would love to see a way to add new tokens directly, too. The idea is to use other (external) parsers and be able to convert their result into SyntaxEditor tokens.

Maybe this is possible already and I just overlooked it.

[Modified at 03/07/2008 09:11 AM]

Comments (2)

Posted 16 years ago by Eric J. Smith - CodeSmith Tools, LLC
I'm curious. Is what you want here really custom tokens? Or is it the ability to colorize the SE text according to the results of an external parser?

Bill and I were just having a discussion about this today and talking about the possibility of seperating out style from tokens. Possibly not even storing tokens with the document any more and just lexing the document, creating style spans from that process and then passing the tokens off to the semantic parser, but not storing them in memory with the document.
Posted 16 years ago by Actipro Software Support - Cleveland, OH, USA
Hi Tobias,

Been working on a new system for the updated parsing model (prototyping with WPF) that lets you review the tokens parsed by the language and either modify them or classify various text ranges in different ways. It's really a nice model so far and lets you even add multiple layers of classifications for text before the editor ever receives anything.

So the gist is that the language lexer classifies the text via tokens. That info is stored in one classification layer. You tie classification types to highlighting styles. So "keywords" classification types can be designated as blue, etc. But then you could add another layer of classification results that classifies a token as something else as well and if that higher level classification has a style associated with it, it will merge into the more base classification layer, allowing you to override coloring, etc.

This should accommodate your request.

Actipro Software Support