Side Question: What do you use tokens for?

I have my own generic AST, semantic parser, and lexer framework.

I use tokens for several things from mouse events to highlighting to semantic parsing, I'll try and look over the weekend and see what exactly I 'need' them for. I'm quite sure that I need to at least be able to go from a mouse hover event to a token, though I may be able to remove this requirement with fancy footwork, as I'm not sure it's a good design right now...

Hi Kelly,

Didn't you say a while back you had some code snippets you wanted to send when we got into talking about this sort of thing? If you still would like to, please do send over what you were mentioning before.

Yep I do. I can't make them public, so I'll have to send them to you only, under the understanding that you won't publish them anywhere.

I'll do so as soon as I get a chance to dig them out of source control and isolate them into a separate project.

Ok, sounds good.

If it helps, I generally use tokens for the following (I am writing a SQL editor):

Assigning keywords to types, and colouring them:
Tables, Stored Procedures, Views, System Objects etc

Providing context-sensitive keyword lookups:
Most of my intelliprompts are based on looking a few tokens back and getting some basic contextual information. I don't really attempt to do a full blown SQL parse because I don't have the time (I have a day job too).

Mouse-over information:
When the user hovers the mouse on a token which is recognised as a specific type of keyword, I provide information about that keyword

Auto-completion:
I use the tokens to provide auto-completion for typing (ctrl+space)

Refactoring:
I use the tokens to provide a basis for simple refactoring functions

Not sure that's of any use - hope it helps!

Matt, do you need tokens to persist, as in, do you store information on tokens as you do the extra parsing. Or do you just consume them and build contextual information for IntelliPrompt based on which tokens pop up?

I just consume them, and find out what they said to figure out roughly what i want to pop up to the user.