Updating a function list

Hi David,

We've been working with a couple customers now that people are getting into doing complicated semantic parsing similar to what you're describing. In a future release we're going to be adding some features to help with this.

The current plan is to track a LastParseAction property on Token. The property will be an enumeration of these members: None, Inserted, Modified, Deleted. Any time a lexical parse occurs on a Token, it will update that property so you know what happened. Any deleted Tokens will probably be needed to be passed as a parameter to PostParse since they are no longer in the Tokens collection of the document. But by using this information you should be able to tell exactly what Tokens were changed during a modification and how they were changed. Remember that each document modification also tells you the range of offsets (via LexicalParseOffsetRange) that were lexically modified so you know which range of Tokens to look at.

If you have any additional suggestions, please post them.

That sounds like it mike work great... hard to say until I understand the problem in more detail, but sounds reasonable.

But, as the component exists today, can I actually use the moodified event to do this myself? It does have the offsets of the changes, but is that useful information? The event and the documentation and everything talks about the modification in the past tense, leading me to believe that the modification already occured by the time this event fires. So having the offsets into the token stream wouldn't be useful because the token stream has changed. Say some text was deleted, if the event fires after I can't see the semenatic tokens of the deleted text anymore.

Or is the event fired before the actual modification? In that case it is also useless in that added text would not have been parsed and not be in the token stream at all.

Or is it before on add and after on delete? MIght also be useless unless text that is being replaced creates two modifications... which I don't think it does form the docs.

So I'm not sure this gets me by, does it? Or am I completely misundersatnding this?

So, the only way I see is to scan the whole token stream looking for function tokens, right? I can identify them by the first non-whitespace token on a line, so would scanning by line be easier? I notice that IDEs that do this (VS.NET whatever) seem to do this sort of table building asynchrounsly... though I doubt SyntaxEdit would let me be scanning the token stream on another thread safely. It could change while I'm looking at it without some special previsions which I doubt are built in, right?


-- David

If you look in the documentation under Parsing Guide -> Document Modification Sequence, you'll see a graph of what happens when a modification occurs. That should help explain the process.

As you said, there currently isn't a way to get the tokens that were removed after a parse. That's why we've designed those changes I mentioned, which should enable handling this sort of thing.

As for parsing, using a TokenStream is the preferred method. By using that you can quickly navigate through Tokens and handle them accordingly. Right now there is no specific support for asynchronous access to objects however that is something that potentially will be done in the future since complex background operations really require it to prevent slowdowns in the UI responsiveness.