SyntaxEditor 2.0 Design Feedback

We are looking for some creative feedback on how to complete the remaining features for 2.0.

Right now 2.0 is mostly finished other than finalizing semantic parsing and outlining. 2.0 has a ton of redesigned features and improvements. A beta will be made available soon however we do want to get some of your input first before we release a beta.

Here are some of the major changes to 2.0, although there are a very large list of things that were changed:

<li>Redesigned XML language definition format
<li>Instant-loading XML language definitions
<li>New NFA regex engine supporting more constructs
<li>Ability to completely modify languages at run-time
<li>Lexical parser that parses all text into tokens
<li>Ability to easily integrate semantic parsing
<li>Line modification markers like in Whidbey
<li>Outlining and auto case correction
<li>New IntelliPrompt member list helper methods
<li>Improved word wrap and bracket matching
<li>New documentation and much, much more

As you can see, the new lexical parser now creates a stream of Token objects. These Token objects enable the run-time parsing information to be much more object oriented. Whereas in 1.0, you had Character objects with properties like LanguageIndex, StateIndex, etc., in 2.0 you now have Token objects that just point right to the appropriate SyntaxLanguage or LexicalState object.

Our ideas for 2.0 are to have a two-phase approach. The first is the lexical parsing phase which is already complete. We need your input on how to design a semantic parsing phase.

The goal of the semantic parsing phase is to allow you to iterate through the Token objects and use them to gather and assign semantic parse data. Semantic parsing means to take a series of tokens and assign them meaningful values. You could place this semantic parse data on the Tokens themselves, on document lines, etc. For instance, maybe a C# line with a "using" statement was just entered. You'd scan the "using" Token, identify it as a "using" keyword and load the name after it into some sort of code model. Then when you required IntelliPrompting later on, you would know that a namespace was referenced.

The trick with semantic parsing is that it has to work for several completely different styles of languages. Languages like C# can flow across document lines but multiple statements can also be placed on one line. Languages like XML somehow need to track what tags are in the document, what attributes the tags might have, etc. Languages like CSS are probably about the easiest to parse and provide IntelliPrompt for.

Then there is outlining... we have an implementation of it semi-working but aren't sure if we like where it is done. Should outlining be part of the lexical parsing phase or do you want more control over it, thus putting it in the semantic parsing phase. If you put it there then the semantic parsing phase much have two purposes: (1) scan tokens and gather semantic data (2) modify the outlining tree as needed.

Based on the above, please post your initial ideas without seeing the actual implementation of 2.0 yet.

Let's get some discussions going so we can get this version finished and out the door.