I've been trying to understand how to update my language from SE4 to the new WPF/Silverlight language framework.
In addition to just updating my language, I wanted to try to use the new dynamic lexer stuff and understand how to use lexical scopes / states to help me with some of the lexing.
My language has several places where certain tokens are "special". For instance, my language looks something like:
formula abc(t) frm applies to LOB1, LOB2 for basis STAT def(t) + xyz(t) for basis TAX def(t) + taxxyz(t) + na(0) inp applies to LOB3 na applies to LOB4
In this codeblock, the abc(t) on line 1 is a "formula name", so I wanted to define a lexical scope / state for "formula name" so that I could syntax colour the abc(t) special. Also, 'frm', 'inp', and 'na' before the 'applies to' lines are keywords in this context, but are not keywords if used in a formula (like on line 6). Similarly, 'applies' and 'to' are special keywords that are only such if they are at the beginning of a applied formula (in other words, they are direct children of 'formula'. Also, the LOB1, LOB2, LOB3, and LOB4 are 'categories' that should only be recognized as category IDs when they are in an 'applies to' (i.e. they immediately follow 'applies to' in a comma separated list.
Some of this stuff seems like the sort of thing that would be done in a parser generally, but I'm trying to understand if I should be trying to do this in lexical scopes / states in order to determine the correct syntax highlighting, or whether the highlighting should be "adjusted" as the result of a semantic analysis of the AST after parsing is complete?
Can you give me some guidance where you think the line should be drawn between lexical states/scopes and doing the work in the parser? Also, how would I 'terminate' the lexical states for things like the formula name (which ends when you see frm/inp/na, but shouldn't "consume" those tokens), or LOB1,LOB2 (the applies to category list) that should end when you exit the "comma separated list"?
Kelly Leahy Software Architect Milliman, USA