proper use of lexical scopes/states

SyntaxEditor for WPF Forum

Posted 12 years ago by Kelly Leahy - Software Architect, Milliman
Version: 12.1.0561

I've been trying to understand how to update my language from SE4 to the new WPF/Silverlight language framework.

In addition to just updating my language, I wanted to try to use the new dynamic lexer stuff and understand how to use lexical scopes / states to help me with some of the lexing.

My language has several places where certain tokens are "special".  For instance, my language looks something like:

formula abc(t)
  frm applies to LOB1, LOB2
    for basis STAT
      def(t) + xyz(t)
    for basis TAX
      def(t) + taxxyz(t) + na(0)

  inp applies to LOB3

  na applies to LOB4

 In this codeblock, the abc(t) on line 1 is a "formula name", so I wanted to define a lexical scope / state for "formula name" so that I could syntax colour the abc(t) special.  Also, 'frm', 'inp', and 'na' before the 'applies to' lines are keywords in this context, but are not keywords if used in a formula (like on line 6).  Similarly, 'applies' and 'to' are special keywords that are only such if they are at the beginning of a applied formula (in other words, they are direct children of 'formula'.  Also, the LOB1, LOB2, LOB3, and LOB4 are 'categories' that should only be recognized as category IDs when they are in an 'applies to' (i.e. they immediately follow 'applies to' in a comma separated list.

Some of this stuff seems like the sort of thing that would be done in a parser generally, but I'm trying to understand if I should be trying to do this in lexical scopes / states in order to determine the correct syntax highlighting, or whether the highlighting should be "adjusted" as the result of a semantic analysis of the AST after parsing is complete?

Can you give me some guidance where you think the line should be drawn between lexical states/scopes and doing the work in the parser?  Also, how would I 'terminate' the lexical states for things like the formula name (which ends when you see frm/inp/na, but shouldn't "consume" those tokens), or LOB1,LOB2 (the applies to category list) that should end when you exit the "comma separated list"?

Kelly Leahy Software Architect Milliman, USA

Comments (3)

Answer - Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Kelly,

By the way, dynamic lexers in WPF/Silverlight function almost identically to SE4.  This language looks to be relatively ambiguous in regards to where things start and stop.  For instance there's no "end formula" sort of thing so even if you enter a lexical state when you recognize the formula keyword, you wouldn't ever know when to exit it.  Also there isn't a way to exit a lexical state without consuming a token.

That being said, you could possibly do things like use LookBehind patterns.  So for an identifier, if it did a look-behind where it watched for "formula" followed by whitespace, you could color that identifier ("abc") differently.

"applies to" might be able to be made into a lexical state start pattern, and then have the line terminator be the end pattern.  Then you could color your "LOB1", etc. differently in that state.

Things like "frm" you can have a LookAhead for to look for "applies to" after it.

See if that helps and gets you going.

Actipro Software Support

Posted 12 years ago by Kelly Leahy - Software Architect, Milliman

Thanks.  It sounds like my decision to not use dynamic languages with SE4 probably is still valid for the WPF control...

Is there any sort of migration guide for migrating SE4 lexers & languages to the new framework?

Kelly Leahy Software Architect Milliman, USA

Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Kelly,

Sorry we don't have a migration guide but the lexer end of things is very similar overall between WPF and WinForms.  In WPF, we also have a lot more samples on how to do everything and the docs are more expansive.

Actipro Software Support

The latest build of this product (v24.1.2) was released 2 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.