Lexical State Transition

SyntaxEditor for Windows Forms Forum

Posted 13 years ago by Nik Verkhovtseff
Version: 4.0.0234
Avatar
Hi,

I merge JScriptDynamicSyntaxLanguage and CSharpSyntaxLanguage using Direct Lexical State Transition and custom tags <% and %>. But if I print tag <% in the middle of JScript string, a state transition to C# doesn't occur (it works fine if I print tag <% right after forward quote).

Could you give me any suggestion?

Thanks,
Nik

Comments (10)

Posted 13 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Yes the lexical parser is greedy for speed optimization reasons. If it starts processing a pattern, it will continue parsing that pattern until the end condition of it is found. So if you have a <% within that pattern, nothing will be looking for it.

So the way to resolve this is to make your pattern so that it breaks on < characters. For instance if you have a pattern that is looking for everything but a double quote, then change it so that it looks for everything but a double quote and a < character.


Actipro Software Support

Posted 13 years ago by Nik Verkhovtseff
Avatar
Hi,

I used your suggestion - it works, thanks. But I faced with another problem. I merge SQL dynamic syntax language and CSharpSyntaxLanguage using Direct Lexical State Transition and custom tags <% and %>. When I type a SQL single line comment:

-- some <% %> comment

I noticed when I press Enter key inside the tags <% %> state transition doesn't pass to the next (new) line.

There's a problem with lexical scope, I suppose. But I don't know how to resolve it.

Could you give me any suggestion?

Thanks,
Nik
Posted 13 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
This is an interesting situation because it is a design feature of dynamic languages that you are running into. Namely, you have an exit scope of a newline character for your comment state. When you are in the <% %> block, the lexical parser sees that a parent language's scope end is found (newline character) and therefore closes out the state. Normally 99% of the time this is a desired feature.

Unfortunately with dynamic languages at least, I don't believe there is a way around this particular situation. If you would like to suggest some way to allow it, we are always open to ideas.


Actipro Software Support

Posted 13 years ago by Nik Verkhovtseff
Avatar
Hi,

Thanks for explanation. I supposed that the lexical parser doesn't see at parent scope, while it is in state transition. I need the lexical parser checks the parent language's scope end only outside the state transition.

It would be nice, if DynamicLexicalState or DynamicLexicalScope will have a property, which will allow to control the behavior of lexical parser.

Thanks,
Nik
Posted 13 years ago by Nik Verkhovtseff
Avatar
Hi,

Reverting to the problem with state transition - if the lexical parser sees that a parent language's scope end is found, it closes out the state.

Could you add some option(s) to override this behaviour?

Thanks,
Nik

P.S. by the way. in SyntaxEditor Grammar Designer (TestApplication), I see similar problem, if type:

"Identifier<@ out variableName 
@>"
or

"Block<@ out blockStatement @><- 
 ->"
tag @> is not highlighted properly, but it may be just a highlighting defect.
Posted 13 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Ok so just to recap, this is the current order that parsing occurs in the lexical parser:
  1. Parent scopes (outside of the current language) to see if one ends the current state.
  2. Child state start scopes to see if one starts a child state.
  3. Pattern groups within the current state.
  4. Parent scopes (inside of the current language) to see if one ends the current state.
You would like to add an option so that when a language transition occurs, only the lowest level parent scope outside of the current language (which is the direct parent of the child language) is allowed to exit the language. Whereas the current implementation allows that AND any other ancestor parent scope to end the language. Do I have it stated correctly?


Actipro Software Support

Posted 13 years ago by Nik Verkhovtseff
Avatar
Hi,

I suppose that you understand me right. To avoid misunderstanding (as I'm not so good at lexical parsing), could I explain this with an example - I'd like that the sample text below will be parsed and highlighted properly (merging SQL dynamic syntax language and CSharpSyntaxLanguage using Direct Lexical State Transition and custom tags <% and %>):

-- start of SQL comment  <% // here's C# code
int i = 0;
i += 1;
%> end of SQL comment 
i.e. state transition ends only if tag %> detected.

Thanks,
Nik
Posted 13 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Ok I added a ILexicalScope.AncestorEndScopeCheckEnabled property for the next maintenance release. If you set that to false on a scope, it won't look at any parent scopes for an exit. So that should handle this situation.

I did a test in the Grammar Designer per the example you gave and it worked there.


Actipro Software Support

Posted 13 years ago by Nik Verkhovtseff
Avatar
It's a good news. Thanks.

Nik
Posted 13 years ago by Nik Verkhovtseff
Avatar
This works fine. Thanks!

Nik
The latest build of this product (v2020.1 build 0400) was released 4 days ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.