Lexical State Transition

Yes the lexical parser is greedy for speed optimization reasons. If it starts processing a pattern, it will continue parsing that pattern until the end condition of it is found. So if you have a <% within that pattern, nothing will be looking for it.

So the way to resolve this is to make your pattern so that it breaks on < characters. For instance if you have a pattern that is looking for everything but a double quote, then change it so that it looks for everything but a double quote and a < character.

Hi,

I used your suggestion - it works, thanks. But I faced with another problem. I merge SQL dynamic syntax language and CSharpSyntaxLanguage using Direct Lexical State Transition and custom tags <% and %>. When I type a SQL single line comment:

-- some <% %> comment

I noticed when I press Enter key inside the tags <% %> state transition doesn't pass to the next (new) line.

There's a problem with lexical scope, I suppose. But I don't know how to resolve it.

Could you give me any suggestion?

Thanks,
Nik

This is an interesting situation because it is a design feature of dynamic languages that you are running into. Namely, you have an exit scope of a newline character for your comment state. When you are in the <% %> block, the lexical parser sees that a parent language's scope end is found (newline character) and therefore closes out the state. Normally 99% of the time this is a desired feature.

Unfortunately with dynamic languages at least, I don't believe there is a way around this particular situation. If you would like to suggest some way to allow it, we are always open to ideas.

Hi,

Thanks for explanation. I supposed that the lexical parser doesn't see at parent scope, while it is in state transition. I need the lexical parser checks the parent language's scope end only outside the state transition.

It would be nice, if DynamicLexicalState or DynamicLexicalScope will have a property, which will allow to control the behavior of lexical parser.

Thanks,
Nik

Hi,

Reverting to the problem with state transition - if the lexical parser sees that a parent language's scope end is found, it closes out the state.

Could you add some option(s) to override this behaviour?

Thanks,
Nik

P.S. by the way. in SyntaxEditor Grammar Designer (TestApplication), I see similar problem, if type: or tag @> is not highlighted properly, but it may be just a highlighting defect.

Ok so just to recap, this is the current order that parsing occurs in the lexical parser:

1. Parent scopes (outside of the current language) to see if one ends the current state.
2. Child state start scopes to see if one starts a child state.
3. Pattern groups within the current state.
4. Parent scopes (inside of the current language) to see if one ends the current state.

You would like to add an option so that when a language transition occurs, only the lowest level parent scope outside of the current language (which is the direct parent of the child language) is allowed to exit the language. Whereas the current implementation allows that AND any other ancestor parent scope to end the language. Do I have it stated correctly?

Hi,

I suppose that you understand me right. To avoid misunderstanding (as I'm not so good at lexical parsing), could I explain this with an example - I'd like that the sample text below will be parsed and highlighted properly (merging SQL dynamic syntax language and CSharpSyntaxLanguage using Direct Lexical State Transition and custom tags <% and %>): i.e. state transition ends only if tag %> detected.

Thanks,
Nik

Ok I added a ILexicalScope.AncestorEndScopeCheckEnabled property for the next maintenance release. If you set that to false on a scope, it won't look at any parent scopes for an exit. So that should handle this situation.

I did a test in the Grammar Designer per the example you gave and it worked there.

It's a good news. Thanks.

Nik

This works fine. Thanks!

Nik