Lexer definition string interpolation csharp.langproj

SyntaxEditor for WPF Forum

Posted 9 months ago by Johan Majoor
Version: 23.1.3
Avatar

My language has something that is similar as the definition for interpolated strings in csharp.

Basically in the interpolated string you should be able to write any expression in the interpolated content.

This would then mean that the "Default" state should be made a child state of the interpolated content state. However since the default state doesn't have a scope (start/end delimiters) this doesn't work. In your example csharp language project you basically make all the children of the default state the children on the interpolated content and then you also duplicate most of the pattern groups. 

Besides the fact that this is not really maintainable to have a duplicate set of pattern groups, it actually still isn't correct.

For example:

var xx = $"before { new Class1() { P1 = "test", P2 = true} } after";

Here I use an expression with curly braces inside a interpolated content. As soon as it sees the curly close, it thinks that the interpolated content is closed whereas it should just close the property initializer. This is due to the fact that you have a pattern group that is the same as the end delimiter on the lexical state.

If the "Default" state would be set as a child state, then this curly close should just be in the default state and pop that state and return to the interpolated string state and things should work. Unfortunately since the default state doesn't have a scope and therefor won't be activated as a child state.

Now in other questions in this forum, you mention that a lexer should just do lexing and not structural interpretations. Though I agree with that, I don't see how you can really do that when there is arbitrary text as in this case.

What is the best way of defining such a definition, how can I make it do a state transition to the default state (as a child state) after the open curly brace of the interpolated content state?

Comments (5)

Answer - Posted 9 months ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hello,

What you could probably do is remove the two pattern groups for "{" and "}" in the CSharp.langproj Default lexical state.  In their place, add a new lexical scope to the Default lexical state where the start pattern group is for "{" and the end pattern group is for "}".  Set the lexical scope's IsAncestorEndScopeCheckEnabled = false.  Then add a child state reference in the Default lexical state to itself.

With that setup, the Default lexical state should be able to transition into and out of itself, maintaining a sense of structure.

I hope that helps!


Actipro Software Support

Posted 9 months ago by Johan Majoor
Avatar

Ok thanks, that does address the use of the curly brace end token inside the interpolated content.

Is there anything that can be done to avoid having to duplicate the pattern groups of the Default state inside the interpolated state?

Posted 9 months ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Unfortunately, the duplication cannot be avoided.


Actipro Software Support

Posted 9 months ago by Johan Majoor
Avatar

Thanks for the reply.

Just as an FYI. In C# you can also include an alignment expression and a format string as part of the interpolated content, which is currently not in your definition (though it is not relevant for my lexer)

Posted 9 months ago by Actipro Software Support - Cleveland, OH, USA
Avatar

You're absolutely correct. The XML-based C# language definition should cover the majority of the core language syntax, but some more advanced features may be missing.  Its primary purpose is to help serve as an example, so not sure we'll update it for the additional interopolated string syntax.

Of course, we do support that syntax with the more fully-featured C# language functionality based on the programmatic lexer available with our .NET Add-on.  I suspect you already know that, so I'm just mentioning it in case others come across this post and aren't clear about the difference between the free sample XML language files you've been discussing compared to the more advanced ones available through add-ons.


Actipro Software Support

The latest build of this product (v24.1.2) was released 2 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.