Implementing a Mergable Programmatic Lexer, How to Define Scopes and Transition between States?

SyntaxEditor for WPF Forum

Posted 5 years ago by Maashes Phillips
Version: 14.1.0602
Avatar

Hi, 

We are implemeting a mergable programmatic lexer for a test language. We currently have all necessary classes such as the Lexer, TokenTagger, StateID, TokenId, etc.

Our issue is, how do we define a child lexical state that our parents lexical state will transition into upon typing of a character?

For example, when the user types the '<' character, we would like to transition into a different state. And when they type the '>' character we would like to transition out of that state. We have read through the documentation and QuickStart examples regarding the programmatic lexer, but they fall short of defining transitions. 

 

Thanks

Maashes

Comments (1)

Posted 5 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hello,

I apologize for the delayed reply.  We never received an email notification for this post and just noticed it now.  Here's how we do scopes with our C# language in the .NET Languages Add-on:

// In lexer constructor:

ProgrammaticLexicalState docCommentState = new ProgrammaticLexicalState(CSharpLexicalStateId.DocumentationComment, "Documentation comment");
docCommentState.LexicalScopes.Add(new ProgrammaticLexicalScope(new ProgrammaticLexicalScopeMatch(this.IsDocumentationCommentStateScopeStart), new ProgrammaticLexicalScopeMatch(this.IsDocumentationCommentStateScopeEnd)));
this.LexicalStates.Add(docCommentState);
defaultState.ChildLexicalStates.Add(docCommentState);

// Scope methods:

private MergableLexerResult IsDocumentationCommentStateScopeStart(ITextBufferReader reader, ILexicalScope lexicalScope) {
	if (reader.Peek() == '/') {
		reader.Read();
		if (reader.Peek() == '/') {
			reader.Read();
			if (reader.Peek() == '/') {
				reader.Read();
				return new MergableLexerResult(MatchType.ExactMatch, new LexicalScopeTokenData(lexicalScope, CSharpTokenId.DocumentationCommentDelimiter));
			}
			reader.ReadReverse();
		}
		reader.ReadReverse();
	}
	return MergableLexerResult.NoMatch;
}

private MergableLexerResult IsDocumentationCommentStateScopeEnd(ITextBufferReader reader, ILexicalScope lexicalScope) {
	if (Array.IndexOf(lineTerminatorChars, reader.Peek()) != -1) {
		reader.Read();
		return new MergableLexerResult(MatchType.ExactMatch, new LexicalScopeTokenData(lexicalScope, CSharpTokenId.LineTerminator));
	}
	return MergableLexerResult.NoMatch;
}

This is a bit more complex than yours will be but it should help.


Actipro Software Support

The latest build of this product (v2019.1 build 0683) was released 1 month ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.