Issues getting started with a simpe language using programmatic lexer/parser

Hello,

You are missing the token tagger provider language service.  Basically that is a language service that other things can call to get tokens (via the lexer) and do syntax highlighting.  I would highly recommend that you walk through our Getting Started series of QuickStarts since they show you step-by-step how to build a language (and some options of things along the way).  Getting Started 3b is a basic programmatic lexer so that's a perfect example of what you want to do.

Thank you - that hint was helpful. I was actually trying to follow the QuickStart series, but have issues sorting out the important aspects in the examples.

I now have a custom lexer with syntax highlighting working in my project. 

The next hurdle appears to be the semantic parser. In WinForms, I have implemented a RecursiveDescentSemanticParser providing a Parse() method that returns a CompilationUnit implementing ICompilationUnit and ISemanticParseData. 

Following GettingStarted02, I would need to implement a class derived from ParserBase and provide Parse(IParseRequest request), returning IParseData. My IParseData implementation implements IParseData, IParseErrorProvider and provides Errors and the Ast. 

public class MyParseData : IParseData, IParseErrorProvider
{
	public IAstNode Ast { get; set; }
	public IEnumerable<IParseError> Errors { get; private set; }
	public ITextSnapshot Snapshot { get; private set; }

	public MyParseData(ITextSnapshot snapshot)
	{
		this.Snapshot = snapshot;
		this.Errors = new List<IParseError>();
	}
}

 Now the parser looks like this:

public class MyParser : ParserBase
{
	public MyParser() : base("my")
	{
	}

	public override IParseData Parse(IParseRequest request)
	{
		var parseData = new MyParseData(request.Snapshot);

		// TODO

		return parseData;
	}
}

 The TODO part is where the parsing should occur. 

In WinForms, the semantic parser derived from RecursiveDescentSemanticParser used to have access to 

• the LookAheadToken (including its Text and StartOffset)
• the (current) Token (including its Text and StartOffset)

Where do I find that in the WPF implementation? Have seen GettingStarted02 doing a ITextSnapshotReader reader = request.Snapshot.GetReader(0), but ITextSnapshotReader only has ReadToken().

Also, WinForms was using a class derived from MergableRecursiveDescentLexicalParser to filter out ignored token (whitespace, comments, ...). Where do I find that in the WPF implementation?

Overall, the transition from WinForms to WPF for syntax parsing appears to be non-trivial. The examples mainly revolve around building a script language with the Language Designer tool and the LL Parser framework. The information on building that 'directly' is pretty scarce. 

Hello, the ITextSnapshotReader has a ton of method for navigating the snapshot.  The Token property is effectively the one you are currently at (via the current offset).  The TokenText gives you its text.  You can use the ReadToken method to advance a single token or ReadTokenReverse to go the other way.  If you do reader.BufferReader.GetSubstring, you can get any segment of text from the snapshot.

All the MergableRecursiveDescentLexicalParser was in WinForms was a buffer layer between the lexer and parser.  We have a similar concept (called token reader) in our LL(*) Parser Framework.  If you are hand-rolling a parser, you could make something similar too.  Just make a class that looks at the snapshot reader and returns the filtered tokens to your IParser.

Another option when implementing this is instead of using the ITextSnapshotReader, work directly with a mergable lexer via something like this:

var coordinator = MergableLexerCoordinator.Create(reader, rootLexer);

If you pass it the core ITextBufferReader and an instance of the IMergableLexer class used by your language, it will give you a way to read the next token in a forward-only fashion.  It also supports look-aheads via Push/Pop methods.  And via the buffer reader, you can get any text substring.