Lexer

A lexer is the lowest-level key piece of a syntax language. It examines a stream of text and parses it into tokens that give text ranges meaning. Tokens are then used to support many other language features such as classification.

Creating a Lexer for a Language

The first step when creating a syntax language is to create its lexer.

Please see the text/parsing framework's Lexing series of topics for details on the various types of lexers and the benefits/drawbacks of each.

Once a lexer has been written, it needs to be associated with the language.

Registering a Lexer with a Language

Associating a lexer with a language is done by registering the lexer instance as an ILexer service on the language.

This code, called from a syntax language constructor, assigns a programmatic lexer named MyLexer to the language:

public MySyntaxLanguage() {
	this.RegisterLexer(new MyLexer());
}