Context-dependent lexical parser

Hello,

Yes this is the one area where our setup could have trouble with certain languages, since the lexer and parser are decoupled.  In scenarios like this, what we generally try to do is figure out by additional text scanning what the '[' might indicate. 

For instance if you know that the former case only works if an identifier is before the '[', then look for that to know it's a list/array access.  It seems like the latter case is more common in the grammar (expression-like) so that could probably be present in lots of scenarios.  Thus the former case is probably the one you should try and narrow down text searching criteria for, and base the decision on what '[' means off of that.

Thank you for the feedback. So since I cannot access the semantic context (since it is decoupled and constructed after the lexical parse), what is the proper approach to getting information about the previous token? 

I have implemented IMergableLexicalParser.GetNextTokenLexicalParseData to perform the lexical parsing and it returns a new LexicalStateAndIDTokenLexicalParseData instance with the correct token ID.

(btw: Lexial state is always 0 and I did not find much information about its proper use or intention)

If the next reader.Read() call returns my '[', then I would like to look at the previous token to determine which token to return. Is there a code snippet how to do that? Or would I need to approach this differently?

Hello,

You have it right, you would use the various reader methods to examine the text.  Note that it will only examine text, not tokens (since you are generating the tokens here), and you should make sure the reader's offset is back to where it was after you move it back.

For an example, here's a method we use in our C# programmatic lexical parser:

/// <summary>
/// Looks backward (skipping whitespace) to get the previous word.
/// </summary>
/// <param name="reader">An <see cref="ITextBufferReader"/> that is reading a text source.</param>
/// <returns>The previous word.</returns>
/// <remarks>
/// This method only looks at letters for sequential runs and will not scan words longer than 10 characters.
/// </remarks>
private string GetPreviousText(ITextBufferReader reader) {
	int offset = reader.Offset;
			
	char ch;
	string result = String.Empty;
	while ((!reader.IsAtStart) && (result.Length <= 10)) {
		ch = reader.ReadReverse();
		if ((Char.IsWhiteSpace(ch)) && (ch != '\n')) {
			// Break out of the loop if we already have a result
			if (result.Length > 0)
				break;
		}
		else if (Char.IsLetter(ch)) {
			// Append the letter
			result = ch + result;
		}
		else {
			// Set the result to the character if there is no result yet
			if (result.Length == 0)
				result = ch.ToString();
			break;
		}
	}

	// Return to the original offset
	while (reader.Offset < offset)
		reader.Read();

	return result;
}

Note how we make sure it returns to the original offset when it's done.  That makes sure that the lexer scan position remains in tact.  You could do something like above but modify the logic for your needs.