using generated parsers

SyntaxEditor for Windows Forms Forum

Posted 12 years ago by Peter Chapman
Version: 4.0.0234
Avatar
I have a dynamic language defined in Xml, and a semantic parser generated from a grammar.

I can see examples of how to bring together lexical parsers and semantic parsers if they are both defined programmatically, but not if i'm using a dynamic language defined in Xml.

I can load the Xml into the document using LoadLanguageFromXml, but i can't see a way to get the lexical parser. I need it to instantiate my semantic parser, as far as i can see. Then, once i've instantiated my semantic parser, how do i set it as the language's semantic parser? I used to use something like:

Document.Language.SemanticParser = parser;

but this has gone and i don't see what it's been replaced by.

Or am i completely off track?

Thanks

Comments (6)

Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Hi Peter,

Everything is much better designed now so that pieces are modular. For instance all SyntaxLanguage classes have a PerformLexicalParse and PerformSemanticParse method. Dynamic languages inherit DynamicSyntaxLanguage and that implements PerformLexicalParse to perform the lexical parsing. So to add semantic parsing to a language, you would implement a custom DynamicSyntaxLanguage class (like in the sample projects in the Languages/Dyanmic folder) and overwrite PerformSemanticParse.

The XML definitions let you specify which code-behind class to load in their root SyntaxLanguage tag with the new SyntaxLanguageTypeName attribute. Look at Languages/Dynamic/Lexers/ActiproSoftware.CSharp.xml for a sample. It uses a code-behind class in a folder above that.


Actipro Software Support

Posted 12 years ago by Peter Chapman
Avatar
Ok, i'm nearly there with this, but there's one last gap in my understanding. I've implemented a custom DynamicSyntaxLanguage. I've updated the xml definition to point to the code-behind class. I've overridden the PerformSemanticParse method, and this is called correctly when it's time to semantic parse.

I've still got a problem with the bridge between the core lexical parser and the generated semantic parser. The constructor of the semantic parser requires a parameter of type IRecursiveDescentLexicalParser. I can't see where this object comes from. The closest i can see is that DynamicSyntaxLanguage implements IMergableLexicalParser.

How do i bridge this gap? I looked at making my own Lexical Parser class like your SimpleRecursiveDescentLexicalParser class, but i don't have a MergableLexicalParserManager to pass in the constructor. Similar trouble with deriving a class from RecursiveDescentLexicalParser.

public override void PerformSemanticParse(Document document, TextRange parseTextRange, SemanticParseFlags flags)
{
GeneratedSemanticParser semanticParser = new GeneratedSemanticParser(WTF??);
}
Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Sounds like you're very close.

IRecursiveDescentLexicalParser is an interface so you can write a bridge class yourself.

Alternatively, and probably the easier method is to use our existing implementations. The Simple language sample is your best resource for looking at what to do. Let's look at some things to do:

First, you need to tell the language to do semantic parsing in a separate thread when the semantic parser service runs. This is how you do that:
/// <summary>
/// Semantically parses the specified <see cref="TextRange"/> of the <see cref="Document"/>.
/// </summary>
/// <param name="document">The <see cref="Document"/> to examine.</param>
/// <param name="parseTextRange">A <see cref="TextRange"/> indicating the offset range to parse.</param>
/// <param name="flags">A <see cref="SemanticParseFlags"/> that indicates semantic parse flags.</param>
public override void PerformSemanticParse(Document document, TextRange parseTextRange, SemanticParseFlags flags) {
    SemanticParserService.Parse(new SemanticParserServiceRequest(SemanticParserServiceRequest.MediumPriority,
        document, parseTextRange, flags, this, document));
}
It makes a request to the semantic parser service and when the worker thread is ready to run the parse data (which may or may not be immediately), will callback to this interface implementation:
/// <summary>
/// Performs a semantic parsing operation using the data in the <see cref="SemanticParserServiceRequest"/>.
/// </summary>
/// <param name="request">A <see cref="SemanticParserServiceRequest"/> containing data about what to parse.</param>
void ISemanticParserServiceProcessor.Process(SemanticParserServiceRequest request) {
    MergableLexicalParserManager manager = new MergableLexicalParserManager(request.TextBufferReader, this);
    request.SemanticParseData = this.PerformSemanticParse(manager) as ISemanticParseData;
}
Note that that code creates the manager and calls another PerformSemanticParse method that is parse of the MergableSyntaxLanguage class (and can be called by the manager when in a merged language scenario).
/// <summary>
/// Semantically parses the text in the <see cref="MergableLexicalParserManager"/>.
/// </summary>
/// <param name="manager">The <see cref="MergableLexicalParserManager"/> that is managing the mergable language and the text to parse.</param>
/// <returns>An object that contains the results of the semantic parsing operation.</returns>
protected override object PerformSemanticParse(MergableLexicalParserManager manager) {
    SimpleRecursiveDescentLexicalParser lexicalParser = new SimpleRecursiveDescentLexicalParser(this, manager);
    lexicalParser.InitializeTokens();
    SimpleSemanticParser semanticParser = new SimpleSemanticParser(lexicalParser);
    semanticParser.Parse();
    return semanticParser.CompilationUnit;
}
That code creates a "bridge" lexical parser between the real lexical parser and the semantic parser. It uses the manager that is passed in.

So to sum up, in the main PerformSemanticParse, you make a request to the semantic parser service. At some later time, the ISemanticParserServiceProcessor.Process method is called. That makes a manager and calls a third method that runs the actual semantic parser.


Actipro Software Support

Posted 12 years ago by Peter Chapman
Avatar
Ok, thanks.

Now it compiles, but when i run it, i get a "Object reference not set to an instance of an object" on the call to lexicalParser.InitialiseTokens().

The lexicalParser object is instantiated, and a look inside shows the xml is loaded. Not sure where to look on this one.

I'll send through an example.

[Modified at 11/23/2006 09:03 AM]
Posted 12 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
This code in your recursive lexical parser had the problem:
protected override IToken GetNextTokenCore()
{
    IToken token = null;
    int startOffset = this.TextBufferReader.Offset;

    while (!this.IsAtEnd)
    {
        // Get the next token
        token = this.Manager.GetNextToken();

        return token;
    }

    // Return an end of document token
    if (this.Token != null)
        return this.Language.CreateDocumentEndToken(startOffset, this.Token.LexicalState);
    else
        return this.Language.CreateDocumentEndToken(startOffset, this.Language.DefaultLexicalState);
}
The line "return (Token);" is casting null to a Token so you are not returning a token, which is required. I think you meant it to be this:
return token;
Change your proc to this and it works:
protected override IToken GetNextTokenCore() {
    int startOffset = this.TextBufferReader.Offset;
    while (!this.IsAtEnd) {
        // Get the next token
        return this.Manager.GetNextToken();
    }

    // Return an end of document token
    if (this.Token != null)
        return this.Language.CreateDocumentEndToken(startOffset, this.Token.LexicalState);
    else
        return this.Language.CreateDocumentEndToken(startOffset, this.Language.DefaultLexicalState);
}


Actipro Software Support

Posted 12 years ago by Peter Chapman
Avatar
Thanks!
The latest build of this product (v2018.1 build 0341) was released 3 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.