Moving Semantic Parsing to the service - SyntaxEditor for Windows Forms Forum

Posted 17 years ago by Paul Fuller

Hi,

I have an 'advanced' SyntaxLanguage based mainly on the NonMergable CSharpSyntaxLanguage example that you provided some time back.

It implements PerformLexicalParse along the same lines as your example code.

I have added PerformSemanticParse myself. It produces a CompilationUnit that is used for outlining etc.

So far everything is great. However both the lexical and semantic processing occurs in the UI thread. Performance is generally very good but it can start to drag in large source files.

Now I would like to move the processing into a separate thread. Presumably using the SemanticParserService.

In various examples I have seen this being done for the semantic parsing so:


        public override void PerformSemanticParse(Document document, TextRange parseTextRange, SemanticParseFlags flags)
        {
            SemanticParserService.Parse(new SemanticParserServiceRequest(
                SemanticParserServiceRequest.MediumPriority
                , document
                , parseTextRange
                , SemanticParseFlags.None
                , this
                , document));
        }

Then the SyntaxLanguage needs to implement ISemanticParserServiceProcessor which is basically a Process method:


        void ISemanticParserServiceProcessor.Process(SemanticParserServiceRequest request)
        {
        ...
        }

The Process request parameter does not give me a document or TokenStream to work with. Instead it has a TextBufferReader. Is it supposed to perform lexical parsing on that stream ? Is this in addition to or in place of the existing PerformLexicalParse ?

The ideal would be to have both the lexical and semantic parsing out of the main UI thread if possible.

Do you have an 'advanced' language example that uses the SemanticParserService ?

Thanks,

Paul

Comments (3)

Posted 17 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Paul,

What happens when using the semantic parsing service is that the tokens are lexically parsed in the main UI thread during the lexical parsing phase. Each of those tokens is stored into the Document's tokens collection.

When semantic parsing occurs, it currently is using your lexical parser again but in a separate worker thread. As it performs semantic parsing and fetches the next token, it calls upon the lexical parser to parse it. So the tokens in this case are not the same token instances as what are in the document, although they may have the same meaning if no changes have been made to the document since the semantic parsing started.

Both of these lexical parsing phases use the ITextBufferReader so that is correct.

The Simple language shows a custom programmatic lexical parser that is used both in the main UI thread (for parsing document tokens) and in the semantic parsing phase.

Actipro Software Support

Posted 17 years ago by Paul Fuller

I suspected as much given the text reader parameter and that is fair enough.

Would it not be faster and easier to simply copy the tokens and give them to the service ? Seems like less overhead than recreating the same set from raw text over again. The copy would be done in the main thread before kicking off the service request.

Anyway I should be able to tackle this now that I understand the division of labour.

Thanks.

Posted 17 years ago by Actipro Software Support - Cleveland, OH, USA

That is something we considered back at the time it was designed however if you have a document that is really large (many MB) then there are potentially hundreds of thousands of token objects and cloning those every time something is typed would become a very lengthy operation, especially since it would be done in the main thread. This would lag typing by the end user. While there is a tad more work in the semantic parser with the way we did it, it is in a worker thread so the end user doesn't see the impact. That's the logic behind the decision.

Actipro Software Support

The latest build of this product (v24.1.1) was released 3 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.