Posted 17 years ago by Hugh Pritchett
Version: 4.0.0238
Avatar
This is probably going to be a long posting, so I apologize. I just don't know what else to do. I am trying to get semantic parsing (using the semantic parser generator generated code) to work. I have been through the forums, the help documentation, and the example programs. I think I am almost there.

I have a syntax language defined in an .xml file. I have the attribute SyntaxLanguageTypeName pointing to my DynamicSyntaxLanguage code-behind class. Please note that I don't really want to use a dynamic syntax language but I guess there's not much of a choice when I want to run the semantic parser in another thread.

In my dynamic syntax class that is derived from DynamicOutliningSyntaxLanguage and implements the ISemanticParserServiceProcessor interface, I implement the ISemanticParserServiceProcessor.Process method:

        void ISemanticParserServiceProcessor.Process(SemanticParserServiceRequest oRequest)
        {
            MergableLexicalParserManager oManager = new MergableLexicalParserManager(oRequest.TextBufferReader, this, this.m_key);
            oRequest.SemanticParseData = this.PerformSemanticParse(oManager) as ISemanticParseData;
        }
Notice that the MergableLexicalParserManager constructor has a 3rd parameter that is a string. In all the examples I've seen in the postings, this parameter doesn't exist. There are no overloads either. I assume that this parameter was added in a later revision. The constructor for my dynamic syntax class has two parameters. One of them is a string with the name "key" (apparently it is suppose to be the filename of the source). In the constructor I assign it to the member variable "m_key". However, when my constructor is invoked, "key" is empty. No filename is being passed in from anywhere. It's important to note that I am not loading the source from a file. I'm doing a direct assignment of the source to the property editor.Document.Text.

Also in my dynamic syntax class, I override the overloaded PerformSemanticParse method:

public override void PerformSemanticParse(Document document, TextRange parseTextRange, SemanticParseFlags flags)
        {
            SemanticParserService.Parse(new SemanticParserServiceRequest(SemanticParserServiceRequest.MediumPriority,
                document, parseTextRange, SemanticParseFlags.None, this, document));
        }
Note that the SemanticParserServiceRequest has a new parameter SemanticParseFlags that does not exist in any of the examples in the postings.

Also in my dynamic syntax class, I override the overloaded PerformSemanticParse method:

protected override object PerformSemanticParse(MergableLexicalParserManager manager)
        {
            My_RecursiveLexicalParser oLexicalParser = new My_RecursiveLexicalParser(this, manager);
            oLexicalParser.InitializeTokens();
            My_SemanticParser oSemanticParser = new My_SemanticParser(oLexicalParser);
            oSemanticParser.Parse();
            return oSemanticParser.CompilationUnit;
        }
By the way, I don't want the class My_RecursiveLexicalParser in my project. I would much rather just use an object of type MergableRecursiveDescentLexicalParser as the "bridge" lexical parser. Is there any reason I need to define my own recursive descent lexical parser that derives from MergableRecursiveDescentLexicalParser? They do this in the examples and that's the only reason I did.

I used the Semantic Parser Generator included with Actipro to generate a Semantic Parser. It is just a very basic one that I made just to get something working. I will post the important properties and methods here:

    public class MyTokenID
    {

        /// <summary>
        /// Returns the string-based key for the specified token ID.
        /// </summary>
        /// <param name="id">The token ID to examine.</param>
        public static string GetTokenKey(int id)
        {
            System.Reflection.FieldInfo[] fields = typeof(MyTokenID).GetFields();
            foreach (System.Reflection.FieldInfo field in fields)
            {
                if ((field.IsStatic) && (field.IsLiteral) && (id.Equals(field.GetValue(null))))
                    return field.Name;
            }
            return null;
        }

        /// <summary>
        /// The Invalid token ID.
        /// </summary>
        public const int Invalid = 0;

        /// <summary>
        /// The DocumentEnd token ID.
        /// </summary>
        public const int DocumentEnd = 1;

        /// <summary>
        /// The ACSModifier token ID.
        /// </summary>
        public const int ACSModifier = 2;

        /// <summary>
        /// The MaxTokenID token ID.
        /// </summary>
        public const int MaxTokenID = 3;

    }

    internal class My_SemanticParser : ActiproSoftware.SyntaxEditor.ParserGenerator.RecursiveDescentSemanticParser
    {
        private CompilationUnit compilationUnit;

        public CompilationUnit CompilationUnit
        {
            get
            {
                return compilationUnit;
            }
        }

        public void Parse()
        {
            this.MatchCompilationUnit();
        }

        protected virtual bool MatchCompilationUnit()
        {
            compilationUnit = new CompilationUnit();
            compilationUnit.StartOffset = this.LookAheadToken.StartOffset;
            while (!this.IsAtEnd)
            {
                while (this.TokenIs(this.LookAheadToken, MyTokenID.ACSModifier))
                {
                   this.Match(MyTokenID.ACSModifier);
                }
            }
            compilationUnit.EndOffset = this.LookAheadToken.EndOffset;

            return true;
        }
The previous code block may be where my problem is. I'm not exactly sure what the MatchCompilationUnit() method should look like or what the CompilationUnit class should look like. All I really have is the example that came with the Semantic Parser Generator.

I put this in the constructor of my main form:

                SemanticParserService.Start();

                this.syntaxEditorCtrl.Document.Text = @"...bunch of source code...";
                DynamicSyntaxLanguage language = DynamicSyntaxLanguage.LoadFromXml(openFileDialog.FileName, nEncryptionKey);
                this.syntaxEditorCtrl.Document.Language = language;

                this.syntaxEditorCtrl.Document.Language.PerformSemanticParse(this.syntaxEditorCtrl.Document,
                                new TextRange(0, this.syntaxEditorCtrl.Document.Length - 1),
                                SemanticParseFlags.None);
Just to note, I would rather just use Document.LoadLanguageFromXml. Can I?

So, with all this said, I set several breakpoints in my code. The ISemanticParserServiceProcessor.Process(...) method and both PerformSemanticParse(...) methods are visited. However, in the ISemanticParserServiceProcessor.Process(...) method, the call oRequest.SemanticParseData = this.PerformSemanticParse(oManager) never returns. Looking deeper, in the overridden PerformSemanticParse(MergableLexicalParserManager manager) method, the call oSemanticParser.Parse() never returns. Back in the main form, after the call to PerformSemanticParse, the Document.SemanticParseData is null.

I know this is a long posting, but since I don't know exactly where the problem is, I wasn't sure on which area to focus. Hopefully someone can point me in the right direction.

Thanks in advance.

Comments (5)

Posted 17 years ago by Hugh Pritchett
Avatar
I found the main issue. Everything was working ok up to the point of where the Semantic Parser is invoked:

        protected virtual bool MatchCompilationUnit()
        {
            compilationUnit = new CompilationUnit();
            compilationUnit.StartOffset = this.LookAheadToken.StartOffset;
            while (!this.IsAtEnd)
            {
                while (this.TokenIs(this.LookAheadToken, MyTokenID.ACSModifier))
                {
                   this.Match(MyTokenID.ACSModifier);
                }
            }
            compilationUnit.EndOffset = this.LookAheadToken.EndOffset;

            return true;
        }
this.IsAtEnd is always false. So everything was pretty much working, it's just since the parsing is occurring in another thread, it never returns. I finally put 2-and-2 together.
I still need to find a way to resolve this but I'll close out this post and start a new post. It should be a little shorter this time :)
Posted 17 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Hi Hugh,

I was just writing the following when I saw your other post followup come through...

Ok let me try and answer your questions...

1) Actually when you want to use the semantic parser service, we generally do NOT use a dynamic language but you can if you want. The Simple language sample inherits MergableSyntaxLanguage which is the base class of DynamicSyntaxLanguage. So the Simple language is a sample of a language that uses the semantic parser service but is not a dynamic language. If you look at the Simple language's source code I bet it will answer most of your questions.

2) Look at SimpleSyntaxLanguage's ISemanticParserServiceProcessor.Process implementation, which is like this:
request.SemanticParseData = MergableLexicalParserManager.PerformSemanticParse(this, request.TextBufferReader, request.Filename) as ISemanticParseData;

3) Look at how SimpleSyntaxLanguage.PerformSemanticParse is implemented for how to use the params there too.

4) Your MergableRecursiveDescentLexicalParser implementation is the bridge between the lexical and semantic parsers. It filters out the non-important tokens that you defined in your language such as whitespace or comments. Since every language is defined differently, this is needed. And more advanced languages do more in this class. Like our C# add-on stores #region ranges, etc. for later use but prevents the semantic parser from ever seeing them.

5) A compilation unit is the root of your language. So in C# for instance, the root can contain using statements, namespace declarations, etc. So you need to declare what terminals/nonterminals are accessible from the root here.

6) Yes you can use Document.LoadLanguageFromXml. As long as your dynamic language has the code-behind type specification in the root SyntaxLanguage tag, it will load your custom SyntaxLanguage class.

7) It looks like your CompilationUnit parsing code is going into an infinite loop. What is the grammar XML you are using there?

Hope that helps!


Actipro Software Support

Posted 17 years ago by Kelly Leahy - Software Architect, Milliman
Avatar
Hugh,

Just as a note, your MatchCompilationUnit is not guaranteed to make progress. One thing about RD parsers is that when they go into a "while !EOF" loop (like yours is), there should be some progress made during each iteration, otherwise the loop can go on forever. In other words, there needs to be some code after "while (this.TokenIs...)" that makes some progress or breaks out of the loop due to syntax error. The easiest way to guarantee this is to have a terminal that terminates each "top level block" in your language (a top level block is basically what you're looking for in the outer "while" loop - in a modular language, this might be function declarations, in C# it might be type declarations or namespaces).

Kelly Leahy Software Architect Milliman, USA

Posted 17 years ago by Hugh Pritchett
Avatar
Thank you both for your comments. My problem is with the semantic parser which loops forever as you both noticed. I'm going to create a new post with questions about the parsing.

Thanks again!
Posted 17 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Kelly is exactly right, you need to have something in there for it to consume like a terminal (token) or a non-terminal. Otherwise this could happen. Please post your grammar XML when you ask your other questions too since that will help us see what is wrong.


Actipro Software Support

The latest build of this product (v24.1.0) was released 1 month ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.