Whitespace Handling in ActiproSoftware.Simple.Grammar.xml

SyntaxEditor for Windows Forms Forum

Posted 8 years ago by Kevin McCormick
Version: 4.0.0289
Avatar
I'm midway through developing a combination of a Dynamic language and a semantic parser in order to write a custom expression editor in our software. One thing I don't entirely understand while looking through the sample grammar file, is the handling of whitespace. Whitespace is identified as a valid token, but it is not included in any of the grammar. Is there something that causes either the lexical or semantic parser to skip whitespace?

Comments (2)

Posted 8 years ago by Kevin McCormick
Avatar
At the risk of answering my own question, I think I got it. Because I needed to "bridge" the dynamic lexer to a lexer that derives from MergableRecursiveDescentParser, I needed to implement GetNextTokenCore.

In my previous implementation (copied from http://www.actiprosoftware.com/support/forums/viewforumtopic.aspx?forumtopicid=1940), the next token from the manager was always returned.

I modified the code to check the .IsWhiteSpace property, and to consume that token as such:

protected override IToken GetNextTokenCore()
            {
                int startOffset = this.TextBufferReader.Offset;
                while (!this.IsAtEnd)
                {
                    //consume whitespace tokens
                    IToken token = this.Manager.GetNextToken();
                    if (!token.IsWhitespace)
                        return token;
                }

                // Return an end of document token
                if (this.Token != null)
                    return this.Language.CreateDocumentEndToken(startOffset, this.Token.LexicalState);
                else
                    return this.Language.CreateDocumentEndToken(startOffset, this.Language.DefaultLexicalState);
            }
Posted 8 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar
Hi Kevin,

Yes the SimpleRecursiveDescentLexicalParser class in the sample is a bridge between the lexer and parser. So it can filter out tokens like comments and whitespace, thereby preventing your grammar from needing to watch for them.


Actipro Software Support

The latest build of this product (v2018.1 build 0341) was released 6 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.