Parser error when using a lexical transition

Hi Craig,

I think you're exactly right.  It's expecting one attribute value token but is getting multiple ones.  So your #2 idea of combining the tokens into a single token might be the best way to go.  Or if you change the XML grammar as you described later, that would handle it too.

For the other issue, I believe the context factory may be coded to expect a single attribute value token.  So the idea of combining multiple tokens into one instead of changing the grammar might help there.

Two questions:

1. How could I combine tokens using the Actipro add-on? I don't see anything like that mentioned in the documentation.

2. Is there any chance you'd be willing to alter the grammar, in an update, to use the ZeroOrMore() token choice?

Thanks again,

-Craig

Hi Craig,

1) There's nothing built in to do that sort of thing.  You'd need to make up new mock tokens in your token reader and pass them up that way.

2) I think your second problem is due to the context factory not expecting multiple tokens.  So both the grammar and that would need to have code updated for this and in ordinary circumstances, there's no issue with how they are now.  I'd suggest trying the token combining route first.

More questions:

1. I came up with a solution to merge the tokens in the token reader. The key portion of the solution is this:

private IToken MergeTokens()
{
    IToken mergableToken = null;

    var attributeValueTextToken = _cachedTokens.FirstOrDefault(item => item.Id == XmlTokenId.StartTagAttributeValueText);
    if (attributeValueTextToken != null)
    {
        // TODO: Can Actipro expose the LexerData property via an interface so we don't need Reflection?
        IMergableTokenLexerData lexerData = null;
        var lexerDataPropInfo = attributeValueTextToken.GetType().GetProperty("LexerData", BindingFlags.Instance | BindingFlags.NonPublic);
        if (lexerDataPropInfo != null)
        {
            lexerData = lexerDataPropInfo.GetValue(attributeValueTextToken, null) as IMergableTokenLexerData;
        }

        mergableToken = new MergableToken(_cachedTokens[0].StartOffset, _cachedTokens.Sum(item => item.Length), _cachedTokens[0].StartPosition, _cachedTokens[_cachedTokens.Count - 1].EndPosition, MergableLexerFlags.None, ((IMergableToken)attributeValueTextToken).LexicalState, lexerData);
    }

    _cachedTokens.Clear();
    return mergableToken;
}

Is this the proper way to merge multiple attribute text tokens together so the parser sees them as one?

2. In an update, can you expose the LexerData property on IToken or IMergableToken, so I don't need to use reflection when creating a new token?

3. Even with this change, my second problem still isn't solved. Hovering over the dat portion of the attribute's text still returns quick info for the element, not the attribute. Thus, it still seems like there's a bug in the generated XmlContext.

4. Also, I have to disagree with your statement that 'in ordinary circumstances', there's no issue with how the grammar and context factory behave now. While I realize I'm using a customize XML based language, what I'm trying to do would be perfectly legal in ASP (multiple ASP code nuggets with an html attribute). Built-in support for this type of embeding would be very helpful.

Thanks again,

-Craig

Hi Craig,

1/2) Since these tokens are just being used by the parser, they probably don't need to be complex at all.  As long as they implement IToken and have Id and the text range-related props set properly, I'd think that would be all you need to do.  The parser just examines token IDs as it iterates tokens, and the code that tracks offsets for the AST nodes and error reporting uses the text range data.

3) That one is hard to say what the problem is without debugging it.  I think you said you have the Blueprint source code.  Perhaps you can run through the context factory code there and see where it's going wrong.

4) All I'm saying is that we built the add-on according to XML specs and it does meet those properly.  Your scenario is where you're merging it with other languages, which does alter the grammar and/or context factory a bit from what works normally.  I don't really know the extent of the changes that would be necessary to support your scenario properly without debugging it.  Once you get #1 working as mentioned, I'm curious what you find in regards to where the context factory in #3 goes wrong.

1/2) I'm not quite clear on your recommendation here. Are you proposing that I author my own Token class that implements IToken, as opposed to using MergableToken?

3) I did a quick debug session and found the problem is that XmlContextFactory's CreateContext method doesn't use XmlParseData or the custom TokenReader I created when reading tokens. Instead, it just obtains a reader from the snapshotOffset's Snapshot. The result is that when examining an attribute with embedded code nuggets, it sees the attribute's text as multiple tokens instead of a single attribute text token.

From my original sample, the first portion of the attribute value (before the first code nugget) works correctly; the correct targetAttribute is identified, along with the correct targetElement. However, the other attribute text pieces are processed as individual tokens. After the call to UpdateTypeAndTextRange(), the targetAttribute is null and the XmlContext's Type is StartTagOther. 

Hi Craig,

1/2) Yes, that might be easiest or make a class that inherits TokenBase.

3) That makes sense since the token reader only gets used by the grammar-based IParser.  You'll probably have to modify the context factory code to accommodate the other tokens it can see, so that you skip over the tokens from the other language.