I am attempting to get syntax highlighting to work in a non-mergable language with a custom Token class. My language is derived from SyntaxLanguage, and my token class is derived from NonMergableTokenBase. I am using the ParserGenerator. Some aspects of the parsing are working correctly. I put breakpoints in the generated SemanticParser class, and I can see that my non-terminals are getting matched and created successfully. So at some level I know the lexical parsing is working.
I have implemented the public override bool ParseDataEquals(IToken token) method in my custom token class. I put a breakpoint in this method and it is NEVER called. I also implemented the public override IToken Clone(int startOffset, int length) method. There is NO documentation concerning what this method is supposed to do. What is it supposed to do? This method is called when my document is first parsed, using apparently the token returned by CreateInvalidToken from the language class, with the start offset set to 0 and the length to the document length. If I type in the editor window, it gets called on apparently existing valid token objects, with a start offset beyond the start offset of the object, and a negative length, which when added to the start offset seems to point to the next position after the token object that is being cloned. IOW, existing object has start offset os, length ol. Clone is called with parameters startOffset = os + ol + k, length = -(k-1).
One more odd thing, which maybe doesn't matter. When calling parseTarget.OnTokenParsed from within the PerformLexicalParse method, modeled after the documentation, after the call returns, the token start offset is reset to 0, and end offset is reset so that the length is still valid.
In the getHighlightingStyle method in the language class, I output debug info to the console, showing the token that is passed in. It is always the same token, with start offset = 0, and ID = the second last non-whitespace token in my document.
So the whole document is highlighted with that style.
I had syntax highlighting working in a mergable language, but I don't need merging. What am I missing here? Do I need to give up on this and just use the existing mergable language framework?
PS - If custom, non-mergable languages are supported, there should be documentation and samples showing how to implement them.
I have implemented the public override bool ParseDataEquals(IToken token) method in my custom token class. I put a breakpoint in this method and it is NEVER called. I also implemented the public override IToken Clone(int startOffset, int length) method. There is NO documentation concerning what this method is supposed to do. What is it supposed to do? This method is called when my document is first parsed, using apparently the token returned by CreateInvalidToken from the language class, with the start offset set to 0 and the length to the document length. If I type in the editor window, it gets called on apparently existing valid token objects, with a start offset beyond the start offset of the object, and a negative length, which when added to the start offset seems to point to the next position after the token object that is being cloned. IOW, existing object has start offset os, length ol. Clone is called with parameters startOffset = os + ol + k, length = -(k-1).
One more odd thing, which maybe doesn't matter. When calling parseTarget.OnTokenParsed from within the PerformLexicalParse method, modeled after the documentation, after the call returns, the token start offset is reset to 0, and end offset is reset so that the length is still valid.
In the getHighlightingStyle method in the language class, I output debug info to the console, showing the token that is passed in. It is always the same token, with start offset = 0, and ID = the second last non-whitespace token in my document.
So the whole document is highlighted with that style.
I had syntax highlighting working in a mergable language, but I don't need merging. What am I missing here? Do I need to give up on this and just use the existing mergable language framework?
PS - If custom, non-mergable languages are supported, there should be documentation and samples showing how to implement them.
