Problem with GoToPreviousNonWhitespaceOrCommentToken()

SyntaxEditor for Windows Forms Forum

Posted 7 years ago by Mike Dempsey - Sr. Developer, Teradata Corp
Version: 12.1.0303
Avatar

My dynamic language file contains the following:

<RegexPatternGroup TokenID="1" TokenKey="LineEnd" PatternValue="{LineTerminatorMacro}" IsWhitespace="True" />
<RegexPatternGroup TokenID="3" TokenKey="Whitespace" PatternValue="{WhitespaceMacro}+" IsWhitespace="True" />

I have the following text in the syntax editor:

Select abc

My code does the following:

Dim ts As TextStream = editor.Document.GetTextStream(8)
>>> ts.ParserText is 'abc' << correct
ts.GoToPreviousNonWhitespaceOrCommentToken()
>>> ts.ParserText is ' ' << incorrect

Since I am telling it to skip whitespace it should be pointing to tokenIndex 0 ('Select') not to tokenIndex 1 (' ').

Is there something I need to set in order to force it to honor the NOT whitespace or Comments part of this function?

 

Just as an aside...

What is the simplest way to create a TextStream that is pointing to the current cursor position?

Comments (5)

Posted 7 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hi Mike,

I'm not sure, I'd think it would work based on what you have.  If you want to make a new simple sample project and email that to our support address, we can debug it.  Please rename the .zip file extension so it doesn't get spam blocked.

We usually get the stream like:

syntaxEditor.Document.GetTextStream(syntaxEditor.Caret.Offset)


Actipro Software Support

Posted 7 years ago by Mike Dempsey - Sr. Developer, Teradata Corp
Avatar

I have sent a small demo app to support@actiprosoftware.com.

Answer - Posted 7 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hi Mike,

Thanks for the sample.  The problem was introduced in the prior build where CharClass was restructured to support Unicode categories.  There was one location in our lexer code that needed to handle things differently if a Unicode category was used (as it is now with WhitespaceMacro).

If you change your usage of {WhitespaceMacro} to be a manually specified class instead like [ \t] etc. then it should work around the problem until we get the next maintenanace release with the fix out.  Then you can restore what you have.


Actipro Software Support

Posted 7 years ago by Mike Dempsey - Sr. Developer, Teradata Corp
Avatar

I can get it to work if I use Explicit pattern groups as in:

<ExplicitPatternGroup TokenID="3" TokenKey="Whitespace" PatternValue=" " IsWhitespace="True" />
<ExplicitPatternGroup TokenID="3" TokenKey="Whitespace" PatternValue="&#x9;" IsWhitespace="True" />

But I have tried a number of variations of RegEx patterns and none of those seem to work.
eg. "\s+", "[\t\x20]+", "[\x20&#x9;]+"

The only problem with using Explicit groups is that we will end up with more tokens since strings of spaces or tabs will be broken up into multiple tokens.

Posted 7 years ago by Actipro Software Support - Cleveland, OH, USA
Avatar

Hi Mike,

Did you try "[ \t]+" as a pattern (note the space in there)?


Actipro Software Support

The latest build of this product (v2020.1 build 0400) was released 0 days ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.