Coordinate lexer/syntax editor and own lexer offset

SyntaxEditor for WPF Forum

Posted 14 years ago by Bernie Schoch
Version: 9.2.0511
I'm in the process of integrating my existing lexer to a programmatic lexer without change much in my existing lexer. The strategy is to create a shell lexer component that 'knows' about the actipro lexer interfaces and then call my existing lexer. Because it's used elsewhere in the code I don't want to make the existing lexer have any knowledge of the actipro interfaces.

So the shell gets control (and I position to start of line) and call OnPreParse with new offset. I then create a reader with that offset. Now here is the tricky part and the question. My existing lexer takes a string (once at initialization) and then I call GetNextoken on it and it returns my tokenid int. So the idea is to get the string from the reader using reader.GetSubString(offset, reader.Lengh - offset) and pass that string to my lexer. I then proceed to call in a loop my GetNextToken in my own lexer and with each return I will call OnTokenParsed. The question is will this work or will the actipro lexer interface be confused about where that token came from in the line (since I took in an entire substring? If so what is the best solution to keep it synchronized? or in general when doing this?


Comments (2)

Posted 14 years ago by Actipro Software Support - Cleveland, OH, USA
Hi Bernie,

As long as you call OnPreParse before and OnPostParse after your sequential OnTokenParsed calls, it should work ok. However one problem could be performance if you start doing very large documents since you are effectively building a full string of the document from the lexer start through the snapshot end for each text change here. If you are doing small-medium size files, it may not noticably impact performance though. A better solution would be to pass the low-level ITextBufferReader directly to your lexer and have it use that to read characters instead of reading via a string.

Actipro Software Support

Posted 14 years ago by Bernie Schoch
I would have preferred to use some more direct method but I don't want to couple the actual lexor with the syntax editor class library since it's used elsewhere. I would have been nice to have a common system class like TextReader (or I suppose I could wrap the ITextBufferReader but then there is also overhead for the extra calls while doing so). The size of the text won't matter since number of lines the people wlll editing would at a maximum (and extremely rarely) 200 lines or so. Usually just 10-20 lines.
The latest build of this product (v24.1.2) was released 2 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.