Hi,
I´ve created a programmatic lexer by implementing the ILexer interface and been running into some serious performance issues. At the moment the implementation always analyzes the entire text and uses the OnTokenParsed-Method of the ILexerTarget to register the tokens.
The problem now is that the Parse-Method of the Lexer is often called hundreds of times. For example if I paste a text with 160 lines in an empty editor, the lexer is called more than 300 times. At the beginning it starts by lexing single lines (lines 1 through 20 - 20 calls). Then suddenly it jumps to lines 159 and 160, only to start right after this at lines 1 through 18 again. There are calls of the lexer that want to lex more than one line as well. After pasting the 160 lines the edior is unuseable. Typing a single character takes like half a second.
The same thing happens as soon as I scroll the text as well. One single scroll with the mouse wheel results in more than 130 calls of the lexer (this time only single lines but each line more than once).
Why is the lexer called so often and how can this be avoided?
Are there any things of special note that need to be considered when implementing a custom lexer?
Thanks in advance.
Tino.
I´ve created a programmatic lexer by implementing the ILexer interface and been running into some serious performance issues. At the moment the implementation always analyzes the entire text and uses the OnTokenParsed-Method of the ILexerTarget to register the tokens.
The problem now is that the Parse-Method of the Lexer is often called hundreds of times. For example if I paste a text with 160 lines in an empty editor, the lexer is called more than 300 times. At the beginning it starts by lexing single lines (lines 1 through 20 - 20 calls). Then suddenly it jumps to lines 159 and 160, only to start right after this at lines 1 through 18 again. There are calls of the lexer that want to lex more than one line as well. After pasting the 160 lines the edior is unuseable. Typing a single character takes like half a second.
The same thing happens as soon as I scroll the text as well. One single scroll with the mouse wheel results in more than 130 calls of the lexer (this time only single lines but each line more than once).
Why is the lexer called so often and how can this be avoided?
Are there any things of special note that need to be considered when implementing a custom lexer?
Thanks in advance.
Tino.
