Posted 17 years ago
by Actipro Software Support
- Cleveland, OH, USA
Eric and I have been in some deep discussions about an alternate approach to the whole token issue. This approach is based on what we see pretty much all the other competition doing.
Here is the summary of our talks:
1) Tokens will now only optionally be persisted with the document. By default they will not be persisted and will be retrieved on-demand by whatever needs info from them.
2) Tokens will remain with absolute offset/length information like now. Since by default tokens will no longer be persisted, the bottleneck is removed in this case. The bottleneck will remain if the option to persist tokens is enabled.
3) Tokens will no longer track highlighting style data. Instead a new StyleSpan class will be added. StyleSpan will track a start offset relative to the line that contains it, the length of the span, and a reference to a HighlightingStyle. The start offset will be an int, the length will be a byte. StyleSpans will not cross line boundaries. Since the spans don't cross line boundaries, and most spans will be < 255 chars, we think the length of 255 is fine. For any tokens that do cross line boundaries or are > 255 chars, they will simply have multiple StyleSpans added.
4) There will be some class or maybe the language itself that creates a StyleSpan from a parsed token.
5) In order to support incremental lexical parsing (for typing changes, etc.), each document line will need to store a stack of its lexical state at the start of the line. When typing occurs on a line, the stack will be retrieved and used to initialize the lexical parser at the line start position (actually probably a previous line start).
6) StyleSpans will only be added for non-default text. Take this line for example:
public int MethodName {
In that case, there would be a StyleSpan over "public" and one over "int". The rest would not have a style span over it and therefore would use the default style for the control.
7) SyntaxEditor will handle the processing/updating of StyleSpans although you would probably have access to methods that let you specify a range of text and a style to apply, along with clearing styles for a certain range. This would let you do ad-hoc coloring.
8) We may even be able to add an option for whether to persist StyleSpans for lines. If we are not persisting, we could just cache the spans for the lines that are visible and build new spans on-demand when we scroll, etc. Persistance would probably be required when word wrap is active and would be needed if doing ad-hoc coloring since otherwise your changes wouldn't be kept. Ability to use this option also depends on if it impacts scrolling performance negatively or not, which we wouldn't know until later.
So essentially we keep tokens focused on text only, they retain their absolute positional info, we have a lighter-weight class for maintaining colored text, we have no bottleneck on large documents, and we overall reduce memory.
Comments?
Here is the summary of our talks:
1) Tokens will now only optionally be persisted with the document. By default they will not be persisted and will be retrieved on-demand by whatever needs info from them.
2) Tokens will remain with absolute offset/length information like now. Since by default tokens will no longer be persisted, the bottleneck is removed in this case. The bottleneck will remain if the option to persist tokens is enabled.
3) Tokens will no longer track highlighting style data. Instead a new StyleSpan class will be added. StyleSpan will track a start offset relative to the line that contains it, the length of the span, and a reference to a HighlightingStyle. The start offset will be an int, the length will be a byte. StyleSpans will not cross line boundaries. Since the spans don't cross line boundaries, and most spans will be < 255 chars, we think the length of 255 is fine. For any tokens that do cross line boundaries or are > 255 chars, they will simply have multiple StyleSpans added.
4) There will be some class or maybe the language itself that creates a StyleSpan from a parsed token.
5) In order to support incremental lexical parsing (for typing changes, etc.), each document line will need to store a stack of its lexical state at the start of the line. When typing occurs on a line, the stack will be retrieved and used to initialize the lexical parser at the line start position (actually probably a previous line start).
6) StyleSpans will only be added for non-default text. Take this line for example:
public int MethodName {
In that case, there would be a StyleSpan over "public" and one over "int". The rest would not have a style span over it and therefore would use the default style for the control.
7) SyntaxEditor will handle the processing/updating of StyleSpans although you would probably have access to methods that let you specify a range of text and a style to apply, along with clearing styles for a certain range. This would let you do ad-hoc coloring.
8) We may even be able to add an option for whether to persist StyleSpans for lines. If we are not persisting, we could just cache the spans for the lines that are visible and build new spans on-demand when we scroll, etc. Persistance would probably be required when word wrap is active and would be needed if doing ad-hoc coloring since otherwise your changes wouldn't be kept. Ability to use this option also depends on if it impacts scrolling performance negatively or not, which we wouldn't know until later.
So essentially we keep tokens focused on text only, they retain their absolute positional info, we have a lighter-weight class for maintaining colored text, we have no bottleneck on large documents, and we overall reduce memory.
Comments?
Actipro Software Support
