CTRL+END performance with large files

Hi Mick,

Could you send us a link to a sample large C# file like the ones you are opening so that we can test with it?  You can email the link to us if you would prefer to keep it private, but please don't email the actual file.  Thanks!

Hi,

Actually the file is a simple text file, what I did was use one of our NC files which contains text as shown below containing 23587564 lines. You could probably copy and paste this code snippet multiple times to make a large file. If this doesnt help I can see about uploading the file to our FTP site.

Thanks for the prompt response and have a great 4th!

$1




M31






G92X0Y0Z0
M116
G0W0.1
G4X0.5
G53X225M78M9
N8001!2!3!4
M01

G59X0Y0Z[L38]
G0Z0
G0Z-[L61]
G97S1=100M3
G95
G1Z-[L61]F8
N9001!2!3!4
T00M92
N9015!2
G140X=X1Z=Z1Y=Y1C=C0
M93
N9016!2
G140X=X1Z=Z1Y=Y1C=C1
N9014!2!3!4
M1

N8003!2!3!4

G59X0Y0Z[L38]
G63M108
G97S1=1800M3S4=1800M404T2121
M114
G0X[L25+1]
G0Z-[L30+L22]Y0
G0X14Y0M8
G314S1
!2!3!4
!4
G1X5F0.04
G1X4.5F0.02
!4
G314
G1X-0.6F0.04 
N9018!3!4

M114
G0X[L25+1]
G0Z-[L30+1.6]

G4X1
M195
G1X5F0.04
G1X-0.2F0.02 

IF[L1331]EQ[0]GOTO1200
G22A8000
L1331=0
N1200
G59X0Z[L38]
!4
G97S1=2000T101M3
M77M11
G4X0.2
G9G94Z[L21]Y0F5000
G4X0.2
G95M10
G4X0.2
G0W0.3
G53X225M109
M78M478
N8004!2!3!4
M30

$2
N102 ( @ INITIALISIEREN TS2  )
N104 G92X0Z0Y0
N106 IF[L1332]EQ[0]GOTO8001
N108 G53X225
N110 (Bereitstellen Revolver 2)
N112 T101
N114 !3
GT2
N116 !3!4
N118 !1!3!4
N120 M01
N122 ( @@ )
N124 !1!3!4
N126 (TORNIRE ESTERNOngcgcghchf)
N128 G59X0Y0Z[L61]
N130 G63G95
N132 G97S1=1800M3T101M108
N134 G0X16Z1Y0
N136 M115
N138 G46
G0Z2
G76X7.188Z-3.K0.42H2F0.75D0.02A57
G0Z40M451
N9011!3
G53X0Y-50Z150M405M477
M94M63(M195)
G4X1
N9012!3
(ABSORTIEREN)
G59Z[L37+L913]
G0Z25T0
G4X0.8
G1G94Z-23 F3000
G4X0.5
M62(M192)
G4X0.8
M411M477
G4X1
G53Z250M194 (...)
N9013!3
M95
G4X1
M63
G4X1
M193
N9014!1!2!3
M1
( @ Abstechen TS4 )
N8003!1!2!3
(Abstechen TS4)
G59Z0M411M477
T00
(M193)
!3
G0Z[L29+L22+L30+L911+5]
G4
G94
!1!2!3
G9Z[L29+L22+L30-L27+L911]F4000
G4X0.1
M410M10M77
G4X0.1
G1W0.01F500
G95
!1
!1
G53Z290
N9018!1!3
!1
N8004!1!2!3
M30
( @@ 4)

   

Hi Mick,

What happens whne you Ctrl+End is that SyntaxEditor needs to run the lexer on the text all the way up to the point at which you are displaying.  That is because in many languages you could have something like a /* up at the top that could affect everything down at the end. 

When I run a test using our normal free dynamic C# lexer, it takes a while but it's all trying to lex the millions of tokens.

When you run this file with the .NET Languages Add-on, it takes a while too but it's also running a parser on all these tokens in addition to all the normal lexing for syntax highlighting.

If I change it to Plain Text language, it will show instantly at the bottom because in that mode, it skips all lexing and of course the related syntax highlighting.

Unfortunately I'm not sure we can do anything else to speed up enormous files like that since the lexing does need to occur.  The only idea would be to have the wait for the lexer to complete to bail out after some point and just show the text unhighlighted at the bottom.  Then continue it on somehow in the background and update it when appropriate.  That would require some major refactoring of how our code works right now though.

Hi

Just picking up on this issue. I'm fine with the lexer having to work its way to the bottom of the file and accept this could take a while, but is their any chance of having some way to provide feedback to the user. Either an on screen progress bar or some kind of call back we could hook our own progress bar to.

Your last comment regarding displaying unhighlighted text, then colouring it as the lexer catches up; any closer to that now?

Thanks

Simon

Hi Simon,

At the time there isn't really a good place to add a feedback mechanism.  Even if there was, a lot of this layout is being done on the UI thread so trying to show something to the user probably wouldn't really show anything until the entire process was done anyhow.

When the view goes to perform a line layout, it uses a tag aggregator for IClassificationTags to gather the syntax highlighting.  The IClassificationTags primarily come from the TokenTagger but can come from other sources as well like indicators, etc. The tag aggregator requests classification tags for the text range that is effectively the view line's text range.  So when you do a Ctrl+End, it's just trying to gather classification tags for the last few lines of the document.  For a token tagger, this has the result of having to run through the entire document (if not yet lexed) and lex all the text, thus the slowdown in an enormous document.

The question is, where in that process would we have the token tagger bail out and how would we have it continue onward in the background, since these calls are made in the main UI thread?

I guess in an ideal world the tagger would be running on a worker thread, and the UI would be drawn in black and white until the tagger completes at which point the newly highlighted version would be drawn, but I guess thats a big change.

As for feedback, if the lexer reported progress via an event, then the editor control could display a progress indiator (overlayed), which it could force a UI refresh periodically. The progress indicator would update, but the rest of the UI would still be locked (and unresponsive) as the main thread is busy and not processing its msg queue, but it would give the user some confidence that things have not crashed.

If neither of these is possible then some kind of progress event from the lexer that we can access would allow us to provide some kind of UI feedback.

Thanks Simon

We'll have to continue to think about this.  Since the tagger is currently called by the UI thread and runs on the UI thread, even if we provided some feedback event that you could handle there, the way WPF works, nothing that updated UI would reflect the change until the entire tagging process was complete.  Also as you said, the option of moving that stuff to a worker thread would be a pretty massive change.