Getting token list for Semantic parser?

Right now the lexer does create a secondary set of tokens for semantic parsing so that they don't interfere with the Document tokens. This is done only when using the semantic parser service since work is done in a secondary thread. However there is a difference.

The Document.Tokens are updated in the main thread and are all loaded upon lexical parsing completion. They stay in Document.Tokens. The semantic parser in a worker thread only calls the lexer one step at a time when reading a token and any old tokens that it has already read are discarded. So it dynamically parses tokens again using the lexer. When working in a separate thread you can't currently access the main Document.Tokens.