Brief pre-sales questions - SyntaxEditor for Windows Forms Forum

Posted 18 years ago by Jeff Cohen

Version: 4.0.0233

Hi,

I just downloaded the 15-day trial and have a few quick questions. I've already played with writing my own dynamic language xml file, and though I can't get everything I need to work, it's probably close enough for my needs. But I have a few quick questions before I can commit to plunking down the cash :-)

[Big apologies in advance if the answers should have been obvious to me - I've immersed myself in the .chm help but some of the links don't work, or the information I want just isn't there.]

I'll need to use the control for both well-known file types (.html, .css, etc.) and a few custom ones (Ruby, and some in-house languages).

1. Our syntax is pretty simple: classes contain methods. Classes start with the keyword "class" and methods start with the keyword "def". Both are terminated with the keyword "end" (internal looping contructs start with "do" and stop with "end"). I've got all the color coding working, but for my real project I'll want to know the current method and class for an arbitrary location in the file. This is because I want a right-click menu at the current cursor locaton to change based on the current method name and class name. Is this easily done? I didn't see a Document method or property that looked promising.

2. Also, I need to easily find out where the current method starts/ends and the current class starts/ends so I can provide refactoring support. How do I do this? The sample has something similar, using a combobox to show the current class and method, but that seems to only support .NET languages. And, I may not want to turn on folding/outlining, but still want to understand the semantics of the code surrounding the current caret location. Is this possible?

3. Lastly, I want to activate a snippet at runtime whenever the user hits the TAB key based on the previous word; in other words, I want to look at the previous word (i.e. "for") and if the user presses the tab key next, I want to run the "for" snippet to provide a skeleton of the for loop. How do I do this?

4. Really lastly, does the purchased version contain more lexers (like for Ruby or JavaScript)? If not, that's no problem, I'm just curious.

Thanks a lot!
Jeff

Comments (10)

Posted 18 years ago by Actipro Software Support - Cleveland, OH, USA

Hi Jeff,

Let me answer your questions...

1) There are two ways you can do that sort of thing.

The first is the easier/quicker way but won't give you as detailed of information. In that you could get a TextStream via Document.GetTextStream and then iterate back through the tokens in the document and look for a keyword like "class" or "def". Then scan the Identifier token after it to see the name. Of course if you have a huge class, it can be somewhat of a perf hit to scan all the way up to the top of the document to find the class name.

The better, although more complex, way to do it is via semantic parsing. Use our semantic parser service (which runs in a separate thread) with our parser generator to construct an AST of your code document. That is what we do in our C# language from the .NET Languages Add-on. It's also what we do in the Simple language sample which has full source in the sample project. Once you have an AST you can easily determine what is where in the document. If you wanted to get this done really quickly you could just make grammar productions for your main constructs like classes, defs, etc. Then ignore a lot of the other keywords.

2) This sort of thing is probably only really able to be done if you do the AST approach mentioned above. Once you have an AST representation of the document all that can be done.

3) First you would load your code snippets. Then in the docs there is a section in this topic "General Features / IntelliPrompt Features / Code Snippets / Code Snippets at Run-Time" that has a code listing for easily activating a code snippet on Tab key press. You can handle that in the KeyTyping event (which also can be handled directly in your SyntaxLanguage class via OnSyntaxEditorKeyTyping).

4) There are a number of free lexers in the Languages/Dynamic/Lexers folder in the C# sample project. JScript is one although I don't believe we have a Ruby one yet. If you'd like to work with us to help develop a Ruby one, we'd be happy to add that.

Actipro Software Support

Posted 18 years ago by Jeff Cohen

Awesome. I'll try out the semantic parser later this week, but it sounds like it can do everything I need.

And thanks for your fast reply. The level of support here seems great, and gives me confidence that I can buy your product and be able to get help later if I really need it.

Jeff

Posted 18 years ago by Jeff Cohen

Once I use the generator to create the C# class for my language, I guess you're saying I'll be able to add methods to that class that utilize the AST model, right? Can you just point me to the classes or methods I should use that can get me started at walking the AST tree from the current caret position?

After starting the background semantic service, does my code need to explicitly tell it when to parse (after every keystroke, I guess) or will the Document communicate with the service automatically?

Thanks again.
Jeff

Posted 18 years ago by Actipro Software Support - Cleveland, OH, USA

IAstNode is the interface and AstNodeBase is the base class we use. Methods like FindNodeRecursive dig down to find the containing node. Then use FindAncestor to find a certain type of ancestor, like a method or class. Also our parser generator can make a class that implements a visitor pattern for you, which may help in certain circumstances.

Check out the Simple language sample in the sample project. It is full source and shows a basic implementation of a lot of really neat concepts.

Basically SyntaxEditor fires PerformLexicalParse and PerformSemanticParse for each modification to the document. You don't need to worry about kicking off parsing. In your language you then can kick off the semantic parser service like SimpleSyntaxLanguage.PerformSemanticParse does. The callback from the service goes to ISemanticParserServiceProcessor.Process. Since the Simple language is a mergable language it calls a mergable language manager, but if you don't need it to be mergable then it is recommended you just do the code like in the other overload of SimpleSyntaxLanguage.PerformSemanticParse (that overrides a method in MerableSyntaxLanguage) right there.

Check out the documentation too... it explains a lot of these concepts.

Actipro Software Support

Posted 18 years ago by Jeff Cohen

I'm continuing to pour over the .chm help file and I think the concepts are starting to sink in, but if I could just make sure I'm thinking about this the right way, I would appreciate it...

I think I understand that the dynamic language .xml file detemines how the syntax coloring works, optionally identifying each colored area as a token. I can also derive my own class for this as a "code behind" class to augment this behavior as well. The .xml is still loaded dynamically at runtime using the static method on DynamicSyntaxLanguage.

Also, the now see that the grammar .xml file can also be used to generate a separate class which can provide semantic interpretation of the Document object. This class must be manually generated and added to the project, and if I want semantic parsing I also use a "code behind" class to handle override PerformSemanticParse, etc., and then I can add my own public methods which can utilize the AstNodes to analyze the Document as desired.

If I'm right so far, then the only thing I don't get is how the semantic tokens correspond to the dynamic .xml "tokens", if at all? I think I understand EBNF, but I'm not seeing in the Simple example how it knows that, for example, Identifiers can start with an underscore or a letter. I was looking for some kind of defintition of that in the grammar .xml file but maybe I'm looking in the wrong place?

Are the <Token> tags in the grammar .xml supposed to match up to TokenKey="" attributes in my dynamic language .xml file? (That would be cool, since I've already described those tokens with regular expression patterns, etc.)

Basically, I just want to leverage the dynamic .xml file as much as possible, yet I do need a small amount of semantics defined, and I guess I just can't connect the last of the dots.

Thanks again,
Jeff

Posted 18 years ago by Actipro Software Support - Cleveland, OH, USA

Yes it sounds like you understand everything pretty well. Basically the semantic parser looks at the tokens that are generated by the lexical parser.

For a semantic parser implementation (using our parser generator at least), it is examining the token ID's of tokens defined by the lexical parser. Each individual keyword needs a unique token ID. Therefore the <Token> tag in the semantic grammar helps build a class containing token ID constants.

What you need is for your lexical parser to create tokens that have the same token IDs as those token IDs defined by the <Token> tag. For the Simple language, you can see how the programmatic lexical parser returns token IDs that come from the SimpleTokenID constants.

If you are using a dynamic language, you will have to assign the TokenID property in each pattern group. Additionally, if you are using one of our dynamic language samples, you will have to break up keyword groups into their own pattern group for each individual token so that they may be assigned a unique token ID.

I hope that helps explain it.

Actipro Software Support

Posted 18 years ago by Jeff Cohen

I think RegexPatternGroup can't handle the ^ character in the pattern value.

<RegexPatternGroup Key="StartBlockGroup" TokenKey="StartBlockToken" Style="BlockKeywordStyle" PatternValue="^\s*if\b" />

This should match lines that start with the word "if", optionally preceded with whitespace. (Ruby, like perl, can have "if" that does not start a line, but in that case it does not start a block; so I need to make sure that there's nothing before the "if" on the line).

The same regex works as expected in the VS2005 immediate window:

(I type:) System.Text.RegularExpressions.Regex.IsMatch(" if days", @"^\s*if\b");

(returns) true

but in the syntax editor, the "if" is not styled and the token assigned to it is the default token for the state.

Bug or oversight on my part?

Posted 18 years ago by Actipro Software Support - Cleveland, OH, USA

It will work fine but is probably a problem with the order of your patterns in your language defintion. Most likely you have a whitespace pattern matching above this pattern. So that is consuming the space. Then on the next character scan, it sees that we are no longer at the start of the line. So you need to move this pattern above the whitespace matching pattern.

You can generally debug these things by loading your language in our SDI editor sample. With that you can move the caret around and the statusbar tells you what token is under it. It displays an asterisk if the token is starting on the current character. That's extremely useful for seeing what the lexical parser is doing. In this case if you move the caret to the whitespace before "if", it will probably say WhitespaceToken or something similar.

Actipro Software Support

Posted 18 years ago by Jeff Cohen

Great, I'll give that a try.

Thanks!

Posted 18 years ago by Jeff Cohen

Purchased my developer license yesterday.

Thanks for the help.

Jeff

The latest build of this product (v24.1.1) was released 4 months ago, which was after the last post in this thread.

Add Comment

Please log in to a validated account to post comments.