Is an XML language definition file absolutely necessary?

You probably want to write your own SyntaxLanguage.
I would recommend writing your own lexer too since its much faster. But you even can use XML file to define the lexer and specify the languge with the SyntaxLanguage Tag.
Using the XML file for the lexer is a good start, because its very easy, but in long-term its best to write your own classes. Not only is it much faster, you are also more flexible.

There is no way to load languages from a different format, but you can do everything you want to, if you implement it in code.
Look at the SimpleLanguage Add-On to get the basic idea how to implement your own language and lexer.

Maybe you should read the "Language Defintion Guide" and "Dynamic Languages Guide" in the docu. Especially the first chapters give you a good hint, how the system from lexer to parser and language works.

Thank you. I shall look at the possibility of making my own lexer, though it sounds rather scary. I am still struggling to get my head around .NET after coming from VB6.

Well, you don't make a code-editor within a week.
That's a complex task, especially if you wan't a good product.

To be honest, I don't think it's a one-man-job either, especially if you struggle with .net.
I have never really used VB, but I find it absolutely unreadable if you are a C# developer. And if I can't really read it, I can't really work with it...

You will need some time to learn all corners of the SyntaxEditor. I started evaluating more than a year ago and by now our first product using it has been released. The second one will come out soon.

Hi David,

Your two options with lexing is to use a dynamic language or a more basic language where you write the lexical parser code by hand.  It sounds like you are trying to stick with the dynamic language lexer.

The good news is that you can easily make your base dynamic language XML definition that defines all your common pieces of the lexical grammar.  It's really not hard to clone one of our sample XML definitions and modify it for your needs.  Then leave some empty stub lexical pattern groups in there for where you'll dynamically put in your instruction set keywords.

At run-time in code you can programmatically modify the dynamic language.  If you look at the DynamicallyCreateForm code in our SdiEditor demo, you can see an instance of that.  Just wrap the language updates with IsUpdating = true and IsUpdating = false calls.  But in between is where you'd dynamically pull in your instruction set keywords into the pattern groups you defined in the XML definition.

As for your questions, the XML reader probably won't like unknown tags in it.  You could however use XML comments anywhere you want.

Only one dynamic language XML definition can be loaded at a time unless you are doing a merged language scenario.  The HTML language sample does that where it transitions to other languages like CSS and back.  You need to have well-defined delimiters to know when to transition and come back.

The only other comment I have is that if you are loading hundreds of keywords for your instruction set, you'll have to see how the performance ends up being.  Dynamic language lexers are better at handling regex-based patterns and not hundreds of keywords.  So if you notice performance issues, you may have to switch to making a programmatic lexical parser.  Our Simple language shows an example of that.