I have ran into the same problem.
In our scripting language, we can create macros
Define Macro mac1 ( id, id, ... );
// Macro content
EndMacro;
In the dynamic language definition I used 'Macro' to move from Default state to Macro state and used ';' to move from Macro state to MacroBody state. To get out of them I used 'EndMacro' to go from MacroBody to Macro and used ';' to go from Macro to Default.
<States>
<State Key="CommentState" TokenKey="CommentDefaultToken" Style="CommentDefaultStyle">
<Scopes>
<Scope BracketHighlight="True">
<RegexPatternGroup Type="StartScope" TokenKey="CommentStartToken" Style="CommentDelimiterStyle" PatternValue="//(?!((Block)|(EndBlock))" />
<RegexPatternGroup Type="EndScope" TokenKey="CommentEndToken" Style="CommentDelimiterStyle" PatternValue="{LineTerminatorMacro}" IsWhitespace="True" />
</Scope>
</Scopes>
<PatternGroups>
<ExplicitPatternGroup TokenKey="CommentDelimiterToken" Style="CommentDelimiterStyle" PatternValue="//" />
<RegexPatternGroup TokenKey="CommentWhitespaceToken" PatternValue="{WhitespaceMacro}+" IsWhitespace="True" />
<RegexPatternGroup TokenKey="CommentLineTerminatorToken" PatternValue="{LineTerminatorMacro}" LookAhead="{LineTerminatorWhitespaceMacro}* //(?!((Block)|(EndBlock)) [^/]" IsWhitespace="True" />
<RegexPatternGroup TokenKey="CommentWordToken" PatternValue="\w+" />
<RegexPatternGroup TokenKey="CommentDefaultToken" PatternValue="{NonLineTerminatorMacro}" />
</PatternGroups>
</State>
<State Key="MacroState">
<Scopes>
<Scope BracketHighlight="True">
<ExplicitPatternGroup Type="StartScope" TokenKey="MacroStartToken" Style="CommandKeyWord" PatternValue="Macro" LookBehind="^|[^\.]" LookAhead="{NonWordMacro}" CaseSensitivity="Sensitive" />
<ExplicitPatternGroup Type="EndScope" TokenKey="MacroEndToken" Style="OperatorStyle" PatternValue=";" />
</Scope>
</Scopes>
<PatternGroups>
<ExplicitPatternGroup Key="Macro" TokenKey="Macro" Style="CommandKeyWord" PatternValue="Macro" LookBehind="^|[^\.]" LookAhead="{NonWordMacro}" CaseSensitivity="Sensitive" />
<ExplicitPatternGroup Key="EndMacro" TokenKey="EndMacro" Style="CommandKeyWord" PatternValue="EndMacro" LookBehind="^|[^\.]" LookAhead="{NonWordMacro}" CaseSensitivity="Sensitive" />
<RegexPatternGroup TokenKey="MacroLineTerminatorToken" PatternValue="{LineTerminatorMacro}" LookAhead="{LineTerminatorWhitespaceMacro}*" IsWhitespace="True" />
<RegexPatternGroup TokenKey="MacroWhitespaceToken" PatternValue="{WhitespaceMacro}+" IsWhitespace="True" />
<RegexPatternGroup TokenKey="MacroIdentifierToken" PatternValue="(_ | {AlphaMacro})({WordMacro})*" />
<ExplicitPatternGroup TokenKey="MacroOpenParenthesisToken" PatternValue="(" EndBracket="CloseParenthesisPatternGroup" />
<ExplicitPatternGroup TokenKey="MacroCloseParenthesisToken" PatternValue=")" StartBracket="OpenParenthesisPatternGroup" />
</PatternGroups>
<ChildStates>
<ChildState Key="CommentState" />
<ChildState Key="MacroBodyState" />
</ChildStates>
</State>
<State Key="MacroBodyState">
<Scopes>
<Scope BracketHighlight="True">
<ExplicitPatternGroup Type="StartScope" TokenKey="MacroBodyStartToken" Style="OperatorStyle" PatternValue=";" />
<ExplicitPatternGroup Type="EndScope" TokenKey="MacroBodyEndToken" Style="CommandKeyWord" PatternValue="EndMacro" LookBehind="^|[^\.]" LookAhead="{NonWordMacro}" CaseSensitivity="Sensitive" />
</Scope>
</Scopes>
<PatternGroups>
<ExplicitPatternGroup Key="EndMacro" TokenKey="EndMacro" Style="CommandKeyWord" PatternValue="EndMacro" LookBehind="^|[^\.]" LookAhead="{NonWordMacro}" CaseSensitivity="Sensitive" />
<RegexPatternGroup TokenKey="MacroBodyWhitespaceToken" Style="StringDefaultStyle" PatternValue="{WhitespaceMacro}+" IsWhitespace="True" />
<RegexPatternGroup TokenKey="MacroBodyLineTerminatorToken" Style="StringDefaultStyle" PatternValue="{LineTerminatorMacro}" LookAhead="{LineTerminatorWhitespaceMacro}* //(?!((Block)|(EndBlock)) [^/]" IsWhitespace="True" />
<RegexPatternGroup TokenKey="MacroBodyWordToken" Style="StringDefaultStyle" PatternValue="\w+" />
<RegexPatternGroup TokenKey="MacroBodyDefaultToken" Style="StringDefaultStyle" PatternValue="{NonLineTerminatorMacro}" />
</PatternGroups>
<ChildStates>
<ChildState Key="CommentState" />
</ChildStates>
</State>
I cannot get out of MacroBody state because I wanted to match any token that has the pattern "[.\w\s]+" within the MacroBody of which 'EndMacro' is one of them. So I think I would need to somehow transition out of this by looking at the tokens in semantic parsing.
So my question is: what can I do to tell the dynamic language to change state within semantic parser code?
