In general, the lexer is what tokenizes text and then those tokens end up determining what highlighting style gets applied. It seems like you want to apply alternate highlighting styles based on the parentheses' nesting level.
One thing you could try is if you are using a dynamic lexer, have a default lexical state that has a child lexical state whose scope start/stop tokens are parentheses. Apply a base highlighting style to those parenthesis. Then have that child lexical state match all of the same lexical patterns as in the default lexical state and have it also have another child lexical state whose scope start/stop tokens are again parenthesis. But for this set of parenthesis, assign alternate tokens and an alternate highlighting style to them. You could also even have the child lexcial state of that scope be the original default lexical state. That way the parenthesis would appear to alternate colors based on nesting level.