Both the TabbedMdiContainer and ToolWindowContainer controls use an AdvancedTabControl in their templates. AdvancedTabControl is a control that provides all the tabbed UI, including handling overflow. That control has an IsOverflowed property that returns true if there are tabs that are overflowed. AdvancedTabControl has the TabOverflowBehavior property defined, which both TabbedMdiContainer and ToolWindowContainer set with their related properties. AdvancedTabControl also has a AdvancedTabPanel that is used in its ItemsPanelTemplate, and that reads the TabOverflowBehavior, and adjusts the tab sizing appropriately.
To customize the tab layout behavior, you'd probably effectively want to update the ItemsPanel property to a custom Panel for the AdvancedTabControl used in the containers. You might be able to do that with an implicit Style that targets AdvancedTabControl and sets the ItemsPanel property with an ItemsPanelTemplate that contains your special Panel. Please note that I believe your custom Panel might need to inherit our AdvancedTabPanel control.