Hi Tim,
Thanks for reporting this. From some further testing, it appears to be a general issue with WPF since I can repro the same thing if I float a tool window and then open the Open dialog. In that case I'm allowed to click on the floated tool window as well. Both the floated tool window scenario and auto-hide flyouts (when non-hosted as in your example) are using modeless Windows that are "owned" by the root Window. Thus this should be easily reproducable without using our controls at all.
I'm not sure there's really anything we can do since it's more core WPF related. For your particular scenario, you might want to manually deactivate any open flyouts when you open dialogs, such as with this code:
dockSite.AutoHideHost.Deactivate(true);