Making a ToolWindow modal essentially defeats the concept of making it a ToolWindow. Since a Modal dialog blocks the owner, there would be no way to interact with the rest of your application while the modal window was opened.
Perhaps you have another goal in mind other than making it modal. Do you just want to close the ToolWindow when the user clicks off of it? If so, this is different than being 'Modal'. I haven't tried this myself, so I'm not certain of the best way to achieve it. My guess would be to look at the events on the DockManager. There are several events for when the focused ToolWindow changes. If the "Focused" ToolWindow is not the one you're wanting to close, that means it no longer has focus and you can close it.
