There are two ways you could do that... one would be to save the layout and load a new one whenever the mode changes.
The other would be to keep your own boolean array for each tool window. When the mode changes, save whether each tool window is visible and then read the settings for the new mode. If they differ, then either Activate or Close the tool window as appropriate.
The first way is a little more robust since it will also store the location of the tool windows, and is probably much easier to implement.