Unfortunately that is a behavior of .NET when trying to reflection load an assembly in the same appdomain where an assembly of the same name was previously loaded. .NET effectively throws:
FileLoadException' occurred in mscorlib.dll
API restriction: The assembly '' has already loaded from a different location. It cannot be loaded from a new location within the same appdomain.
It's an annoying restriction in .NET that many consider an unnecessary bug in the platform.
We have an item on our TODO list to possibly look into adding an option, where the assemblies would be loaded up into a separate AppDomain instead and the AppDomain immediately removed after the cache information is retrieved. We also could tie up with something like Cecil or Roslyn however those would require external dependencies that we're prefer to not have, so the first option is probably best. I'm not sure of a timeframe on that yet.