Yep -- based on your description I'm almost ceretain it is the dueling references issue. The delay it occurs only happens when you start a build -- which is why you aren't seeing when you make changes while under the debugger.
Basically, the .refresh files within your \bin directory contain a pointer path to an assembly to dynamically refresh in the project anytime it changes. As part of this refresh process, it will also update and bring over other assemblies used by that target assembly if it has dependencies on it. For example, if you have a .refresh file pointing to AssembyA.dll which has a dependency on AssemblyB.dll, both will get copied over into your \bin dircectoy whenever one changes.
What can happen is that you might have two or more .refresh files pointing to different assemblies -- and that some of those assemblies are built and have dependencies on different versions of files.
For example: You have two .refresh files -- one pointing to assemblya.dll and one pointing to assemblyc.dll. Both have a dependency on assemblyb.dll, but the assemblyb.dll in their local directories are different versions. As such, you can end up "refreshing" and copying it over, and doing a clean build, everytime you do a build or hit F5. This will slow your build time down incredibly.
There are a couple of ways to fix it:
1) Make sure any shared assemblies that you have references to are in sync and don't have version mis-matches. This is the ideal approach and would be what I'd recommend unless there is some technical reason why they can't be versioned in sync.
2) Delete the .refresh files in your \bin directory. The .refresh files are only needed if you want to dynamically update the assemblies on changes. If you want to manually update an assembly, you can just delete its .refresh file (but keep the .dll).
If you can't fix the cirtuclar reference in #1 and/or don't want to loose the dynamic update capability by doing #2, you can kind of cheat the build system by creating a new utility library as a class project. Have this utility class project reference these dueling assemblies, and then reference the utility project from your web-project. This will then cause all of the latest assemblies you need to be copied into the web project -- and avoid having the duplicate cause a reset each time.
Let me know if this works for you or if I can help more.
Thanks,
Scott