It seems like Xcode 15.1 might have made some improvements here 
In my project, it seems now that my incremental builds take 30s with SWIFT_USE_INTEGRATED_DRIVER: NO and 36s without this build setting. So still a +6s regression, but that's much better than the +52s that I had been seeing before.
I heard from another engineer that the regression is gone entirely too, so worth trying again once you have upgraded to Xcode 15.1.
Just started using Xcode 16 RC to build our apps, and the Verify Modularization step seems to have regressed 10x or more. From 3-7 seconds per framework (we have 8 frameworks) to 45-75 seconds, each!
This comes from a regression in compiling for objective-c++ modules.
To workaround it for now, you could disable the module verifier with the Xcode build setting
ENABLE_MODULE_VERIFIER = NO // Can toggle between release and debug
or just disable verifying with objective-c++
MODULE_VERIFIER_SUPPORTED_LANGUAGES = objective-c
Fantastic, thank you! We have disabled it for debug builds for now.