Probably the same or related issues to EnableSwiftBuildSystemIntegration slows down incremental builds - #15 by Jon_Shier. I suggest you report a bug to Apple. I haven't found any workarounds for my issue, but it doesn't affect my incremental builds much.