If your incremental (i.e. Debug) builds seem to be too slow, here some things to try (in no particular order):
In Xcode, go into the Report Navigator and examine your build log to see what's going on and where the time might be going.
See if some stage of your build is always replacing an imported file,
In "Other Swift Flags", add
-driver-show-job-lifecycle. Then expand the build log portions of the "Compile Swift Source Files" build log entry. These options will cause the compiler to emit "remarks" explaining why the compiler decided to compile files.
If you are making lots of changes to top-level types, the incremental system won't be able to be as selective as possible. If you wrap one or more top-level types in an enclosing
extension, the compiler can be more selective.