I'm currently working on the benchmarking infrastructure. Mostly adding tests and refactoring to remove the accumulated technical debt before adding new features. I've been asked to structure my PRs as a set of small, incremental commits. It took it to mean: present an easy to follow step-by-step story of evolution to make the reviewer's job more pleasant (big monolithic code dumps are hard to assess).
Now I have a workflow productivity problem. I'm probably doing something wrong, by not yet fully grasping the effective use of Swift's build system.
I'm in the swift-source directory. I have set up the env-var shortcut to the build directory:
export SWIFT_BUILD_DIR=`pwd`/build/Ninja-ReleaseAssert/swift-macosx-x86_64
First I've done an initial full release build with benchmarks (only once after syncing with master):
$ swift/utils/build-script -R -B
… few hours later I have the latest release optimized llvm, clang and swift compilers that were used to build the benchmarks. Now I do my development in swift/benchmarks/. I compile my changes:
$ ninja -C ${SWIFT_BUILD_DIR} swift-benchmark-macosx-x86_64
and run the tests:
./llvm/utils/lit/lit.py -qsv ${SWIFT_BUILD_DIR}/test-macosx-x86_64/benchmark
When everything works out, I write a nice and descriptive message and commit the incremental change. 
Erm… that's when all the stars align, but usually I have to repeat the compile and test cycle a few times. The incremental build during this phase usually takes a few seconds, up to a minute max. 
But after the commit, the swift repo moves to new revision, which triggers a recompilation of the swift compiler (its --version depends on the git revision). This is still an incremental build, but it takes about a 50 minutes on my ancient machine. 
This workflow seems to be in line with @codafi's workflow advice, but it doesn't work well when making small incremental commits. The 50 minute recompilation delay is killing my flow.
For the work that I'm currently doing — which doesn't modify the compiler or stdlib — it should suffice to build the benchmarks with a different swiftc, not the one built from the same tree. Recent dev snapshot of swift should suffice. I've looked at @Michael_Gottesman's updated benchmark readme, which seems to point towards a solution in the Building standalone with cmake section. But I wasn't able to follow the instructions to successful solution to my problem… (My current limited understanding is that cmake would generate a bunch of build scripts that would be later executed by ninja? I'm unclear on how to pull this all together. Build option flags, ninja targets — all a mystery to me… 
)
What should I do to rebuild benchmarks with default system compiler and maintain the ability to run tests against that build? For extra bonus points, I'd like to build just the Benchmark_O (skipping Onone and Osize builds) to get to test even sooner.