Advice on in Tree Compile and Test Workflow for Benchmarks

I'm currently working on the benchmarking infrastructure. Mostly adding tests and refactoring to remove the accumulated technical debt before adding new features. I've been asked to structure my PRs as a set of small, incremental commits. It took it to mean: present an easy to follow step-by-step story of evolution to make the reviewer's job more pleasant (big monolithic code dumps are hard to assess).

Now I have a workflow productivity problem. I'm probably doing something wrong, by not yet fully grasping the effective use of Swift's build system.

I'm in the swift-source directory. I have set up the env-var shortcut to the build directory:

export SWIFT_BUILD_DIR=`pwd`/build/Ninja-ReleaseAssert/swift-macosx-x86_64

First I've done an initial full release build with benchmarks (only once after syncing with master):

$ swift/utils/build-script -R -B

… few hours later I have the latest release optimized llvm, clang and swift compilers that were used to build the benchmarks. Now I do my development in swift/benchmarks/. I compile my changes:

$ ninja -C ${SWIFT_BUILD_DIR} swift-benchmark-macosx-x86_64

and run the tests:

./llvm/utils/lit/lit.py -qsv ${SWIFT_BUILD_DIR}/test-macosx-x86_64/benchmark

When everything works out, I write a nice and descriptive message and commit the incremental change. :tada:

Erm… that's when all the stars align, but usually I have to repeat the compile and test cycle a few times. The incremental build during this phase usually takes a few seconds, up to a minute max. :ok_hand:

But after the commit, the swift repo moves to new revision, which triggers a recompilation of the swift compiler (its --version depends on the git revision). This is still an incremental build, but it takes about a 50 minutes on my ancient machine. :scream:

This workflow seems to be in line with @codafi's workflow advice, but it doesn't work well when making small incremental commits. The 50 minute recompilation delay is killing my flow.

For the work that I'm currently doing — which doesn't modify the compiler or stdlib — it should suffice to build the benchmarks with a different swiftc, not the one built from the same tree. Recent dev snapshot of swift should suffice. I've looked at @Michael_Gottesman's updated benchmark readme, which seems to point towards a solution in the Building standalone with cmake section. But I wasn't able to follow the instructions to successful solution to my problem… (My current limited understanding is that cmake would generate a bunch of build scripts that would be later executed by ninja? I'm unclear on how to pull this all together. Build option flags, ninja targets — all a mystery to me… :exploding_head:)

What should I do to rebuild benchmarks with default system compiler and maintain the ability to run tests against that build? For extra bonus points, I'd like to build just the Benchmark_O (skipping Onone and Osize builds) to get to test even sooner.

Terms of Service

Privacy Policy

Cookie Policy