I spent some time trying to understand why the results from CI and the build-script are so different from my usual local builds. I was shocked to realize the stdlib assertions are enabled in our "standard" builds as follows:
-D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS
This is the default behavior when running build-script, and it's the behavior for CI tests using preset "buildbot_all_platforms,tools=RA,stdlib=RA".
This is unsettling because enabling stdlib asserts will mislead anyone working on Swift performance, particularly tuning stdlib performance, and will confuse anyone working at the SIL level looking at inlined stdlib code.
- Why does the build-script default to stdlib assertions?
It's only useful for verifying correctness, once, after changing stdlib algorithms. It's wrong for most purposes and potentially terrible because it goes undetected.
- How are swift compiler/stdlib developers supposed to build for benchmarking? And how are they supposed to know how to do that?
We're expecting developers to build benchmarks by default, run benchmarks against their patches, and make decisions based on the results, but we're telling them to build in a way that can't be used for benchmarking.
swift-ci benchmark is turning off all assertions, but its output doesn't show a top-level build-script invocation.
It isn't very useful showing PR benchmark results without providing a way to reproduce them. I'm not sure how anyone would know that there is a "right" way to build for benchmarking that's different from default Release builds.