Locally benchmarking changes to Swift

I’m attempting to make some optimizations to the Swift standard library that should enable more elision of bounds-checking. In the interest of not wasting everyone’s time, I’ve been trying to run benchmark comparisons on these changes before submitting a pull request.

Ideally, I’d like to replicate the @swift-ci's smoke benchmark” preset exactly. Unfortunately, I couldn’t figure out which preset that actually is, so I’ve defaulted to using utils/build-script -RB.

This has proven less than helpful, as the benchmark inexplicably shows (consistent!) improvements and regressions from the main branch even when no changes have been made. That is, running the benchmark on the main branch and simply running it again afterwards consistently shows the same “changes”.

I’m obviously doing something wrong here, but I have no idea what it could be. Would anyone care to shed some light on the issue? I can provide more information if needed.

I’ve made a draft pull request with some of the changes, but I’m still not sure how to proceed.

The CI output seems to imply that almost all of the changes are noise, but that still doesn’t explain why I’m seeing consistent changes on local benchmarks even without doing anything.

@Erik_Eckstein You’re listed as a point of contact in that output. Do you think you could shed some light on the matter?

Terms of Service

Privacy Policy

Cookie Policy