After @dan-zheng's post on informative assertions, I made some measurements on the overhead of assertions and shared them in a comment. Turns out, it's not really a good idea to make measurements without understanding what you are measuring.
So I've fixed a bug or two and started making more systematic measurements.
Configuration
- 2017 10-core iMac Pro. Swift and LLVM built with Clang in Xcode 12.
- Almost top-of-tree swiftc & LLVM (swift/main)
- swiftc @ 5de95a596f6c81e7084a5e854985fdaca9e2829b (current HEAD for [build-script][CMake] Define and use SWIFT_ENABLE_ASSERTIONS pervasively. by varungandhi-apple · Pull Request #35115 · apple/swift · GitHub)
- llvm with ⚙ D93433 [IR] Use LLVM_ENABLE_ABI_BREAKING_CHECKS to guard ABI changes. cherry-picked on top of 051158df6ec66bd0796b65c1dc3ec364c7334737
- Package versions are from mid-November ~ early December.
- Alamofire @ 9e0328127dfb801cefe8ac53a13c0c90a7770448
- SwiftNIO @ 076fda15213a9cc1da26b1e3467f1daba2407391
- swift-syntax @ e28671a7650bd54cc381dc21d75e214685f2ac48
There are three main compiler build configurations:
- SwiftNoAssert_LLVMNoAssert - Compiled with
build-script --no-assertions. - SwiftAssert_LLVMNoAssert - Compiled with
build-script --no-assertions --swift-assertions. - SwiftAssert_LLVMAssert - Compiled with
build-script --no-assertions --swift-assertions --llvm-assertions.
(--swift-assertions only turns on assertions for the compiler, not the stdlib.)
Turning off Swift's assertions also turns off the SIL verifier. Because of this, the first configuration only has a _NoSILVerify variant, whereas the second and third configurations have an additional _NoSILVerify variant, which is exercised by passing -Xfrontend -sil-verify-none. So there are 5 variants in total. For each variant, we compile SwiftPM packages in debug mode and release mode. (swift build -c debug or swift build -c release).
Everything else (apart from swiftc/LLVM) has assertions turned off due to --no-assertions. I suspect this might be the cause of the differences between the numbers here and in my comment on Dan's post.
Stats
We consider the SwiftNoAssert_LLVMNoAssert_NoSILVerify variant as the baseline, since it's the fastest. For the rest, I've reported times as +X% relative to the baseline, meaning that it takes time = (1 + X/100) * baseline_time.
| Package/Config | SwiftAssert_LLVMNoAssert_NoSILVerify | SwiftAssert_LLVMAssert_NoSILVerify | SwiftAssert_LLVMNoAssert | SwiftAssert_LLVMAssert |
|---|---|---|---|---|
| Alamofire/debug | +10% | +15% | +10% | +15% |
| SwiftNIO/debug | +12% | +16% | +18% | +23% |
| SwiftSyntax/debug | +14% | +18% | +21% | +25% |
| Alamofire/release | +13% | +22% | +25% | +32% |
| SwiftNIO/release | +16% | +23% | +31% | +38% |
| SwiftSyntax/release | +13% | +16% | +26% | +29% |
To summarize the numbers:
- Based on the values in column 1: the overhead of enabling assertions for Swift-only is about +10%~15% to compile times.
- Based on the differences
column 3 - column 1andcolumn 4 - column 2: the overhead of enabling the SIL verifier has some variation: it can be negligible, or it can be like +7%, or it can be as high as +15%. I don't know much about the verifier to be able to judge whether this much variation is expected or not. - Based on the differences
column 2 - column 1andcolumn 4 - column 3: the overhead of enabling assertions in LLVM-only is roughly +4%~7% to compile times.
Details
More detailed stats are available here: Swiftc assertions overhead · GitHub
I made the measurements using a couple of small shell scripts:
#!/usr/bin/env bash
# run.sh
for package in "Alamofire" "swift-nio" "swift-syntax"; do
for buildType in "debug" "release"; do
hyperfine \
--prepare "rm -rf $package/.build" \
--warmup 1 \
--runs 15 \
--parameter-list compilerconfig 'SwiftNoAssert_LLVMNoAssert_NoSILVerify,SwiftAssert_LLVMNoAssert_NoSILVerify,SwiftAssert_LLVMNoAssert,SwiftAssert_LLVMAssert_NoSILVerify,SwiftAssert_LLVMAssert' \
--style full \
"./compile_package.sh $package $buildType {compilerconfig}"
done
done
#!/usr/bin/env bash
# compile_package.sh
echo "-c $2 $([[ $3 == *"_NoSILVerify" ]] && echo "-Xswiftc -Xfrontend -Xswiftc -sil-verify-none")" | xargs xcrun /Users/varun/foss-swift-alt/build/$(echo $3 | sed 's|_NoSILVerify||')/toolchain-macosx-x86_64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift build --package-path $1
