After @dan-zheng's post on informative assertions, I made some measurements on the overhead of assertions and shared them in a comment. Turns out, it's not really a good idea to make measurements without understanding what you are measuring.
So I've fixed a bug or two and started making more systematic measurements.
Configuration
- 2017 10-core iMac Pro. Swift and LLVM built with Clang in Xcode 12.
- Almost top-of-tree swiftc & LLVM (swift/main)
- Package versions are from mid-November ~ early December.
- Alamofire @ 9e0328127dfb801cefe8ac53a13c0c90a7770448
- SwiftNIO @ 076fda15213a9cc1da26b1e3467f1daba2407391
- swift-syntax @ e28671a7650bd54cc381dc21d75e214685f2ac48
There are three main compiler build configurations:
- SwiftNoAssert_LLVMNoAssert - Compiled with
build-script --no-assertions.
- SwiftAssert_LLVMNoAssert - Compiled with
build-script --no-assertions --swift-assertions.
- SwiftAssert_LLVMAssert - Compiled with
build-script --no-assertions --swift-assertions --llvm-assertions.
(--swift-assertions only turns on assertions for the compiler, not the stdlib.)
Turning off Swift's assertions also turns off the SIL verifier. Because of this, the first configuration only has a _NoSILVerify variant, whereas the second and third configurations have an additional _NoSILVerify variant, which is exercised by passing -Xfrontend -sil-verify-none. So there are 5 variants in total. For each variant, we compile SwiftPM packages in debug mode and release mode. (swift build -c debug or swift build -c release).
Everything else (apart from swiftc/LLVM) has assertions turned off due to --no-assertions. I suspect this might be the cause of the differences between the numbers here and in my comment on Dan's post.
Stats
We consider the SwiftNoAssert_LLVMNoAssert_NoSILVerify variant as the baseline, since it's the fastest. For the rest, I've reported times as +X% relative to the baseline, meaning that it takes time = (1 + X/100) * baseline_time.
| Package/Config |
SwiftAssert_LLVMNoAssert_NoSILVerify |
SwiftAssert_LLVMAssert_NoSILVerify |
SwiftAssert_LLVMNoAssert |
SwiftAssert_LLVMAssert |
| Alamofire/debug |
+10% |
+15% |
+10% |
+15% |
| SwiftNIO/debug |
+12% |
+16% |
+18% |
+23% |
| SwiftSyntax/debug |
+14% |
+18% |
+21% |
+25% |
| Alamofire/release |
+13% |
+22% |
+25% |
+32% |
| SwiftNIO/release |
+16% |
+23% |
+31% |
+38% |
| SwiftSyntax/release |
+13% |
+16% |
+26% |
+29% |
To summarize the numbers:
- Based on the values in column 1: the overhead of enabling assertions for Swift-only is about +10%~15% to compile times.
- Based on the differences
column 3 - column 1 and column 4 - column 2: the overhead of enabling the SIL verifier has some variation: it can be negligible, or it can be like +7%, or it can be as high as +15%. I don't know much about the verifier to be able to judge whether this much variation is expected or not.
- Based on the differences
column 2 - column 1 and column 4 - column 3: the overhead of enabling assertions in LLVM-only is roughly +4%~7% to compile times.
Details
More detailed stats are available here: Swiftc assertions overhead · GitHub
I made the measurements using a couple of small shell scripts:
#!/usr/bin/env bash
# run.sh
for package in "Alamofire" "swift-nio" "swift-syntax"; do
for buildType in "debug" "release"; do
hyperfine \
--prepare "rm -rf $package/.build" \
--warmup 1 \
--runs 15 \
--parameter-list compilerconfig 'SwiftNoAssert_LLVMNoAssert_NoSILVerify,SwiftAssert_LLVMNoAssert_NoSILVerify,SwiftAssert_LLVMNoAssert,SwiftAssert_LLVMAssert_NoSILVerify,SwiftAssert_LLVMAssert' \
--style full \
"./compile_package.sh $package $buildType {compilerconfig}"
done
done
#!/usr/bin/env bash
# compile_package.sh
echo "-c $2 $([[ $3 == *"_NoSILVerify" ]] && echo "-Xswiftc -Xfrontend -Xswiftc -sil-verify-none")" | xargs xcrun /Users/varun/foss-swift-alt/build/$(echo $3 | sed 's|_NoSILVerify||')/toolchain-macosx-x86_64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift build --package-path $1
