I have looked at the constraint system debug output between 5.4 and 5.5 and they are identical so it might be that cost of some operations increased somewhat if no assertions build still shows the difference.
1 Like
Current build off of main:
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -typecheck a.swift
Time (mean ± σ): 122.8 ms ± 2.1 ms [User: 78.2 ms, System: 39.8 ms]
Range (min … max): 120.5 ms … 129.1 ms 23 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -typecheck b.swift
Time (mean ± σ): 173.6 ms ± 2.9 ms [User: 128.1 ms, System: 40.8 ms]
Range (min … max): 170.9 ms … 183.4 ms 17 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -typecheck c.swift
Time (mean ± σ): 627.7 ms ± 5.9 ms [User: 567.0 ms, System: 55.7 ms]
Range (min … max): 614.8 ms … 635.4 ms 10 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -typecheck d.swift
Time (mean ± σ): 161.6 ms ± 2.1 ms [User: 115.4 ms, System: 41.4 ms]
Range (min … max): 159.4 ms … 167.6 ms 18 runs
Not much improvement by removing assertions, aside from c. Actually looks like a further regression from Xcode 13b1.
On a hunch I tried disabling the new driver:
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck a.swift
Time (mean ± σ): 64.3 ms ± 1.0 ms [User: 49.1 ms, System: 13.4 ms]
Range (min … max): 62.4 ms … 67.3 ms 45 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck b.swift
Time (mean ± σ): 114.9 ms ± 1.4 ms [User: 98.5 ms, System: 14.5 ms]
Range (min … max): 112.7 ms … 119.4 ms 25 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck c.swift
Time (mean ± σ): 568.7 ms ± 5.5 ms [User: 537.1 ms, System: 29.6 ms]
Range (min … max): 560.5 ms … 578.1 ms 10 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck d.swift
Time (mean ± σ): 104.2 ms ± 1.6 ms [User: 87.3 ms, System: 15.0 ms]
Range (min … max): 101.1 ms … 107.9 ms 28 runs
Much improved but still a regression from Xcode 12.5.
Also benchmarked just parsing the files and all of them were around 21ms on my machine, with a slight improvement from 12.5 to the latest main.
1 Like
Ouch, filed [SR-14794] ~60ms startup overhead on using new swift-driver · Issue #57143 · apple/swift · GitHub (~60ms startup overhead on using new swift-driver). The 10%-15% percent difference from Xcode 12.5 could very well be down to LTO (normal development toolchains don't use LTO but toolchains in Xcode do), I wouldn't say that's clearly a regression.
I suspect the increase in system time with the new driver is because there are 2 extra calls to exec when it's enabled:
-
swiftandswiftcstill launchswift-frontend, which then immediatelyexecsswift-driverunless the new driver is disabled. Now that it's on by default we should optimize this for the new driver instead of the old one. -
When the driver encounters an input like this that only requires one frontend job, it will run it in-place. With the old driver, we could call directly into the frontend entry point, but the new driver needs a second call to exec to launch
swift-frontendagain.
The new driver also calls swift-frontend -print-target-info during planning which could contribute. It's all fixed overhead so it's probably not noticeable in larger compiles, but I could see it impacting JIT/REPL startup times in addition to less-used actions like -typecheck
The overhead of spawning a process ought to be around 5ms-10ms on macOS so I think 50ms-60ms is still too high to be solely down to spawning 2-3 additional processes.
Only difference in behavior in c: is related to "static member reference in generic contexts" feature - it requires one more inference step once result of the chain is bound, but I don't think that would be anywhere close to a 100 ms. I wonder whether it's related to https://github.com/apple/swift/blob/main/lib/Sema/CSSimplify.cpp#L1466-L1489 - could you try removing this block of code and see whether it makes any difference?
No difference after I commented out that block.
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck a.swift
Time (mean ± σ): 64.3 ms ± 0.8 ms [User: 49.2 ms, System: 12.2 ms]
Range (min … max): 62.1 ms … 66.0 ms 44 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck b.swift
Time (mean ± σ): 115.8 ms ± 1.6 ms [User: 99.5 ms, System: 13.1 ms]
Range (min … max): 112.9 ms … 119.0 ms 25 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck c.swift
Time (mean ± σ): 570.0 ms ± 4.0 ms [User: 538.5 ms, System: 28.2 ms]
Range (min … max): 565.2 ms … 579.1 ms 10 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck d.swift
Time (mean ± σ): 104.1 ms ± 1.1 ms [User: 87.4 ms, System: 13.6 ms]
Range (min … max): 101.8 ms … 106.1 ms 28 runs
it seems in all the tests this is the fastest ?
if so, my intution says this is because swift doesnt need to actually type check the left vs right declaration?
since right hand side is obviously a string and the left is untyped, there is no further comparison to do...?
In this case, you're right that there are fewer comparisons to do. If there was a type annotation on the left side, the compiler would indeed need to make sure that the type inferred on the right is convertible to the type in the type annotation.
Leaving out the type annotation isn't always faster, though. For example, when there are overloads involved, a type annotation can help the compiler find the correct overload faster, which allows it to prune subsequent overloads that obviously won't work and avoid extra checking.
5 Likes
Hm, it would be interesting to look at the flamegraphs for these runs, it might shed some light on what is going on here...
1 Like
I can capture an Instruments run, just let me know how to do so with this hyperfine command. Or if you know how to capture actual flamegraphs on macOS, as most of what I can find is for Linux.
If I remember correctly there is a way to specify arguments to swiftc in Instruments and run it from there.
I've figured that out, but my optimized release build has no symbols. What build flag do I need for a release build with symbols?
-r instead of -R
1 Like
Before that goes, do you want that section of code you mention included or not?
Sure, let's include it to keep changes to the minimum.
Instruments doesn't seem to like symbolicating these runs very much. Many times it never symbolicates. When I try to find the symbols (where are they?) Instruments eventually hangs. The trace I shared doesn't really make sense to me so I'm not sure it's symbolicating correctly even when it works.
Yeah, it doesn't really make much sense to me either. Maybe try it with compiler from beta and more runs?
Upping the lines in the Swift files that are parsed to 5000 gives me these results:
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck a.swift
Time (mean ± σ): 224.5 ms ± 2.8 ms [User: 200.9 ms, System: 20.7 ms]
Range (min … max): 218.6 ms … 229.6 ms 13 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck b.swift
Time (mean ± σ): 1.148 s ± 0.010 s [User: 1.113 s, System: 0.032 s]
Range (min … max): 1.131 s … 1.160 s 10 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck c.swift
Time (mean ± σ): 3.089 s ± 0.016 s [User: 2.986 s, System: 0.100 s]
Range (min … max): 3.061 s … 3.117 s 10 runs
Benchmark #1: /Users/jshier/Desktop/Code/swift/build/Ninja-Release/swift-macosx-x86_64/bin/swiftc -disallow-use-new-driver -typecheck d.swift
Time (mean ± σ): 765.4 ms ± 3.6 ms [User: 731.2 ms, System: 31.1 ms]
Range (min … max): 759.9 ms … 771.1 ms 10 runs
Looks like the discrepancies grow with more work. I'll see if I can get a good trace.