I just ran a round of a package's benchmarks with 6.4.x-snapshot which is installed via swiftly.
The default Swift version is Swift 6.3 (swiftly again).
This is on Ubuntu 24.04.
The results are mixed, more on the side of "things got worse". How seriously should I take these results?
Do the snapshots have some caveats that can affect release builds and thus benchmarks (e.g. "assert builds enabled")? Or no, these must be plain regressions in LLVM/Swift?
If yes, what can I do short of recompiling Swift, to be able to compare Swift 6.4 to 6.3, before 6.4 is released?
Can’t answer the question of the compiler snapshot, but good to see you focus on instructions (I’d not use wall clock for something like this running on public runners really) - some of the deltas looked significant - I’d probably compile the largest ones first with assemblyvision, if no obvious diff, look at SIL or assembly — an LLM can be helpful there for a quick analysis really.
It's not a public runner! It's a dedicated-core VM but long-story short it has been proven to me on different occasions that it's very very close to stability of a bare metal.
Ok so digging into @_assemblyVision, I can see I already had "liked" this post from 2021, so apparently I had noticed it, but practically no, it wasn't anywhere in my memory.
It's pretty nice and helpful. It's more like "SIL vision" than "assembly vision" but I guess they meant "vision into assembly of the code" and didn't mean to reference the assembly language?!
Usually for this I'd just look at the assembly (swiftc -emit-assembly) or the SIL (swiftc -emit-sil) but @_assemblyVision is a pretty nice for ease of use. I definitely prefer it over reading SIL output. (I have put together some mini-setup for looking into compiler and LLVM decisions, and like you mentioned, LLMs are pretty helpful to speed up the work.)
I still haven't gone for checking 6.3 vs 6.4 compiler differences though, and even then @_assemblyVision is not guaranteed to be useful as a lot of the code is optimized at assembly level so I'll have to look into LLVM decisions as well. Essentially it can get cumbersome, but I'll task Claude with it and see if it can come into non-bogus conclusions (It should do fine I'd say).
would still analyze the output and see what is the change in codegen.