It doesn't seem like many devs are using the Swift snapshot toolchains to provide feedback, would be good to hear why.
Some informed speculation as to the reasons:
The snapshot toolchains are too hard to use and set up, should no longer be the case with the official github workflows, but may still be for other reasons, eg they have assertions enabled so they flag too much stuff?
The places to report such info, ie github issues and rdar, are not being listened to enough? Github is certainly a mess, with 1,129 of the 7,476 open compiler issues currently untriaged.
Too much effort needed to minimize and extract the problem? Other languages provide automated tooling to help with this bug minimization from source code, perhaps Swift needs such a tool.
Please share why you think more such early testing of Swift snapshot toolchains is not taking place. Obviously some of us are, the question is how we get more devs to do so.
I had no idea of their existence before seeing this thread. If you look on the download page, the explanation is below the fold and it basically just says "these exist", which kind of comes across as "tired of your compiler working? Want a less reliable experience? Try main-snapshot!"*
There's no call to action on why I might want to install an untested automated build. If it's to find bugs, where should they be reported? Swift has a lot of repositories, and outsiders don't know where something belongs and will be put off because of that.
(* It's also unclear to me if main-snapshot is floating -- will it keep being updated? How do I do so?)
the bottleneck for me is minimizing and extracting the crash, even though it’s usually straightforward. i could probably add that to my CI but without the minimal reproducer it’s unlikely a compiler engineer will pick it up. on the flip side, if you do have the reproducer on hand, the turnaround is generally great, it will almost always be fixed by the next release.
as someone with a lot of libraries where i could turn this on, it does feel like there’s an opportunity for outside contributors to pick up compiler crashes and reproduce them, but if a nightly crashes in the forest and no one is around to reproduce it, it’s just not really worth the effort to set up the infrastructure on the libraries needed to probe the forest in the first place.
Compiling your working codebase with a snapshot toolchain to see if the snapshot toolchain broke something that was previously working.
Encountering a compiler bug/crash on the release toolchain and using a snapshot toolchain to verify the bug is still present before filing an issue.
Personally, I'd probably never do (1) because I'd assume (perhaps naively) the snapshot toolchain only temporarily broke the thing I'm observing and it will be fixed for the final release. (Edit: also, as a Swift user, I don't expect existing features/code to break, so regularly building my projects with a prerelease compiler doesn't seem worthwhile.)
As for (2), I do test on the latest snapshot sometimes but not always before filing an issue, depending on how much time I want to invest or the quality of my current internet connection (toolchains are big). Installing the latest snapshot is not a regular thing for me, so it always involves some additional effort.
Also, filing issues on the swift github is a million times better than filing Feedbacks with Apple, because it's not actively hostile to developers and most issues do eventually get looked at, but it's still not a great experience. I have open issues that are still "triage needed" 3 years later. I do it, but I spend a bunch of time wondering whether it'll be worthwhile "this time" or not. (and often I find that issues I report are fixed "incidentally" by someone working on something other than deliberately fixing my issue; sometimes they don't even know about my issue)
It just doesn't make sense from business POV... there is always a dozen "must be done yesterday" things and another dozen of "nice to have things" that we never have time for... testing with toolchains would be somewhere at the bottom of that priority list.
I would happily test the toolchains if they worked for iOS and compatible platforms, as that’s where all my business code lives, but they’ve haven’t worked there in about a decade.
Also, the development and early version branches are shockingly buggy, especially when it comes to assertions, and I have no desire to be manual QA for a multi-trillion dollar company, aside from the feature I’m there to test.
it seems there is a lot of sentiment in this thread that can basically be summarized as “why should i do these Apple employees’ jobs for them?” so i will just say that i want to see these crashes fixed and any concerns about moral hazard or enabling employees of multi-trillion dollar companies to slack off are secondary to me. it really just feels like a circle of bystanders pointing fingers instead of doing anything to solve the problem.
so i just want to reiterate that i have dozens of packages with hundreds of thousands of lines of modern Swift code that i maintain and thus i am well-positioned to flush out a cacophony of likely compiler crashes, if only i could get some help reducing those crashes to get the minimal reproducers. my point is the CI and “testing” side of the equation is solvable, it really just needs some volunteers to turn those targets into swiftlang/swift bugs.
Well, it depends on your POV - we use a wide range of the feature set of the language (on two platforms) and invariably have been blocked adopting new toolchain major versions due to problems - so we try to test and provide bug reports on the new toolchains to proactively try to get these things sorted before the release (sometimes successful, sometimes not...), but it a fairly small cost to have this running on CI to try to help fix those issues before the release is cut.
I'll share a contrary view point: because the snapshot toolchains do not include the necessary/desired content.
As a concrete example, DS2 is still not part of the toolchain distributions on swift.org. Until recently, the releases did not include the NoAsserts toolchains. Android is fortunately now being included, but it was staged earlier. There are customisations that are possible to do that are not possible with the snapshot toolchains. Additionally, they are not at a rapid enough pace (there often are weeks without a snapshot).
While I would love to use the snapshot toolchains, until we are able to include more content and iterate more rapidly, it is more convenient to stick to custom toolchain distributions.
Reducers are a bonus, not required for reporting. A simple report saying that repo A at commit B causes crash C with toolchain D should be enough for engineers to start investigating. The reducer can come after. But I can't even get that far without support Apple's platforms, nor can I fully report bugs since my largest codebases are private.
One thing that might be helpful here is a bot of some kind that can take in crash reports and determine whether they may have been fixed in newer toolchains or in other branches. This is especially bad in Apple's toolchains, but even when testing say, the 6.4 prerelease toolchains, you'll run into issues that are already fixed on main. It would be good to know that without having to rely on the person who fixed the issue seeing your report.
I'm not sure how much it is in use these days, but there is/was Swift Source Compatibility | Swift.org. Which I would hope provides a reasonable about of signal.