SE-0346 and library evolution

It is also being pitched as a way to constrain protocol types, which there is similarly no other way of writing.

Constrained protocol types and constrained opaque result types are both true extensions of the language; there's no way to get their effects without cumbersome workarounds. But this cuts two ways if you need to support earlier tools versions. On the one hand, you can't enable them for clients using new tools (by declaring a primary associated type list) without finding a way to avoid breaking clients using old tools (by suppressing the primary associated type list). On the other hand, you probably can't use them yourself at all, because you have no way of getting their effect in code that needs to work for clients using old tools.

6 Likes

The fact that the community doesn't already take this approach should tell you you're missing some things here.

First and foremost, Apple doesn't support Swift on macOS for more than one year, meaning any move to adopt the latest version immediately drops thousands of developers, making it a non starter. It's generally unhealthy for the ecosystem to immediately drop support for large numbers of developers every year.

Secondly, Swift lacks, or has stopped using, tooling to enable automatic upgrades to new versions. This would make it easier to adopt new versions, both as library authors and library consumers. Even if I wanted to immediately jump to the new syntax, there's no way for me to mark up my library to make that transition automatic or even easy for my consumers. In some cases it's possible to keep older declarations using old syntax which can be deprecated, but it's not possible here or in many other cases. Lack of basic things like an included linter or formatter even preclude automated, whole scale transformations to new syntax versions.

So it's not so simple to require a new Swift version, at least not in a way that's healthy for the community.

7 Likes

Oh there is absolutely a balance to strike here, and as Swift grows ever more mature, I expect the window of support will expand ever more. But I think it's absolutely reasonable for package maintainers to stop providing new features to clients on toolchains below some particular cutoff point.

Note that bumping the required toolchain does not cut off older clients from using the package -- they are able to continue using the last release that supported their tools. Package maintainers can also continue to provide them with bug fixes. (SwiftPM handles this quite gracefully on its own -- it won't update to versions that it can't build.)

Where the support line is drawn is up to each individual package, as well as its intended audience. This varies wildly even amongst the packages I'm closely involved with -- e.g., I expect swift-atomics to be far more likely to need to bump its required toolchain than, say, swift-collections.

3 Likes

I'm not sure what you mean by upgrading to new syntax. Your package requiring a new tools release does not require your clients to adopt the newest language mode offered by that release and shouldn't necessitate widespread source upgrades. I know it's sometimes not that simple, between compiler bugs and (probably more importantly) platform SDK changes that are often picked up simultaneously, but those problems can't usually be addressed by an automatic upgrader anyway.

2 Likes

Adopting the SE-0346 syntax, the initial point of discussion in this thread.

This doesn't seem true if there isn't actually a language mode to go along with the release, as will be the case with Swift 5.7. As I understand it, there will only be the Swift 4(-ish) mode and Swift 5(.7) modes. And if libraries require a particular Swift language version (not just Swift compiler version, which is much more rare), it's virtually guaranteed it's because they adopted a feature that requires that version (like the new SE-0346 syntax). Obviously this necessitates library and consumer source updates.

Odd you think this given that Apple has been very consistent in the amount of support macOS gets (one year) since the release of at least Swift 5.

What you're really talking about here is yearly major version releases on the same cadence as macOS support. While library authors could do this, it's both a maintenance burden (I really don't want to support 3 or 4 versions of the library simultaneously) and inherently unhealthy for the Swift ecosystem, as it creates a lot of breaking churn among dependencies, likely leading to continued use of older, unsupported versions.

1 Like

We would not want a tool which automatically turns associated types into primary associated types; that is a decision that requires human thought and attention.

We could have a tool that automatically upgrades constraints in generic functions to use primary associated types, but that wouldn't solve any of the problems that this thread is about, which are the burdens of library developers around declaring primary associated types while still supporting older tools. In fact, it would actually make that problem much harder, because upgraded libraries would be much harder to make parseable by older tools.

It doesn't, though. Your library can adopt a new feature that requires a new language mode without requiring your clients to adopt that language mode, unless that feature is exposed as part of your API that they specifically want to use. They will, however, need new tools in order to build your library, since you're distributing your library as source.

Also, I don't think anything in SE-0346 requires a new language mode.

1 Like

To step back here, adopting the new syntax means using it in a library, more specifically updating an existing library. Essentially the case outlined in the initial post. In doing so existing users will no longer be able to build my library without also updating their compiler. This may or may not be possible depending on the OS they're running. These updates are not automated. So if I want to use the new syntax it necessitates a breaking release of the library, barring #if swift heroics and somehow maintaining bifurcated APIs. Language modes really don't seem to be relevant here.

On a related but separate note, there's also no way for me to help my users update to a version of my library that uses the new syntax. There's no new way mark up my declarations so the compiler can automatically migrate use sites, nor can I keep the old declarations around with a deprecation note which uses the existing deprecation rename capability.

So my original point to @lorentey is that saying "just adopt the latest version" is unrealistic, both from a tooling perspective (nothing helps users update) and from a community perspective (constant breaking churn is unhealthy). Personally, I'm not arguing that Swift should be prevented from making such changes, just that their impact is much more significant than Apple seems to realize and can't simply be dismissed with "just require the latest version".

2 Likes

this can have an unintended ripple effect on the ecosystem. for swift-grammar specifically, it is one of the dependencies of swift-json. so if swift-grammar drops supports for swift 5.3 ... 5.6, then either swift-json has to pin itself to the current version of swift-grammar (0.1.4), in which case it would not receive updates, or swift-json would also have to drop swift 5.3 ... 5.6.

most APIs can be easily gated by toolchain version. this is how we adapted (successfully, i think) to new language features in the past. however the topic of this thread is about compatibility for protocols, which can’t really be divvied up in the way you describe.

I'm not sure what you mean here. I'm not at all talking about the release & support schedule of any particular OS vendor. What I'm in fact talking about here is how to keep the maintenance burden manageable for open source Swift packages, given that Swift is a vibrantly evolving language that keeps improving with each release.

As an example, in most practical cases, I don't think it would be at all reasonable today for an actively maintained package to spend effort to support building their code with, say, the Swift 2 release*. If a package maintainer feels like they need to do that today, then they're doing something really weird. I hope we can at least agree on this.

* Again, note that I'm talking about the toolchain release that the code builds with, not an OS release that the resulting binary may run on.

Therefore, it seems obvious to me that there is some limit to how far back a package can be reasonably expected to actively support older toolchain releases. I don't think this is a matter of opinion -- it's just a practical consequence of the fact that each additional supported toolchain adds a nontrivial amount of cost & complexity to package releases, and package maintainers never have an infinite amount of resources available to throw at this problem.

From this it also follows that it also must be possible for a package to raise its expected toolchain version -- or long term package development becomes completely impractical.

(My implicit assumption here is that packages are always expected to remain compatible with the latest toolchain releases. I don't think we can have a healthy ecosystem (or any sort of coherent ecosystem) without that.

Note: keeping up with toolchain releases emphatically does not equal immediately adopting every new language feature. It's perfectly fine for a package not to annotate its protocols for SE-0347 the day 5.7 comes out. It's also fine (expected, even) for some packages to never gain such annotations.)

The only way a package maintainer can practically avoid working on multiple branches is for them to never publish new feature releases. Some clients will refuse to (immediately) update to new feature releases, in an effort to manage risk. It's not unreasonable for these clients to still expect to receive bug fixes that affect them. The tooling around Swift Packages has been carefully designed to support this with as little friction as possible.

What I'm suggesting is in fact to significantly reduce the maintenance burden by simply recognizing that if someone refuses to update their toolchain, then the most likely reason behind this is that they are worried about the upgrade destabilizing their builds. If so, then they probably think (or at least, ought to think) the same way about upgrading their other dependencies, too. (If I'm too afraid of destabilizing changes to upgrade my compiler, why would I be interested in getting new features for my package dependencies?) Therefore, package maintainers should not be overly shy about dropping support for ancient toolchain releases in their feature releases if and when continued support becomes untenable for them.

4 Likes

The main pain-point IMO is the extremely tight coupling of Swift versions to Xcode versions to macOS versions. "Just upgrade to a newer version of Swift" would be a lot more viable of a position if the newest version of Swift didn't require a version of macOS released in the last few months.

4 Likes

No, the most common reason among users we've surveyed is that they either don't want to install Monterey or have a pre-2015 Macbook which cannot run Monterey.

4 Likes

Again, I'm not at all recommending that swift-grammar "drops support" for Swift 5.3 ... 5.6. You can keep supporting those toolchains as long as you like, whether or not you decide to require a newer toolchain in a feature release.

If for some reason you decide that version 0.2 of swift-grammar will require Swift 5.7, then clients that build with Swift 5.6 (or an earlier release) will automatically remain on version 0.1.4. They won't need to do any manual action such as pinning their dependencies to get this -- SwiftPM's dependency resolution automatically takes this into account.

(Of course, given that the API of swift-grammar isn't stable, this whole issue is moot -- clients who care about source stability (which in practice is all clients) will need to pin it to a particular version anyway.)

Whether or not you want to take this step now, I do think that eventually you'll want to drop support for some of the toolchains you currently support. Assuming that Swift continues to evolve anywhere near its current pace, and assuming that you'll keep actively working on swift-grammar for the foreseeable future, will you want to keep supporting Swift 5.3, in, say, a decade from now? For package maintenance to be practical in the long term, there must be a way to phase out support for older tools. Luckily, SwiftPM has been designed to gracefully support this.

2 Likes

You still seem to be missing the point. None of us want to support older versions of Swift. We have to support older versions of Swift because Apple doesn't support older macOS versions. If macOS had the same level of support as Linux, I think virtually everyone would be willing to upgrade almost immediately, baring any show stopping bugs (which Apple's release cadence also exacerbates). Few people are refusing to upgrade their tooling, they simply can't.

You also really don't need to keep explaining how SPM works here. Not only do we know, but it's also a narrow view of the Swift community, given the continued use of CocoaPods and Carthage. Those tools exist, are important, and still fill niches SPM just can't yet.

In the end, the community's concern here is that, while we love new Swift features, especially features with such high user value, there are really no affordances for libraries to adopt the features while still maintaining their user base. This includes things like source compatibility, upgrade automation, OS compatibility, and release cadence.

8 Likes

SPM automatically picking the latest version supported by the current toolchain rather than the absolute latest version is also just not relevant. It's a nice feature, but I can't imagine any circumstance where it'd factor into a decision about when to drop support for an old version of Swift compared to a world where SPM users on old Swift versions did have to manually pin a release.

I for one appreciate all the explanations. This is after all a many to many public conversation.

7 Likes

Please consider your primary point duly noted. Unfortunately I am not the right person to engage with it.

As a part-time professional package maintainer, I'm extremely concerned about the long-term practical viability of supporting more than a handful Swift toolchain release at any given time. The more releases I'll need to support, the less time I'll have to work on literally anything else.

However, at the same time, I'd also hate to see language evolution be overly constrained by forward compatibility. I don't really see how we can generally expect older compilers to build code that uses language features from the future that they do not understand.

With that said, it would certainly be helpful if forward compatibility concerns had at least some weight in the design of the language -- as I wrote earlier, library authors should not be considered second class citizens, and I think it's neither helpful nor healthy to encourage us to reach for ad hoc solutions such as running regex replacements over our source trees. (It's already bad enough when we need to reach for such tools to solve uncommon project configuration issues.) This road will simply lead to each project using their own ad hoc & buggy equivalent of the C preprocessor, or, worse, most libraries standardizing on some supercharged version of it. (gyb is best used in strict moderation, if at all.)

The more tools we are given to deal with older toolchains, the more of them we'll be able to practically support at the same time.

Of course, for most packages, it will always remain an option to refrain from adopting new language features that can't be conditionally disabled on older toolchains. The language is backward compatible, after all! (I don't think primary associated types would be particularly affected by this, though. Wrapping a full protocol declaration is an odious task, but I don't really see why it wouldn't be technically doable in most projects. On the other hand, SE-0345 (if let shorthand) would clearly belong in this category.)

By allowing package authors to limit a feature release to clients who are on a certain toolchain version, we enable libraries to eventually adopt language features that aren't compatible with earlier compilers/build systems, without breaking compatibility with folks who are stuck using older tools. I believe this is vastly preferable to not adopting these language enhancements at all.

I do want to point out that in this particular passage, you're expecting that people who (for whatever reason) won't update one piece of critical infrastructure (the OS) will on the other hand will enthusiastically upgrade another piece of critical infrastructure (the Swift compiler). Based on my personal experience, this doesn't ring true to me at all; in fact, from what I can tell, for a large chunk of users, the stability of the language environment is likely to be just as important (or more!) than the stability of the operating system.

(Edit: evidently I wasn’t clear here — I was referring to my personal experience working as a developer prior to my current job, in the days when I still had fully working vision :stuck_out_tongue_closed_eyes:. Switching to a new compiler release (or runtime or OS) was, and afaict, remains a Big Deal to many risk averse projects. And the longer it gets put off, the riskier it gets to finally update, so projects tend to get stuck on wildly outdated tools, sometimes way past their expiration date. The point I was trying to make is that I think this sort of fracturing by inertia is inevitable, no matter what Xcode’s deployment target is — I was in fact mostly thinking about server side applications running on Linux, which are able to import the packages I happen to work on. These packages must have a strategy to reduce the maintenance burden resulting from clients who deliberately keep using ancient toolchains.)

1 Like

I have a developer who would love to update Xcode to 13.x, but he can't. Xcode 13 doesn't support the OS he's running, and newer versions of the OS don't support the hardware he has. I think Apple employees have a bit of tunnel-vision on this one. They don't see how Apple's abysmal back-compat support affects 3rd-party developers.

13 Likes

I have a work computer where the macOS version is fixed by central IT fiat, but on which I have broad latitude to install and run any tools I need to do my job. Therefore, the version of Swift I have on that machine is increasingly out of date, but not by choice.

The general gist of the feedback you're getting is that, in fact, the scenario that doesn't ring true at all actually is the crux of the matter. That it is unaddressed is unsurprising given that there's such a lack of recognition that the problem exists at all.

22 Likes