Piecemeal adoption of Swift 6 improvements in Swift 5.x

Evolution Pitches
1

Hi all,

As we travel down the road to Swift 6, we are accumulating a number of improvements to the language that have enough source-compatibility impact that we cannot enable them by default in Swift 5.x. So, they remain implemented in the Swift compiler behind the "Swift 6" flag that's only available to regression testing.

In a small number of cases, we've explicitly provided a compiler flag to enable these features in Swift 5.x: -warn-concurrency came from SE-0337, and -enable-bare-slash-regex is under discussion in the review for SE-0354. The desire for a flag to require any on all existentials came up in the review of SE-0335. One-off compiler flags for experimental features are commonplace in the development of the Swift compiler, as a staging mechanism.

I think we should explicitly embrace the piecemeal, intentional adoption of these Swift 6 features in Swift 5.x modes by providing a generalized compiler flag to indicate which features to enable. This will allow developers to incrementally move their code closer to the Swift 6 model feature-by-feature, minimizing disruption along the road to Swift 6 while getting more of its benefits with each step.

What features are we talking about, really?

There are a number of proposals we have accepted that delayed source-incompatible changes until Swift 6, going back more than two years. These are the ones I know about:

As we continue refining Swift 6, I expect we'll have more of these.

A general compiler flag

We should introduce a flag -enable-feature X, where X is a name for the feature to enable. Each proposal will document what X is, so it's clear how to enable that feature. For example, SE-0274 could use ConciseMagicFile, so that -enable-feature ConciseMagicFile will enable that change in semantics. One can of course pass multiple -enable-feature flags to the compiler to enable multiple features. Unrecognized compiler features will be ignored, and the flag will become unnecessary in Swift 6 mode because these features will automatically be enabled.

We should also extend the SwiftPM manifest format to have a general "language features" target-level option, e.g.,

.target(name: "myPackage",
        languageFeatures: ["ConciseMagicFile", "ExistentialAny"])

Detecting features in source

During the development and rollout of Swift Concurrency, we found that it was important for source code to be able to detect the availability of a particular feature, so we hacked in some support using $ identifiers for language features. For example, one can detect async/await with, e.g.,

#if compiler(>=5.5.) && $AsyncAwait
func f() async -> String {
  /* implement in terms of f(completionHandler:) */
}
#endif

Let's codify this: when a given feature X is enabled, whether through -enable-feature X or because it's implied by the language version, the compiler should define $X so that one can check for this feature with #if $X. That way, one can more easily adapt code to new features without losing support for older features and compilation modes.

Streamlining experimental features

We can apply the same approach for the whole lifecycle of features, from their experimental state until they are enabled by default in a language mode. In addition to -enable-feature X, we should add -enable-experimental-feature X. This flag should not be available in shipping compilers, but unifies the way we handle features. A feature can be developed and reviewed under -enable-experimental-feature X and then graduate to either -enable-feature X (if it has source-compatibility impacts that need to be delayed until Swift 6) or just become always-enabled. If we do this consistently, for all proposals, then #if $X will still work even if there was never an -enable-feature X.

But... aren't dialects bad?

Swift has, for the most part, avoided adding one-off feature flags because doing so creates additional dialects. Source-incompatible changes with enough impact that we cannot make them within a release are delayed until the next "major" Swift release, e.g., Swift 6, so we limit the number of dialects.

What we are trying to do here is limit permanent forks in the language. By limiting -enable-feature X to features that are going to be enabled in Swift 6, it means that folks can move along a path from Swift 5.x to Swift 6 incrementally. So long as we reject the idea of a feature that can be enabled but remains off-by-default in the next major Swift version, we have a smooth path forward that doesn't introduce the usual downsides of having disjoint dialects.

Doug

71 Likes
2

Thanks for writing this up—this is an important part of the overall vision that helps to clarify things and alleviate some of my concerns over in the other thread. Having a unified framework around upcoming features will be important to avoid the dialect problem and overall I like the direction.

Some bikeshedding:

This flag/capability is named similarly to those in other compilers where it usually refers to more permanent capabilities of the compiler, like __has_feature(X) in Clang. If the desire is to limit this to features that are upcoming in the next major language version (which I agree with), should we choose a naming that doesn't risk it being abused later on for permanent features? How about -enable-future-feature (or maybe -enable-future)?

Out of curiosity, is there a reason the $ syntax was chosen for the conditional check instead of something more traditional like #if feature(AsyncAwait)? Are there any concerns about using $... now vs. reserving that privileged syntax for something else later? For the permanent implementation of this, does it need to have such a concise syntax?

19 Likes
3

+1 on this pitch – given the temporary nature of all this, it would be useful to lay some groundwork for a far-flung future where we introduce further source breakages (such as in Swift 7, say). I'd prefer #if feature(AsyncAwait) if it's no easier or harder to leverage, for consistency and to not further overload $..., which is already used for property wrappers' projected values.

3 Likes
4

All in all this seems like a good idea and I'd also be on board. For the conditional check I would maybe propose #if hasFeature(AsyncAwait) because it plays nicely with the similar #if canImport(Darwin) we already have.

14 Likes
5

Or even #if canHasFeature(AsyncAwait) ?

It would be preferable if instead of having to look up that "Concise magic file names" is enabled with "ConciseMagicFile", this was an enum or OptionSet so you could not only get the right name with autocomplete, but also a list of all the other features you might want to enable.

6 Likes
6

Without bikeshedding the spelling too much, I’d like to argue in the spirit of the post (which I believe is good and clarifies the use of flags in language evolution here) that it would make sense to move it even more explicit;

Make it clear that a given feature flag is part of the migration to a given major language feature.

So in the case of swift 6, it could be e.g. ‘-enable-swift-6-migration’ or similar instead.

The argument why is that it provides clearer discoverability of what flags can be used for migration to a given major version (and the cadence of major version is fairly slow so it wouldn’t be to many variations) while also conveying the true intent of the flag - to smooth migration to the next major language version piecemeal.

Hope I managed to convey the point.

3 Likes
7

Using an enum for all features has better discoverability. However, is it possible to remove the enum's cases in a new SwiftPM version while still keeping it backward compatible?

2 Likes
8

Is there any plan for a webpage where one can find all possible beta features available through flags?

On swift.org?
Here?
Swift gihub?
Bundled in Documentation?

3 Likes
9

Hm, not sure. Considering that these flags are meant to be used temporarily, not to enable dialects, and that there is plenty of time between major version bumps, I think backwards-compatibility shouldn't be a concern. Am I wrong? Should it be supported to enable certain features without ever updating to the next major version?

In my mind the enum cases would remain for as long as the flags are recognized by the compiler, and when the compiler no longer recognizes old flags, it would be a relatively quick fix to remove those flags from the package manifest.

10

I am in favor of the overall idea. There are already a half-dozen cases of this on the road to Swift 6 and having a generalized mechanism in place for Swift 6 and beyond would be very helpful I think.

Compiler control condition

I'm not fond of introducing a completely new syntax for compiler control conditions, so would prefer something like #if feature(AsyncAwait) or #if languageFeature(AsyncAwait) to mirror the proposed naming for SPM manifests.

Warn / log about unused flags

I also think that instead of ignoring a flag for a feature that has been incorporated into the language that the compiler should emit a log or warning that the flag is no longer used or necessary.

There are many projects that carry around unnecessary build flags/settings of various kinds, the person who introduced the flag has left the project, newcomers don't want to risk screwing things up by making a change, etc. I think it would be useful to signal that a flag is no longer needed once that is the case.

Swift 6 Aggregate Flag

Some developers will want to write Swift using Swift 6 syntax as soon as they are able, but will have to track down the various Swift Evolution proposals to find out the language features they should enable to do so. (Or possibly keep revisiting a webpage on swift.org for an updated list, if one should exist in the future.)

Since the intent is to enable features that will be incorporated in the next major version, it may make sense to have an aggregate Swift6Preview or Swift6Features that turns on all features currently slated for that version.

A flag named that way also is a big hint that as of Swift 6, that flag will not be necessary.

Flag names

I like the simplicity of the naming -enable-feature X and -enable-experimental-feature X.

The only drawback I can think of is that the name -enable-feature X makes it tempting to propose features as permanent dialects and newcomers to the evolution process will have to have its intent explained, whereas something like the proposed enable-future-feature X makes it more clear.

That said, I believe people are always going to propose flags to have language features turned on or off no matter what these flags are named, so I think the names are fine as-is.

SPM option question

Is the intent that the language feature options specified for a SwiftSPM target would accept experimental language features as well?

13 Likes
11

My previous post might be too terse, so I probably didn't express my concern properly. My line of thought is this:

If we want to use a first-party LanguageFeature enum in the manifest, it needs to come from the Package module, which comes from SwiftPM. Every time when we have a new version of the language, we have a new version of SwiftPM that recognizes the new language version, and a new (at least as new as the previous one) Package module. Because newer versions of SwiftPM must support manifest files using previous versions' Package module, APIs defined in Package must be maintained. So, if we were to introduce such an enum, then I worry that we might not be able to remove any of its cases if we want SwiftPM to understand manifests written before the cases' removal. In my opinion, since there will only be more staged language features going forward, enum cases accumulated over the years spanning multiple major versions will only hinder discoverability in the long term.

Maybe supporting availability attributes on enum cases is a solution, but I don't know when/if it will ever happen. Or, maybe instead of an enum, it could be a struct with lots of static Selfs annotated with availability, but it feels like a hack. Edit: enum cases can be annotated with availability. I never used it and didn't know before.

1 Like
12

Because newer versions of SwiftPM must support manifest files using previous versions' Package module

Aha, so the intent is to allow outdated feature flags to be specified and simply ignored? I don't think that's the best way to go because of the long release cycles and the ease of deleting those removed enum cases from the manifest (you would get a clear error).

But what do I know, I don't have a stake in backward compatibility myself, so I'm not going to argue too hard against this. Looking up these strings every once in a while isn't so difficult either. I just thought I'd bring it up in case it was an oversight.

13

I can definitely see the appeal in having these language features be more discoverable, potentially via an enum for use in SwiftPM manifests.

However, since the intent is for them to be temporary, I don't think it makes sense to add them to API formally. As @wowbagger mentions, any symbols added would need to kept around for backwards compatibility, so over time this would be an ever-increasing enum of mostly deprecated values.

I do think that having some well-known spot that lists the current set of valid language features is important, including the version of Swift the language feature was added.

9 Likes
14

If this happens, would the core team also consider adding an unstableLanguageFeatures entry to package manifests, where packages could declare their use of non-source-stable features such as _modify, @inline(__always), @_specialize, etc?

I mentioned this before, but we have a serious problem with these features being used in popular first-party and third-party source packages, to the point where they already are de-facto language features and dialects.

It would be nice to introduce some discipline over this stuff. Next month it will be 7 years since Swift was first announced; you shouldn't need unstable language features to write a good Deque, and we shouldn't keep this technical debt around forever.

In the future, I think it would be better to move all unstable features to an opt-in system, in the same way we're describing here. Packages should be built with fallbacks (or just omit API which requires unstable feature), and downstream clients should explicitly consent to enabling APIs built on unstable features through a special package build flag.

8 Likes
15

This would seem to be antithetical to the point that this plan is supposed to "limit permanent forks in the language." For that to be the case, the features in question being enabled must already be an accepted part of the language to be used without feature flags at some concrete future release. By contrast, formalizing unstable language features with supported flags would be creating forks in the language with no expiry date.

8 Likes
16

+1 and thanks for the clarification.

Bikeshedding:

How about -enable-swift-next AsyncAwait, ... for the flag and #if SwiftNext(AsyncAwait, ...)

1 Like
17

+1, Love this idea. I was looking for a flag for requiring any on existential the other day.

Agree with the posts above that something along the lines of #if hasFeature(ExistentialAny) is nicer than #if $ExistentialAny.

5 Likes
19

+1 from one in the peanut gallery

As others mention, features to better manage these transient dialects:

  • Discoverable: how would I enable this? what does this message mean?
  • Configurable...
  • Traceable: sources, build scripts, compiler messages, even binaries?
  • Mergeable: once released, what's required to finalize these configs?

To get more specific...

  • As a concept, "enable-feature" is not limited to pre-(future)-release.
  • Java has successfully used the notion of "early-access" at scale since its beginning.

Wrt discoverability, prose terms like ConciseMagicFile can make things harder: people have to find and remember them, and then you have to associate them with the specific feature. (me: What was that name? Where do I find docs on this magic file? (Googling...))

I prefer the id's we have (e.g., SE-0354), as familiar to those using early-access features and quickly found on search.

So perhaps the "early access" tag for a a feature is just e.g., "EASE_0354".

It can be used as a compiler flag, in sources, as a tag on compiler warnings (and in binary metadata?):

  • flag: --early-access EASE_0354,EASE_0274
  • source: #if compiler(>=5.5) && $EASE_0354
  • error: You done wrong here! (EASE_0354)

As for reducing community friction and speeding adoption, I think of these not as forks/dialects but as peer discussions with both emerging and trailing consensus. The friendlier/easier, the better.

Which I think means making it easy to clearly distinguish different voices at one point, and then wiping the distinctions when consensus takes over.

The Package.swift internal declarations are enough for individuals to try it out, but they're a bit hidden. I imagine users would like to find compliant libraries without a purpose-built parser, and the team would like to discover compliant open-source packages -- e.g., to run a compiler test suite against the known-compliant world :slight_smile:

So it would be nice for package to be able to be compiled/distributed with standard settings, but also publish future settings if compliant.

One clear way is a second file with a qualified name, e.g., Package-6.swift.

It might also be easier for the team to implement SPM-6 alongside SPM-5 when declarations can differ, or for users to transition Package-6.swift to Package.swift once Swift 6 is adopted.

Ideally we'd like to enable library packages to migrate to future features ASAP, to bring early adopters along.
But it's likely library packages will straddle the divide for some time as clients catch up. So after Swift-6 there may be cause to keep Package-6.swift, or even to support a Package-5.swift (shifting the burden of not changing to laggards).

Anyhoo, thanks for the great proposal and community!

1 Like
20

Would there be an issue with using availability annotations like @available(swift, deprecated: 6) on enum cases? Because I noticed that deprecated cases are sorted after other cases in autocomplete (in Xcode at least).

Would there be other downsides to keeping those cases around forever?

Another benefit of using enum cases would be that they could have a short documentation comment with a link to the relevant proposal.

2 Likes
21

I don't care about the $ being concise, and it wasn't chosen for aesthetics. The $ syntax was chosen because existing Swift compilers already support it: $ can be at the start of a normal identifier (since at least Swift 3, probably earlier), but as with other $ identifiers in the language (for property wrappers, implicit closure arguments) you cannot define an entity with a $ name yourself. So the $ handy for compatibility with older tools... which is the point of the feature. (Note: @beccadax came up with the idea of using $ back when we introduced async/await into the compiler, I'm just codifying it now).

That doesn't mean we can't have the hasFeature(AsyncAwait) syntax, but it has downsides. If I were to write:

#if hasFeature(AsyncAwait)
// ..
#endif

I'll get an error like this on all compilers that predate the hasFeature syntax:

error: unexpected platform condition (expected 'os', 'arch', or 'swift')

Now, one can work around it with nesting:

#if compiler(>=5.7)
#if hasFeature(AsyncAwait)
// ..
#endif
#endif

... but if you have an #else block, you need to duplicate it: once for the inner if, once for the outer if.

Clang got around the issue when it introduced __has_feature(X) because you could use macro tricks to deal with older compilers, e.g.,

// C, not Swift!
#ifndef __has_feature
#  define __has_feature(X) 0
#endif

We don't have that option in Swift.

So, while I'm usually on the side of "pick the best syntax and we'll evolve toward it," a feature like this that's 100% about backward-compatibility benefits from working with existing and older compilers.

I'm happy with one of these... perhaps -enable-future-feature and -enable-experimental-feature.

I intentionally proposed strings so that we don't need to revise the SwiftPM manifest format for each new feature we add. For example, let's say that Swift 5.9 adds conciseMagicFile. If you adopt that feature in the manifest like this:

  .target(name: "MyPackage",
          languageFeatures: [ .conciseMagicFile ])

Then your manifest will only compile properly on Swift 5.9; SwiftPM in Swift 5.8 won't know about the conciseMagicFile case. Now, you can deal with this with versioned manifest files, but it requires duplicating the manifest for each Swift version in which a feature was used that you want to enable. Personally, I think the cost of having to look up a string "ConciseMagicFile" once in a while is lower than the ongoing burden of maintaining multiple manifest files, hence the stringly-typed features.

I think it would be great for swift.org to contain a list of features w/ their associated versions and, if we go forward with this proposal, the feature-enablement name.

Yes, backward-compatibility is important here. Package authors especially would like to opt-in to new features when they are available, without breaking compatibility with older tools versions. With the proposal as written, they can add the appropriate feature to their package manifest and then use #if $FeatureName to conditionally use the feature. This was quite important with the introduction of concurrency, for example, and a number of libraries made use of the semi-documented #if compiler(>=5.5) && $AsyncAwait to add async interfaces when built with new-enough tools.

I agree with this in the narrow case where one has specified both a feature-enablement flag (-enable-future-feature ConciseMagicFile) and a Swift language version that enables that feature by default (-swift-version 6). Did you mean for the warning/log to occur in other cases than that?

A flag like this would necessarily change meaning from one Swift release to the next as we add new "Swift 6" features, breaking source at each step, so I'm concerned about adding it. Perhaps there could be a different flag whose role is to warn about any features that the compiler knows about that haven't been explicitly specified. That way, you would get warnings if you haven't opted into a feature that's available, but the list of features that's enabled is still what was specified by the developer.

No, I should probably have both futureLanguageFeatures: and experimentalLanguageFeatures:.

We could put them into the "experimental" category, rather than inventing something new.

I don't think the SE numbers have a lot of meaning to folks outside this forum. Identifiers like AsyncAwait and ConciseMagicFile are far more immediately recognizable when you read them in, e.g., a package manifest or in source code.

Doug

15 Likes