[Pitch] Extensible enums for non-resilient modules

[tshortli (Allan Shortlidge)]

@xwu

A library turning on this feature would be the moral equivalent of adding a new case to every enum declared in that library—which (unless it explicitly documented otherwise) the library could always do. For cases where there was a documented promise of being frozen, now there is a way to express it in code.

Turning on this feature does not require any existing library to lie about whether its enums are frozen: rather it exercises an option that the library has always explicitly reserved. If that cannot be tolerated in practice by a library’s clients, then that library’s enums were in fact frozen anyway.

I agree in principle that the effect of turning this feature on is the same as exercising a right that was already afforded to library authors. That said, we don't have precedent (that I am aware of) where turning on a Swift language feature implies a source break for clients all on its own. If we settle on the conclusion that adopting ExtensibleEnums in a module is inherently a semver breaking change when that module already vends non-frozen enums, then I have no problem with that. I do think it creates a novel problem for the ecosystem regarding adoption of language features, though. And if we decide we need a way to mitigate the problem, then I'd like to see something like @preEnumExtensibility provided rather than encouraging library owners to use @frozen inappropriately (which I thought I had seen suggested elsewhere in the discussion but I could have misinterpreted).

[8675309]

If the solution is to put all your code is in one giant package, then you don't actually have packages, you're just being expedient with the build system. I.e., you cannot actually make use of the package keyword to split something like a monorepo into subsystems.

Unless this is changed as well then package isn't a solution either.

[FranzBusch]

@tshortli

I agree in principle that the effect of turning this feature on is the same as exercising a right that was already afforded to library authors. That said, we don't have precedent (that I am aware of) where turning on a Swift language feature implies a source break for clients all on its own. If we settle on the conclusion that adopting ExtensibleEnums in a module is inherently a semver breaking change when that module already vends non-frozen enums, then I have no problem with that. I do think it creates a novel problem for the ecosystem regarding adoption of language features, though. And if we decide we need a way to mitigate the problem, then I'd like to see something like @preEnumExtensibility provided rather than encouraging library owners to use @frozen inappropriately (which I thought I had seen suggested elsewhere in the discussion but I could have misinterpreted).

I don't see a way around enabling ExtensibleEnums not resulting in a SemVer major without the library authors going around and marking all existing public enumerations as non-extensible (regardless of what we call the attribute). All public enums in the ecosystem must be treated as non-extensible.

I thought about introducing a similar attribute to @preconcurrency like you suggested as well but I fail to see how it would work in practice. Let's imagine module A turns on the new language feature and marks all existing public enumerations with the hypothetical @preEnumExtensibility annotation. So far I could only think of two effects that annotation could have on consuming modules:

1. Downgrade the error of requiring an @unknown default to a warning
2. Synthesize at compile time a @unknown default. I think the only valid thing to do in the synthesized code is to fatalError though.

Let's say we pick either of the two options. Now the new minor version of the library is released and all consuming modules continue to compile potentially producing new warnings. A few releases later the authors of module A want to add a new case now to one of the previously marked @preEnumExtensibility enumerations. What would happen now in the consuming modules? If we went with option 1. this will now produce an error that we are no longer exhaustively switching. I don't see a way that we can downgrade that error. If we went with option 2. the code would continue to compile, the consuming module would get a new warning of the missing case. However, it would now open up a real possibility to hit the fatalError. @tshortli do you see any other option how we could handle @preEnumExtensibility in consuming modules that also works when new cases are added?

I also briefly want to take a step back and talk about the goals we want to achieve here:

1. Align the differences between the two language dialects in a future language mode
2. Provide developers a path to opt-in to the new behavior before the new language mode so they can start declaring new extensible enumerations
3. Provide a migration path to the new behavior without forcing new SemVer majors

The following is a nice-to-have but not required:

1. Enable existing enums in packages to be extended without a new major

If we can solve the nice-to-have creatively I would be delighted but in my opinion it isn't a requirement to achieve the overall goals. As a package author myself, I'm happy to accept that existing public enumerations are non-extensible until I decide to mint a new major. Importantly, I want to be able to declare new extensible enumerations before that new major though.

[Karl]

@8675309

If the solution is to put all your code is in one giant package, then you don't actually have packages, you're just being expedient with the build system. I.e., you cannot actually make use of the package keyword to split something like a monorepo into subsystems.

Yeah you can - the point is that multiple physical packages can be treated as one logical package.

Let's imagine something like Clang and LLVM (as an example of a monorepo with various subsystems). Let's imagine they are two separate SwiftPM packages. Clang depends on LLVM, and it has a choice:

1. It can say that its code and LLVM's code are separate logical packages. That means they might be distributed and versioned separately, and it will need to account for future cases in enums from LLVM using @unknown defaults.

2. It can say that its code and LLVM's code are part of the same logical package. That means they are distributed together and are version-locked. It won't need to account for future cases in enums from LLVM, but they will cause build failures if not handled exhaustively.

You can think of it as taking LLVM's targets and products and merging them in to your own SwiftPM package. And this is all a separate consideration to where the actual source files are located, in which git repositories - maybe the same, maybe different.

If you are going for the version locking option, you probably want to more carefully manage when your dependencies get updated - it will be more difficult to do, potentially preventing you from quickly incorporating important bug fixes and security updates (which is why version locking should not be the default).

In Clang/LLVM's case, they can go for version locking because the projects are so tightly connected that changes to LLVM would test that Clang was updated as part of CI (which I believe is currently the case). If you have that kind of relationship with the developers of your dependencies, you can do that.

The Swift project itself could do that as well for its various libraries (string processing, swift-syntax, swift-driver, etc). Some clients of these libraries will want the first style of dependency, but the compiler itself could use the latter (version-locked) dependency.

[dimi (Dimitri Bouniol)]

Calling this out so we don't forget about it, since it wasn't mentioned in the pitch proper: SwiftPM's diagnose-api-breaking-changes will also need to be updated to not mark new cases in extensible enums as API breaking.

[xwu (Xiaodi Wu)]

@FranzBusch

• Align the differences between the two language dialects in a future language mode
  [...]
• Provide a migration path to the new behavior without forcing new SemVer majors

I think, definitionally, there has to be a point along the migration path where these are mutually incompatible goals, unless every public enum has to be annotated in some way (which I would consider actively undesirable). I think that's what your first paragraph is saying also, if I understand you correctly.

Adding even a single case to a single enum is a source-breaking change for clients of a non-resilient library in the current state of things, but adding any number of cases to any number of extensible enums ceases to be source-breaking with the full-fledged feature you pitch.

Therefore, for any non-resilient library that will ever add any case to any enum, turning on this full-fledged feature in one go is the least source-breaking migration path possible for users. This leaves only non-resilient libraries that will never add any case to any enum: they can turn on the full-fledged feature and annotate all of their enums as @frozen and not break any clients' source.

What am I missing?

[FranzBusch]

@xwu

I think, definitionally, there has to be a point along the migration path where these are mutually incompatible goals, unless every public enum has to be annotated in some way (which I would consider actively undesirable). I think that's what your first paragraph is saying also, if I understand you correctly.

I disagree that those goals are mutually exclusive but we have to be cognizant of the current state of the ecosystem and the impact of source breakage.

If we introduce this new language feature a new package or any package without a public enum can turn this feature on and start declaring public enumerations that are extensible by default. For such packages this new language feature aligns both language dialects.

For existing packages with existing public enumerations I strongly believe that those libraries must continue to treat the public enums as effectively @frozen until they actively want to break API and release a new SemVer major.

@xwu

Therefore, for any non-resilient library that will ever add any case to any enum, turning on this full-fledged feature in one go is the least source-breaking migration path possible for users. This leaves only non-resilient libraries that will never add any case to any enum: they can turn on the full-fledged feature and annotate all of their enums as @frozen and not break any clients' source.

I think we agree here. If a module wants to introduce a new case then it's best to break everything at once and change the behaviour for all public enumerations at the same time. I would advise any maintainer to audit all their enums before a new major and decide on a case-by-case basis if an enum should be extensible or not.

Importantly, we have a huge range of packages that rarely release new majors and actively try to avoid new majors. Packages such as swift-collections or swift-nio fall into this category. If such packages would release a new major it would effectively split the ecosystem until all upstream packages have adopted the new major. Such packages may still want to declare new public extensible enumerations. The proposed migration path with enabling the language feature and annotating existing enumerations with @frozen allows exactly that.

[xwu (Xiaodi Wu)]

@FranzBusch

For existing packages with existing public enumerations I strongly believe that those libraries must continue to treat the public enums as effectively @frozen until they actively want to break API and release a new SemVer major.

Right, so for an existing package with public enums, there is no migration path to adopting such a feature aligning the two language dialects that doesn't end up with an API-breaking change at some point.

And—since the point is to align language dialects and not make more of them—we'd want this feature to become non-optional as part of some future language mode.

So, if you're a non-resilient package that already exists and has a public enum, the moment we state this as a plan of record, either you are committing to be frozen in time (ha) or you're fated for an API-breaking change.

[FranzBusch]

@xwu

Right, so for an existing package with public enums, there is no migration path to adopting such a feature aligning the two language dialects that doesn't end up with an API-breaking change at some point.

No existing packages can adopt the feature they just have to stay committed to their existing API until they make the choice to break API and release a new major. I personally believe that's the right approach since it empowers package maintainers to make the choice.

@xwu

So, if you're a non-resilient package that already exists and has a public enum, the moment we state this as a plan of record, either you are committing to be frozen in time (ha) or you're fated for an API-breaking change.

I disagree that such packages stay frozen in time or are forced to break API. Packages can adopt the feature or the new language mode. In my opinion, it is a totally valid approach as maintainer to accept that existing enumerations are forever frozen but new ones are extensible by default. Each package maintainer should weigh the benefits of making existing enums extensible vs the impact of a major on their adopters.

On a related note, I hope that our package manager gains the capabilities to support multiple conflicting major versions of a single package in a single dependency graph at some point which would significantly reduce the impact of a new major release. However, I don't think we should wait with making extensible enums available in non-resilient modules until our package manager gains such capabilities.

[xwu (Xiaodi Wu)]

@FranzBusch

I disagree that such packages stay frozen in time or are forced to break API. Packages can adopt the feature or the new language mode. In my opinion, it is a totally valid approach as maintainer to accept that existing enumerations are forever frozen but new ones are extensible by default. Each package maintainer should weigh the benefits of making existing enums extensible vs the impact of a major on their adopters.

Sorry, I'm not sure I understand where we are disagreeing.

An existing package either has to freeze all existing public enums explicitly, or break API, or not adopt the feature (or any new language mode incorporating the feature)—that is the complete set of possible options regardless of what other bells and whistles are added to roll out this feature, no? As soon as we commit to this as the plan for the language, there is no escape from making one of these choices for existing package maintainers.

[FranzBusch]

@xwu

An existing package either has to freeze all existing public enums explicitly, or break API, or not adopt the feature (or any new language mode incorporating the feature)—that is the complete set of possible options regardless of what other bells and whistles are added to roll out this feature, no? As soon as we commit to this as the plan for the language, there is no escape from making one of these choices for existing package maintainers.

Yes these are the only available choices. My disagreement was with the term "frozen in time" since packages that choose to freeze all existing public enums can still continue to evolve by adding all kinds of new APIs. They just can't change the API of the existing enums.

[xwu (Xiaodi Wu)]

@FranzBusch

Yes these are the only available choices. My disagreement was with the term "frozen in time" since packages that choose to freeze all existing public enums can still continue to evolve by adding all kinds of new APIs. They just can't change the API of the existing enums.

Sure. But to be clear: if an existing library (a) declares a public enum without @frozen; and (b) wishes to "align" the meaning of that declaration as it is written in the non-resilient and resilient dialects by means of this pitched feature—I can see no migration path which does not force a new semver major. Aligning and not forcing a new semver major are mutually incompatible goals for such a declaration, we agree?

I don't consider freezing all public enums or outright not adopting the feature flag/future language mode as a form of "aligning"—they're just ways of working around or dodging the issue. Just like, trivially, any non-resilient library that doesn't declare any public enums is already fully "aligned," but not in any meaningful way.

[FranzBusch]

@xwu

Sure. But to be clear: if an existing library (a) declares a public enum without @frozen; and (b) wishes to "align" the meaning of that declaration as it is written in the non-resilient and resilient dialects by means of this pitched feature—I can see no migration path which does not force a new semver major. Aligning and not forcing a new semver major are mutually incompatible goals for such a declaration, we agree?

Thanks for spelling it out explicitly. I understand what you mean now and I agree that for ecosystem wide alignment every package with a public enums needs to revision a new major.

For what it's worth, I don't think that this is problematic though since there are more compiler errors that pop up when building a module that is normally build without resiliency. The vast majority of packages in the Swift ecosystem do not support compiling in both dialects currently unrelated to the extensible enum difference.

[FranzBusch]

Thanks for the healthy discussion so far! I updated the proposal with the following changes:

• Added future directions for adding additional associated values
• Removed both the @extensible and @nonExtensible annotation in favour of re-using the existing @frozen annotation
• Added the high level goals that this proposal aims to achieve
• Expanded on the proposed migration path for packages with regards to their willingness to break API
• Added future directions for exhaustive matching for larger compilation units
• Added alternatives considered section for a hypothetical @preEnumExtensibility
• Added a section for swift package diagnose-api-breaking-changes

[erithacus (Robin)]

Isn’t the proposal to add the @extensible attribute and it would be something you could choose to apply to your enums or not.

If you don’t add it then the enum would behave as before. If you add it then consumers would be required include @unknown default to handle future cases.

[tshortli (Allan Shortlidge)]

@FranzBusch

This is great! We recently pitched an overhauled version of Extensible enums for non resilient modules that is using a language feature to change the behavior of enums inside a module. This would be a perfect candidate to offer an adoption mode where each existing public enum in the module would generate a warning with a fix-it to add @frozen to it in preparation to turning the language feature to avoid any API breaks.

Although the quoted post is from a different thread about migration tooling, I wanted to discuss it here to avoid derailing that thread. Part of my earlier participation in this thread where I suggested the @preEnumExtensibility attribute was motivated by this topic and I want to return to it because I feel I didn't really get my point across well and this post hopefully provides the right context to try again.

I'm concerned with the idea of actively promoting application of @frozen to existing enums because I don't think it's the right action take in enough cases to make it the default suggestion. If I understand the logic of this approach correctly, it is motivated by the idea package owners will, by default, want to minimize source breaks while adopting EnumExtensibility. To achieve that goal, then, package owners are encouraged to preemptively freeze all of their existing enums, whether those enums were actually frozen or not in practice, and promise to introduce new enums and new APIs using those enums in the future if the need to extend the frozen ones arises. Is that a fair assessment of why we would implement migration tooling for EnumExtensibility this way?

If it is, I worry that the implications of this will be hard to communicate to developers that they are making this promise, and that as a result they may later regret the decision to take the suggestion to mark their enums frozen. In my mind, it would be better to emphasize to developers that adoption of EnumExtensibility is by default expected to be source breaking and to plan their releases accordingly. Developers who are very motivated to mitigate this can do so using appropriate techniques, but my perspective is that doing so is a thoughtful activity that is not conducive to being the default behavior of an automated migration.

My opinion here is of course predicated on the idea that in general, most enums in existence are extensible in nature (that is why @frozen is an attribute and not the default behavior). Is there contention about that? Am I missing something else about this idea?

[FranzBusch]

@tshortli

My opinion here is of course predicated on the idea that in general, most enums in existence are extensible in nature (that is why @frozen is an attribute and not the default behavior). Is there contention about that? Am I missing something else about this idea?

I think this is the fundamental difference in our opinions. While I understand your point that all existing enums are extensible since they haven't been marked @frozen. I would strongly argue that the reality is different. In non-resilient modules all existing enums are non-extensible. The concept of extensible enumerations simply doesn't exist there.

We can pretend that the non-existence of extensible enumerations in non-resilient libraries means that authors haven't yet made the decision if their enums are extensible or not but in my opinion this whole argument falls down when looking at the state of the package ecosystem. Many packages have already been bitten by extending public enumerations without new SemVer majors and broke adopter code that exhaustively matched over them. In the end, the current semantics of the language in non-resilient modules has made the decision for maintainers that all public enumerations are frozen.

@tshortli

To achieve that goal, then, package owners are encouraged to preemptively freeze all of their existing enums, whether those enums were actually frozen or not in practice, and promise to introduce new enums and new APIs using those enums in the future if the need to extend the frozen ones arises. Is that a fair assessment of why we would implement migration tooling for EnumExtensibility this way?

Your assessment is correct about the proposed migration path. However, with the above in mind I think we can't make the argument that public enums in non-resilient modules aren't frozen. While the developers didn't actively make the choice to freeze them, the language itself did.

Don't get me wrong. Overall, I'm very sympathetic to the view of developers haven't consciously made the choice of freezing their public enums and I would love if we could just flip a switch and all public enums become extensible. However, just changing the behavior would incur a massive source break and that's why I personally believe we need to offer a migration path that gives the package authors the chance to decide between breaking their public API and tagging a new major or actively acknowledging that their public enums are frozen allowing them to prevent an API break and to create new public extensible enums.

@tshortli

In my mind, it would be better to emphasize to developers that adoption of EnumExtensibility is by default expected to be source breaking and to plan their releases accordingly.

I just don't see how this would play out nicely in the ecosystem without keeping certain packages from adopting the feature or a new language mode. Some packages try everything possible to avoid breaking their public API since they are so fundamental to the rest of the ecosystem that an API break would have wide ranging impact. Just take swift-nio as an example. The latest release is 2.81.0. This means NIO released more than 81 times without breaking API since 2019.

[cal (Cal Stephens)]

@FranzBusch

that's the reason for why the compiler emits a warning when a known case isn't handled even when a @unkown default case is defined

Is this true in Swift 6 mode?

To check this I wrote a switch case over a random non-frozen enum that I could think of (Calendar.Identifier in Foundation):

import Foundation

switch Calendar.current.identifier {
case .gregorian:
  print("Gregorian")
@unknown default:
  print("Unknown default")
}

Under -swift-version 5 I get warning: switch must be exhaustive; this is an error in the Swift 6 language mode, and under -swift-version 6 I get error: switch must be exhaustive.

So with this, I want to bump a question I asked earlier but haven't seen discussion about:

@cal

One question -- if I have an exhaustive switch case plus an @unknown default case, and then the library adds another enum case later, what happens? Does my switch case continue to compile even though it is no longer exhaustive? If not, then I don't think this fixes the API evolution problem for libraries.

For example, when I test this locally with a non frozen enum, I see that even with @unknown default I have to switch exhaustively (or use default). If we have:

switch pizzaFlavor {
 case .hawaiian:
     throw BadFlavorError()
 case .pepperoni:
     try validateNoVegetariansEating()
     return .delicious
 case .cheese:
     return .delicious
 @unknown default:
     try validateNoVegetariansEating()
     return .delicious
 }

and then later the library adds the .veggieSupreme flavor, wouldn't we now get the error:

error: switch must be exhaustive
   | `- note: add missing case: '.veggieSupreme'

So, as is, I'm not sure how this helps API authors confidently add new enum cases without the possibility of breaking existing consumers.

Based on what I see here it seems like adding a new enum case to a non-frozen enum is still source-breaking, since it would break any existing exhaustive switch statement. If this is true then I don't understand how non-frozen enums help library authors add new enum cases without breaking source compatibility.

Am I missing something here, or mistaken about this resulting in errors?

[FranzBusch]

@cal

Based on what I see here it seems like adding a new enum case to a non-frozen enum is still source-breaking, since it would break any existing exhaustive switch statement. If this is true then I don't understand how non-frozen enums help library authors add new enum cases without breaking source compatibility.

Am I missing something here, or mistaken about this resulting in errors?

Thanks for bringing this up again. The change to upgrade the warning to an error was introduced here [Sema] Non-exhaustive switch statements are always an error in Swift 6. by hborla · Pull Request #73472 · swiftlang/swift · GitHub. However, as you rightly pointed out that made adding a new enum case a source breaking change even in resilient modules. @tshortli fixed this here Sema: Allow resilient modules to add enum elements without breaking c… · swiftlang/swift@0ac8cd3 · GitHub and it looks like the nightly Swift 6.1 toolchains already behave correctly and emit a warning.

[tshortli (Allan Shortlidge)]

@FranzBusch

We can pretend that the non-existence of extensible enumerations in non-resilient libraries means that authors haven't yet made the decision if their enums are extensible or not but in my opinion this whole argument falls down when looking at the state of the package ecosystem.

I think you're right that in the open source package ecosystem it is the case that existing enums might as well be @frozen by default. Most of these enums are modeling something that is in fact extensible instead, but until a language feature like this one is implemented, there isn't a way to express that unless your code is part of a resilient library.

Although I accept that premise, I still doubt that it makes sense to widely promote an automated language mode migration that adds @frozen to all existing public enums. It's not clear to me that such a migration would be appropriate for a majority of Swift code. Let's consider how this proposal applies to the different categories of Swift modules that are out there:

• Packages that are distributed as open source and are core to the ecosystem. This kind of code is the motivation for the proposal and having migration tooling for these packages makes sense.
• Libraries built with library evolution. A migration to add @frozen must not be offered when building these modules. I assume it should be straightforward to make sure that the migration tooling takes the presence of -enable-library-evolution into account (though there is some code out there that is built both as a package and also as a resilient library, depending on context, and I don't know how the tooling would detect that).
• Application code, or any other kind of leaf-node module. The only clients of a module like this are tests, and therefore source compatibility does not seem relevant.
• Packages that aren't open source. It's hard to state anything concrete about the requirements of this kind of code. For some of these packages, maybe source compatibility is important to the organization that maintains them. For others, source compatibility may not really be relevant at all.

I think the harm in offering to make all existing public enums @frozen is that it may over-emphasize the concept of frozen-ness to a lot of developers who shouldn't need to be aware of it and create confusion about when the attribute is appropriate to use. Without sufficient guidance accompanying the migration, it could be easy to draw the conclusion that "public enums should be @frozen by default." According to the ethos of progressive disclosure, the @frozen attribute should only be introduced to developers when they need it, but if this migration is offered to every developer when upgrading to the latest language mode then they are going have to become familiar with it as a concept whenever their code contains public enums, and I think that's probably making too many assumptions.

I think it's fine to implement a migration tool that can put @frozen on every public enum as a convenience, but I'd prefer to see it reserved as a tool that is available to experts who know they need it, rather than something that is promoted to everyone adopting the latest Swift language mode that includes this proposal. Either way, I think we will need some thorough documentation about the ramifications of this proposal and how to address its impacts contextually in your project.