Extensible Enumerations for Non-Resilient Libraries

Hey folks. Over the weekend I put together a quick draft patch that would enable non-resilient Swift libraries to provide "extensible" enums, where adding a new case would not be a Semver major breaking change. This patch does the absolute minimum required to enable the functionality, so before I go any further with it I wanted to get the community's opinion on whether this is a problem worth addressing.

Below is a draft proposal that covers the issue. Please give it a read and let me know your thoughts.


Extensible Enumerations for Non-Resilient Libraries

Introduction

This proposal adds an attribute to allow Swift enumerations to opt-in to an extensible behaviour. This reconciles a feature mismatch between resilient and non-resilient dialects of Swift, and makes Swift enumerations vastly more useful in the public API of non-resilient Swift libraries.

Motivation

When Swift was enhanced to add support for "library evolution" mode (henceforth called "resilient" mode), the Swift project had to make a number of changes to support a movable scale between "maximally evolveable" and "maximally performant". This is because it is necessary for an ABI stable library to be able to add new features and API surface without breaking pre-existing compiled binaries. While by-and-large this was done without introducing feature mismatches between the "resilient" and default "non-resilient" language dialects, the @frozen attribute when applied to enumerations managed to introduce a difference. This difference was introduced late in the process of evolving SE-0192, and this pitch would aim to address it.

@frozen is a very powerful attribute. It can be applied to both structures and enumerations. It has a wide ranging number of effects, including exposing their size directly as part of the ABI and providing direct access to stored properties. However, on enumerations it happens to also exert effects on the behaviour of switch statements.

Consider the following simple library, an SDK to your favourite pizza place:

public enum PizzaFlavor {
    case hawaiian
    case pepperoni
    case cheese
}

public func bakePizza(_ flavor: PizzaFlavor)

Depending on whether the framework is compiled with library evolution mode enabled, what the caller can do with the PizzaFlavor enum varies. Specifically, the behaviour in switch statements changes.

In the "standard", "non-resilient" mode (with library evolution disabled), users of the PizzaSDK can write exhaustive switch statements over the enums vended by PizzaSDK:

switch pizzaFlavor {
case .hawaiian:
    throw BadFlavorError()
case .pepperoni:
    try validateNoVegetariansEating()
case .cheese:
    return .delicious
}

This enumeration will happily compile. If the author of the above switch statement was missing a case (perhaps they forgot .hawaiian is a flavor), the compiler will error, and force the user to either add a default: clause, or to express a behaviour for the missing case. The term for this is "exhaustiveness": in the default non-resilient dialect, the Swift compiler will ensure that all switch statements over enumerations cover every case that is present.

There is a downside to this mode. If PizzaSDK wants to add a new flavour (maybe .veggieSupreme), they are in a bind. If any user anywhere has written an exhaustive switch over PizzaFlavor, adding this flavor will be an API and ABI breaking change, as the compiler will error due to the missing case statement for the new enum case.

Because of the implications on ABI and the requirement to be able to evolve libraries with public enumerations in their API, the resilient language dialect behaves differently. If PizzaSDK was compiled with enable-library-evolution turned on, when a user attempts to exhaustively switch over the PizzaFlavor enum the compiler will emit a warning, encouraging users to add an @unknown default: clause. Thus, to avoid the warning the user would be forced to consider how new enumeration cases should be treated. They may arrive at something like this:

switch pizzaFlavor {
case .hawaiian:
    throw BadFlavorError()
case .pepperoni:
    try validateNoVegetariansEating()
    return .delicious
case .cheese:
    return .delicious
@unknown default:
    try validateNoVegetariansEating()
    return .delicious
}

When a resilient library knows that an enumeration will not be extended, and wants to improve the performance of using it, the author can annotate the enum with @frozen. This annotation has a wide range of effects, but one of its effects is to enable callers to perform exhaustive switches over the frozen enumeration. Thus, resilient library authors that are interested in the exhaustive switching behaviour are able to opt-into it.

However, in Swift today it is not possible for the default, non-resilient dialect to opt-in to the extensible enumeration behaviour. That is, there is no way for a Swift Package Manager package to be able to evolve a public enumeration without breaking the API. This is a substantial limitation, and greatly reduces the utility of enumerations in non-resilient Swift. As a non-exhaustive list of problems this can cause:

  • Using enumerations to represent Errors is inadvisable, as if new errors need to be introduced they cannot be added to existing enumerations. This leads to a proliferation of Error enumerations. "Fake" enumerations can be made using structs and static lets, but these do not work with the nice Error pattern-match logic in catch blocks, requiring type casts.
  • Using enumerations as tagged unions to allow multiple types to be passed to a single argument slot is not a good idea unless the library author is confident that no new types will be added. This forces developers into the pattern of overloads instead. Doing this on functions is ok, but doing it with properties or getters is very painful, as this can lead to confusion in the type system.
  • Using an enumeration to refer to a group of possible ideas without entirely exhaustively evaluating the set is potentially dangerous, requiring a deprecate-and-replace if any new elements appear.
  • Using an enumeration to represent any concept that is inherently extensible is tricky. For example, SwiftNIO uses an enumeration to represent HTTP status codes. If new status codes are added, SwiftNIO needs to either mint new enumerations and do a deprecate-and-replace, or it needs to force these new status codes through the .custom enum case.

This proposal plans to address these limitations on enumerations in non-resilient Swift.

Proposed solution

This solution would add a new attribute to Swift, analogous to @frozen. We propose to name this attribute @extensible. When applied to a public enum, this will cause the behaviour of the switch statement in calling code to be the same as if the module that provided the enum was compiled in library evolution mode. Specifically, this will issue warnings if there is no @unknown default clause provided, and will not lead to compile failures if new clauses are added to the enumeration in the presence of such a clause.

In essence, this attribute will cause the compiler to treat an enum annotated with @extensible to be as though it were defined in a resilient library for the purposes of switch statements. It will change nothing else about its representation: these enumerations will continue not to be ABI stable in the absence of -enable-library-evolution.

Source compatibility

Adding new attributes is not a source breaking change. This proposal does enable the ability to make more non-source-breaking changes for non-resilient libraries.

Effect on ABI stability

This attribute does not affect the ABI, as it is a no-op when used in a resilient library.

Effect on API resilience

This proposal only affects API resilience of non-resilient libraries, by enabling more changes to be made without API breakage.

Alternatives considered

Resolving the dialect issue

The Core Team has made it clear that they are opposed to language dialects in general. One may then conclude that the Core Team would ideally like for the current language dialect situation to go away. In order to do this, one of the two language dialects must undergo a source-breaking change, as their default enum exhaustibility behaviour will have to change.

Source-breaking changes are extremely heavyweight. It is likely that no further source-breaking changes will be accepted until Swift 6, which has no concrete timetable. For that reason, attempting to resolve this discrepancy by resolving the dialect issue cannot happen until Swift 6 until the very earliest: a time that does not currently concretely exist.

Additionally, it's not clear to the authors exactly which direction the language dialect would be resolved in. Will resilient libraries have enums default to exhaustive? This seems like a deeply problematic outcome, as it's non-obvious to library authors (who always exhaustively switch over their own enums) and so will likely not be noticed until users complain about source breaking changes. If this did happen, of course, an attribute would be needed to mark an enum as extensible: we could therefore add that attribute now.

Alternatively, the non-resilient dialect could be changed to make all enums extensible unless marked otherwise. This is easier to motivate, as the downsides of forgetting to mark your enum @frozen is user-annoyance, not user-breakage. Additionally, making a non-frozen enum @frozen is not source-breaking. This is a plausible alternative to this proposal, and could be pursued by the community in lieu of adopting this work.

Versioning

SE-0192 notes that:

Earlier versions of this proposal included syntax that allowed all public Swift enums to have a frozen/non-frozen distinction, rather than just those in the standard library and overlays. This is still something we want to support, but the core team has made it clear that such a distinction is only worth it for libraries that have binary compatibility concerns (such as those installed into a standard location and used by multiple clients), at least without a more developed notion of versioning and version-locking. Exactly what it means to be a "library with binary compatibility concerns" is a large topic that deserves its own proposal.

This is addressing a specific problem with having extensible enumerations. When switching over all cases, even with @unknown default, the compiler will warn if a known case is not explicitly handled. This means that code written against library version X.Y will emit warnings when compiled against X.Y+1 if X.Y+1 introduces new enum cases. For users that compile with warnings-as-errors, this is annoying. When paired with the absence of @available guards for non-OS Swift modules, this essentially makes it impossible to use warnings-as-errors if you are depending on Swift libraries that use extensible enums.

This problem currently exists today for all non-@frozen public enums defined in non-OS libraries compiled with -enable-library-evolution. However, making extensible enumerations available to non-resilient libraries without supporting version guards will definitely make this problem more pervasive. It's the opinion of the author of this pitch that this is an acceptable trade-off, but the community may well disagree.

To avoid this issue would require a much more substantial change than this proposal envisions, requiring extensions to the Swift module system to include a notion of versioning, as well as extensions to the availability system and the Swift Package Manager. Such a wide-ranging change is not to be entered into lightly!

32 Likes

Thanks for putting this together @lukasa . In my mind it's absolutely clear that we need this ASAP. So +100.

4 Likes

Seems to be a problem worth addressing.

Wouldn't it fit better to add a source-only resilience mode? With the proposed approach if you later switch to binary resilience mode it'll change what enums need an unknown default (those with no attribute at all would flip over to the other side).

The question for a source-only resilience mode is what it covers. If it only covers enums, then I think that we are discussing what I mentioned in my Resolving the dialect issue section, though by adding a third dialect. This strikes me as the kind of thing we'd only do as a stop-gap with a concrete plan to make it the default.

However, yes, we could do that too. It's just more work than this pitch. :wink:

+1 this is definitely a problem, and a real pain to work around. There's a reasonable amount of boilerplate to be written in the case of errors, which end up being tedious to switch over since they require up-front knowledge of the possible types to cast to. It's bad for library authors, but probably worse for the library user.

+1, I've bumped into the same quite a bit where we had to "hide the real enum" because someone might switch over it, so there has to be a 1:1 mirror of all the cases but exposed publicly to users as functions. With this proposal the enum could be simply marked as extensible and we would not risk source breaking users by exposing such enums.

I can see this appear in a number of places: defining error cases, or settings which a library naturally wants to express as enum but can't really "because someone might switch over it" leading to much boilerplate code to "hide the enum".


Minor typo perhaps:

should this be @unknown default: try ...?

1 Like

Yup, good catch, thanks. Amended in my local copy and above.

1 Like

+1

I'm +1 on this, but I wonder if a protocol might be better than an attribute. Something like this (please don't looks too closely at the names):

protocol HasUnknownCase {
  static var unknown: Self { get }
}

Then, a switch can know if the value it is switching over conforms to this protocol, which could even be a conditional conformance (though I can't think of an example where this is needed off the top of my head).
Another benefit is that you can handle the unknown case anywhere in your switch: (before, the default case duplicated the logic in .pepperoni)

switch pizzaFlavor {
case .hawaiian:
    throw BadFlavorError()
case @unknown .unknown, .pepperoni:
    try validateNoVegetariansEating()
    return .delicious
case .cheese:
    return .delicious
}

An added benefit would be that this protocol could be used in conditional conformances to provide different behavior if an enum is "frozen" or not even outside a switch. One example of this is decoding: GitHub - airbnb/ResilientDecoding: This package makes your Decodable types resilient to decoding errors and allows you to inspect those errors.

I am fairly strongly disinclined to invent a new system for this. SE-192 defined a system for enum extensibility already, and provided machinery for controlling it with an attribute: it just didnā€™t provide for doing this in non-resilient libraries. Itā€™s a substantially broader change to try to provide new machinery via a ā€œmagicā€ protocol, and I donā€™t think itā€™s worth the added complexity.

4 Likes

Do you think there should there be a way to opt-out of the warning (for missing @unknown default) even when the enum is marked @extensible? I understand we generally don't want to have fine-grained functionality for enabling/disabling warnings, but I can see a certain subset of users be willing to update their switch statements when upgrading between library versions, because that might be preferable to throwing errors (or doing something else) for unhandled cases.

For example, if I am maintaining several modules in a monorepo and those modules are rev-locked, I might want external clients to use @unknown default but internal clients to keep matching exhaustively.

1 Like

You donā€˜t have to take my post seriously, but it would be kinda cool if the attribute was an antonym of frozen such as hot or unfrozen. :hot_face: Other than that I think itā€˜s a good idea to support both variants to help library authors.

1 Like

Yeah, this is a bit awkward. There is no such functionality for doing this today across Swift modules. We could arrange to do so, but Iā€™d want to do so holistically across both resilient and non-resilient modules, and so I kinda think itā€™s out of scope for this pitch.

I think this shows that we might've made a bad call with diverging the behavior between resilient and non-resilient libraries here. I support adding an attribute to allow for explicitly opting enums into extensible behavior. There could also be an opportunity for us to eliminate the language dialect here in Swift 6, and make it so that the next breaking change of the language always defaults enums to non-frozen behavior in all modules.

12 Likes

I agree with @typesanitizer that there's something interesting to investigate in the idea of version-locked vs. non-version-locked dependencies, thoughā€”the choice about whether to version-lock is usually up to the client, not the library vendor. This could also enable more optimization when a client app (i.e. an end product) bundles a module with it that has library evolution enabledā€”it can make more assumptions because its dependency is bundled. (This has come up before but I'm not sure what terms to search for.)

I do see that there's a gap here, but I'm not convinced that adding an opt-in for extensible enums is the right way to go, as opposed to something like what @Joe_Groff is saying. Maybe it should be a warning to have a missing @unknown default when the dependency is not resilient, and you can opt out of that warning by saying the dependency is "rev-locked" or "shipped with the product", either through an attribute on the import or a command line flag passed by SwiftPM or Xcode.

(As to why this wasn't considered during the initial round of SE-0192, that proposal was contentious enough, and did generate a bunch of annoyance from users about what it did start requiring.)

4 Likes

I agree, deciding whether modules are "version-locked" or not sounds like a better basis for the effect we were originally trying to get with the design here than library-evolution vs not. That would still support the "typical app developer" use case, where someone uses modules only for code organization still gets exhaustivity checking for enums for free without added ceremony, while allowing for uniform behavior for libraries that use modules for versioned distribution, whether they do as resilient binaries or source packages.

2 Likes

Presumably this would bring @frozen to non-resilient libraries, such that developers can say that specific enumerations will never be evolved, regardless of whether the dependencies are version-locked.

3 Likes

I feel like it would make more sense to make this a compiler error. When I use the -enable-library-evolution compiler parameter when compiling a library, that is me (the user) saying 'I know that an enum could add a new case later'. But when a developer adds @extensible to an enum, that is the developer telling the user 'hey, I'll probably be adding more cases down the road', and so I think it makes sense to force the user of that library to handle that case, instead of potentially ignoring it and having their code randomly break when they pull a new version of the library.

If you want to argue that in the end, it's the user's fault if they ignore the warning, I'm fine with that also.

Apologies if it was already mentioned and I missed it. Do I understand correctly that this would allow us to fix the core librariesā€™ issue where theyā€™re not entirely sourceā€compatible with themselves on different platforms simply because some use library evolution and others donā€™t. Here are some old threads describing what Iā€™m talking about:

I agree - whether or not an enum in a library will evolve to add cases is a decision for the library (and it makes sense that it should default to yes/maybe). The client may decide that it doesnā€™t care about new cases in new library versions and regain exhaustive switching.

That really does make the most sense, and I think itā€™s an overall simpler model for users to understand. Itā€™s also nice that it applies module-wide rather than on a per-enum basis.

The only place where it gets hairy is if we want to allow third-parties to add cases to enums (which IIRC was something we werenā€™t sure about in the initial proposal). I would say it's not worth supporting that usecase. It would mean that even the declaring module canā€™t exhaustively switch over its own enum, so we would need some kind of special annotation for it, and if youā€™re doing that anyway, you could always add a .custom(Any) case to get the same behaviour.

1 Like