Handling future cases of enums in libraries without binary stability concerns

Evolution Pitches
1

Hi swift-evolution,

I'd like to pitch a proposal that is basically SE-0192, but for library authors that don't care about binary stability.

I've complained about the problem earlier, and now I'd like to suggest a solution.

Handling future cases of enums in libraries without binary stability concerns

Introduction

SE-0192 introduced a mechanism to add new cases to enums defined in the standard library and overlays in a source-compatible way. SE-0260 later lifted this restriction, allowing to use this mechanism in all libraries built in library evolution mode.

We propose to remove this restriction completely by:

  • allowing all library authors to mark their public enums as @frozen;
  • warning the clients that exhaustively switch over a non-frozen enum declared in another module, suggesting them to add an @unknown default case.

Motivation

First, let's agree on terminology. We'll call a library with binary compatibility concerns (and therefore built in library evolution mode) a resilient library. Other libraries are referred to as non-resilient.

Suppose we're developing an open source library, and we are not planning to distribute it in binary form (for example, we want to use Swift Package Manager). A part of our library's public API is an enum with several cases, and we want to be able to add new cases to that enum without breaking the code that depends on our library.

public enum Unit {
	case minutes(Int)
	case seconds(Int)
	case milliseconds(Int)
}

Right now we can't do it, unless we build our library in library evolution mode. So, the clients of the library that exhaustively switch over the enum will stop compiling, should we ever add a new case. This is unexpected, since we already have a way to handle unknown enum cases by using @unknown default. Suppose our client knows about this and adds this case to their switch:

public func printUnit(_ unit: Unit) {
    switch unit {
    case . unit minutes(let n):
        print("\(n) minutes")
    case .seconds(let n):
        print("\(n) seconds")
    case .milliseconds(let n):
        print("\(n) milliseconds")
    @unknown default:
        print("unknown")
    }
}

The client will get a warning saying that the default case will never be executed. There is a client-side workaround to avoid the warning and breakage in case a new case is added:

public func printUnit(_ unit: DispatchTimeInterval) {
    switch interval {
    case .seconds(let n):
        print("\(n) seconds")
    case .milliseconds(let n):
        print("\(n) milliseconds")
    case .microseconds(let n):
        print("\(n) microseconds")
    case .nanoseconds(let n):
        print("\(n) nanoseconds")
    default:
        if case .never = interval {
            print("never")
        } else {
            print("unknown")
        }
    }
}

This workaround accomplishes the goal, but has two disadvantages:

  • When a new case is added, the client will have no way of knowing that they should fix their code to handle the new case. If the client used @unknown default, they would get a warning (rather than an error) that suggests adding a missing case;
  • As already mentioned, this workaround is client-side. There is no way for library authors to enforce this.

This is weird: even if we only want better source stability, we have to build our library in library evolution mode, which is all about binary stability.

This also has a very unpleasant side-effect: since there is no notion of resilient libraries on non-Darwin platforms like Linux and Windows, all the enums declared in swift-corelibs-foundation and swift-corelibs-libdispatch are frozen. This affects users that want to write cross-platform code: if they switch over an enum like DispatchTimeInterval that is non-frozen on Darwin, and they use @unknown default case, they get a warning on Linux, but not on Darwin. If they don't use @unknown default, they get a warning on Darwin, but not on Linux.

Proposed solution

  1. Make all public enums in non-resilient libraries non-frozen by default.
  2. Allow authors of those libraries to mark their public enums @frozen to indicate that no new cases will be added.

Detailed design

This section is excatly the same as in SE-0192.

Here we list some enums that would benefit from this proposal:

  • All enums in swift-corelibs-foundation and swift-corelibs-libdispatch that are non-frozen in Foundation and Dispatch overlays on Darwin platforms, including: Calendar.Component, Calendar.Identifier, Calendar.MatchingPolicy, Calendar.RepeatedTimePolicy, Calendar.SearchDirection, Data.Deallocator, DispatchData.Deallocator, DispatchIO.StreamType, DispatchPredicate, DispatchQoS.QoSClass, DispatchQueue.AutoreleaseFrequency, DispatchTimeInterval, JSONDecoder.DataDecodingStrategy, JSONDecoder.DateDecodingStrategy, JSONDecoder.KeyDecodingStrategy, JSONDecoder.NonConformingFloatDecodingStrategy, JSONEncoder.DataEncodingStrategy, JSONEncoder.DateEncodingStrategy, JSONEncoder.KeyEncodingStrategy, JSONEncoder.NonConformingFloatEncodingStrategy, POSIXErrorCode
  • WebSocketErrorCode from swift-nio
  • Logger.MetadataValue and maybe Logger.Level from swift-log.

Source compatibility

Users that exhaustively switch over a public enum declared in another module will get a warning that will suggest adding a @unknown default case.

So, this change is completely source-compatible, unless you use the -warnings-as-errors flag.

Effect on ABI stability

This change affects only non-resilient libraries and therefore has no effect on ABI stability.

Effect on API resilience

Marking a public enum @frozen is a source-compatible change. Users that used @unknown default case when switching over a formerly non-frozen enum will get a warning "default case will never be executed".

Removing the @frozen attribute from a public enum is also a source-compatible change: users that exhaustively switch over a formerly frozen enum will get a warning that suggests adding an @unknown default case.

Alternatives considered

We could leave it as-is. For the reasons described in the Motivation section, this is suboptimal.

8 Likes
2

My main point of confusion here: Isn't this already handled, in a different way, by using semantic versioning? Adding a new case to an enum currently requires a major version bump, so you won't break existing code. Does this proposal boil down to wanting to add new enum cases without requiring a major version bump?

3

Yes. But not completely.

It also mentions swift-corelibs-foundation and swift-corelibs-libdispatch that suffer from this limitation, and they are special in that sense: their API must be the same as in Darwin's Foundation and Dispatch, and they share the same source compatibility concerns as their Darwin siblings. So, if, say, Apple decides to add a new case to the DispatchTimeInerval enum on Darwin and ship it with Swift 5.2, the same case must be added to swift-corelibs-libdispatch and also shipped with Swift 5.2 (because swift-corelibs-libdispatch aims to be compatible with Darwin Dispatch). But here's the thing: the source-compatible change on Darwin turns out to be source-breaking on non-Darwin platforms. And this can't be worked around using SemVer, because, well, Foundation and Dispatch don't use SemVer.

1 Like
5

Sorry, I've misunderstood initial problem.

Does that mean that if some project don't want to have @unowned default all over the code, they need to mark all enums as frozen?

6

Not all enums, but only the public ones declared in a different module.

For example:

// Module A

public enum Unit {
	case minutes(Int)
	case seconds(Int)
	case milliseconds(Int)
}

func printUnit(_ unit: Unit) {
    switch unit {
    case . unit minutes(let n):
        print("\(n) minutes")
    case .seconds(let n):
        print("\(n) seconds")
    case .milliseconds(let n):
        print("\(n) milliseconds")
    }
    // No need for @unknown default: we're in the same module
}

// Module B

import A

func printUnit(_ unit: Unit) {
    switch unit {
    case . unit minutes(let n):
        print("\(n) minutes")
    case .seconds(let n):
        print("\(n) seconds")
    case .milliseconds(let n):
        print("\(n) milliseconds")
    @unknown default: // This is needed since the enum is declared in a different module
        print("unknown")
    }
}

There are really two cases here:

  • Your project consists of multiple internal frameworks, and those frameworks are not used outside of your project. In this case, you will want to mark the enums as @frozen. If a new case is added, your project will stop compiling, and you can then fix your switches all at once.
  • Your project is a package that you want to distribute as source code, and the package is divided into multiple modules. An example of this design is swift-nio. Suppose one of your public modules (let's call it B) depends on another public module A defined in the same package, and there's a public enum Unit in A, which is not only used by B, but also by the clients of your package. You don't want to make it @frozen (otherwise you'll break your clients if you add a new case), but you also want to be able to exhaustively switch over the enum in B. Then you'd have to use @unknown default, probably fatalErroring in it.
2 Likes
7

That will result in a a lot of source changes for project with many modules ā€“ I've grep public enum in our, and there are over a thousand entries. I don't think that it is a very optimal solution.

But I see your problem, what about some new attribute, for example @extensibility(open)? And @frozen is just an alias for @extensibility(closed)? Compiling in library-evolution mode would result to all public enums be open by default, unless specified otherwise. Compiling in normal mode would result to all public enums be closed (frozen) by default, unless specified otherwise. Or, you could specify a special flag to compiler, what to use by default for public enums, like -fvisibility=hidden/default in clang.

btw, @extensibility(closed/open) name is taken from clang attributes: Attributes in Clang ā€” Clang 9 documentation

1 Like
8

This could possibly be a better angle to work on.

9

Iā€™d like to highlight for anyone who didnā€™t read the original thread that the impetus for this came from the very incongruous and annoying behaviour in the core libraries:

As it stands, libraries containing unfrozen enumerations have incompatible APIs when built with and without library evolution mode. Throughout the reviews of enumeration stability and library evolution, I (and presumably others) didnā€™t bother considering the case of the same library needing both modes. It seemed an unlikely distribution scenario. But I was forgetting that due to the disparity between platforms, every library that is distributed in evolution mode for Darwin must still be distributed in normal mode for all other platforms. And that makes even the core librariesā€™ APIs sourceā€incompatible with themselves (at the warning level) between platforms.

@broadway_lamb has expanded the proposal to involve extending the use of @frozen and @unknown to apply API instead of ABI implications outside of library evolution mode, but my original suggestion was much smaller. For anyone providing feedback, it would be helpful if you could evaluate the pitch not only as a whole, but in pieces. Even if you happen to think the proposal as a whole involves more change than necessary, are there parts of it that you could see as helpful? Or do you have alternate suggestions which would alleviate some of the underlying issues?

For me, the API usage of @frozen is interesting and might turn out to be useful, though it is also not something Iā€™ve ever found myself wishing for. So as far as it goes, I stand neutral.

On the other hand, the inability to switch over Foundation enumerations without incurring warnings on one or the other platform is silly and I think is absolutely worth resolving. The core libraries strive for parity, but this is the in compilerā€™s domain, so there is nothing they can do about it on their own.

  • @unknown:

    My original suggestion was simply to have the compiler allow @unknown default switch cases for any enumeration regardless of compilation mode. Right now it throws a warning that ā€œDefault will never be executedā€. That is true. However the fact that it is a warning not an error demonstrates the compiler is perfectly capable of compiling it with that thereā€”and presumably also optimizing it into oblivion. Due to very real presence of libraries that exist both with and without library evolution mode, the veracity of that warning relies on the current compilation environment. I think this warning is not helpful and should be removed.

    But then the question is how should the removal be done. Simply obliterating the warning and letting it compile as it currently does would suggest to users that @unknown is doing its job. But that is not true, it is only behaving as a normal default case. Adding a case to the enumeration does not trigger a warning about something being left unhandled. The easiest way to avoid confusion and reduce related user mistakes would be to let @unknown do its job regardless of frozenness. And that inevitably results in a design very near what was pitched above. The only difference is that in normal mode all enumerations are still frozen by default, so the @unknown default is only necessary if the library is sometimes in evolution mode. If you never compile against a stable edition of the library, youā€™ll never be asked to add it and you wonā€™t have to care about frozenness, just like today.

  • @frozen:

    This hasnā€™t bitten us in the Swift project itself yet, because the core libraries are built from different source code on Darwin vs Linux. But someday they may be. And now that library evolution mode is available to everyone, there is the real possibility of the same source code aiming to support evolution mode for Darwin and normal mode for Linux at the same time. Right now declaring something as @frozen and then compiling in normal mode results in a warning: ā€œ@frozen has no effect without -enable-library-evolutionā€. Again the warning is true, but not exactly helpful, since it depends on the compilation environment.

    Iā€™m less sure what to do about this one.

    • Should it not warn at all? Or would that make it look like it is having some effect?
    • I am a fan of the fact that enumerations are currently frozen by default. But if it didnā€™t warn, the source would seem to imply the opposite. Would it make it look as though unfrozen enumerations were a thing too? There are already users with that misconception even with the current warning.
    • Is the cleanest solution just to let it behave exactly like library evolution mode? That is essentially the design the pitch above landed on. But it would mean giving up on frozen by default. Do I like that? Not particularly, but I could live with it.
    • Would it be better to add an @unfrozen attribute, and have @frozen and and @unfrozen work the same in both modes, but let unmarked enumerations continue to use the same differing defaults? That is less source churn, and enables all the same features. But it is also more confusing.

    I donā€™t really know what would be best. But at least it is less important than @unknown, since it only affects the library itself, and not the librariesā€™ clients.

  • For the sake of completeness, I read over the other differences library evolution mode causes. But from what I can see, none of the others cause discrepancies. Library evolution imposes some more restrictions, but except for @frozen and @unknown, anything that compiles under library evolution will also compile under normal conditions. So these are the only two things that cause irreconcilable source differences.

3 Likes
10

I think this is a problem worth solving. API incompatibilities in libraries like Foundation and Dispatch between Apple platforms and other platforms are bad because they create language dialects.

Another point I wanted to highlight is that even libraries that do not want to commit to strong source stability guarantees would still benefit from being able to signal to their users about whether some enum is intended to be open or closed. Sure, libraries without source stability guarantees can add cases to enums that they marked as closed previously, but if they do, they probably will not do so often, and will have a very good reason to do so. Similarly, libraries are very likely to add new cases to enums that they have marked as open, signaling to the user that doing exhaustive switches over them, while possible (because there's no source stability), is going to create maintenance work.

In other words, in the world of libraries that don't provide ABI stability, source stability is not all-or-nothing. It is a spectrum, and libraries often make calls to break source stability on a case by case basis.

3 Likes
11

I just ran into what I consider a very straightforward reason to support both evolution and non-evolution mode even just on Darwin: a library which is distributed as either a xcframework or a swift package. xcframeworks require library evolution mode and SPM doesn't support it, so you unavoidably have a useless warning.

4 Likes
12

Thomas pointed out a common situation that may apply to lots of framework developers. We need to write and test the package using the source code, and distribute, test, and use it as an xcframework or binary Swift package. When the client code grows larger, there are so many unknown default warnings that they overshadows the other "real" warnings.

Hope to at least have a flag to suppress this type of warnings. :pray:

1 Like