Pitch: Library Evolution for Stable ABIs

[jrose (Jordan Rose)]

It's not reflected in the mangling of the type because we don't want generic types to change, which makes things tricky around, say, Optional<FrozenInALaterRelease>. But maybe we can come up with something.

[Slava_Pestov]

@Michael_Ilseman

Do you have any inkling of what this might look like? Is @frozen -or-not is present in mangling? If not, we might be permanently closing the door to this.

The plan was to add a @frozen-with-availability attribute. The calling convention would still have to pass @frozen-with-availability structs indirectly, but inside a sufficiently constrained availability context, the compiler will still be able to make assumptions about the struct's layout.

[Slava_Pestov]

@jrose

Both of these should just say "at compile time". We're not talking about the layout algorithm in this case, and there is no run-time data layout impact of @frozen .

That is not quite true. A frozen enum can only make use of spare bits if all payload types are recursively frozen in all resilience domains that can see the enum. A non-frozen enum can make use of spare bits even if payload types are resilient ,as long as they're defined in the same module.

For example:

public class C() {}
public struct StoresReference { let c: C }
public enum NonFrozenEnum {
  case first(StoresReference)
  case second(StoresReference)
}
@frozen public enum FrozenEnum {
  case first(StoresReference)
  case second(StoresReference)
}

The first enum is 8 bytes, the second enum is 9 bytes.

[Slava_Pestov]

@jrose

I guess I don't really see it as a separate restriction from "you can't have a custom setter".

I'm not sure what that means. We allow computed properties in frozen structs, and you can simulate an observed stored property with a private stored property and public computed property wrapping it. I guess I just don't see the point.

[Karl]

The thing that I wonder is: is resilience an intrinsic property of the library, or is it an access pattern for clients?

Take a system library like Core Animation for example, and let's imagine it was written in Swift. As part of the operating system, I suppose it should be compiled with resilience/evolution mode 'on', right?

Now consider a 3rd-party application which uses Core Animation. It might end up linking against CA v1.0, 1.1 or 2.0 at runtime, so it will have to avoid assuming specific binary details of the library, like struct and enum sizes. It uses the resilience/indirection mechanism described here, and all is good.

Now consider another system library which depends on Core Animation, such as UIKit. Every version of UIKit targets one and only one version of Core Animation (let's say. I guess it is possible for Apple to update the libraries independently, but that is a very deliberate action by the maintainer of the overall bundle of libraries). It is perfectly fine for it to assume specific struct/enum sizes, even if they are not @frozen in general, and it never needs to handle unknown default cases in enums.

I've read the proposal quite a few times now, and I see this pattern come up quite often. Take the section about making binary-stable mode the default, for instance - it says the downside with such an approach is that "there is no benefit to handling structs and enums indirectly". Ah, so it's about how you handle values of those types. That's a client problem. It makes me think that resilience/indirection should be an import-level modifier.

If we accept this proposal as written, would it be possible to override the default resilience? So, assuming some resilient, Swift version of Core Animation as above, would a Swift version of UIKit be able to import it non-resiliently and bypass the indirection?

[jrose (Jordan Rose)]

:-) You've found the concept we've been calling "resilience domains": libraries that are distributed in binary form but which can rely on another binary library having a specific version. There's some description of what those might look like in docs/LibraryEvolution.rst (note: this doc is overdue for a cleanup and update, so it's not necessarily the best resource at the moment). In short: yes, this is doable, but we don't quite know what we want it to look like yet.

That said, it's not entirely a client problem, because it does change the calling conventions on the library side. It's not really feasible to offer both "frozen" and "non-frozen" entry points for everything because you run into combinatorial explosion when you have multiple parameter types, each one from a different module. It may be the case that there's a balance to be struck, paying some additional code size and metadata in exchange for some additional efficiency from clients that can assume a particular version of a library, but we want to start with the most general form.

[bobergj (Jonas)]

@Ben_Cohen

Marking a struct @frozen only guarantees that its stored instance properties won't change.
...
However, it does not provide a handful of other guarantees that a C struct might:

• It is not guaranteed to be "trivial" (in the C++ sense ). A frozen struct containing a class reference or closure still requires reference-counting when copied and when it goes out of scope.

That said, the compiler is allowed to use its knowledge of the struct's contents and layout to derive any of these properties. For instance, the compiler can statically prove that copying Point is "trivial", because each of its members has a statically-known type that is "trivial"

Isn't it interesting to have a separate attribute @trivial, to be able to avoid reference counting for non-frozen structs? @trivial would not make the struct "trivial", but assert to the compiler "This struct will always be trivial", and gives an error if the compiler can see that it is not actually trivial. The change of removing @trivial would Affect ABI.
@trivial would also be useful as an assertion outside of -enable-library-evolution mode.

Edit: I think trivial should not be in quotes in the proposal. The term is already used elsewhere as if it was an established Swift concept, for example in the UnsafeMutablePointer API documentation: initialize(to:) | Apple Developer Documentation

• The fields are not guaranteed to be laid out in declaration order. The compiler may choose to reorder fields, for example to minimize padding while satisfying alignment requirements.

I think the compiler doesn't do this optimisation yet ([SR-3723] Determine if the layout algorithm for structs should layout fields in a different order than declared · Issue #46308 · apple/swift · GitHub). I suggest it should do at least some kind of reorder, even if not the optimal one in Debug mode as soon as possible, given this will break existing code. For example I see a lot of Metal code out there with Swift structs assumed to have identical layout to a C-struct. Even if that is technically illegal Swift now, stuff that many programs rely on tend to turn into features.