Swift.org blog: Library Evolution in Swift

[tkremenek (Ted Kremenek)]

There is a new blog post on Swift.org titled "Library Evolution in Swift" that talks Swift's library evolution capabilities:

Library Evolution in Swift

The post is written by @Slava_Pestov. Please feel free to ask questions about the blog post here!

[Nevin]

Typos

Swift 5.1 shipped with two new features related to binary stability that enables binary frameworks that can be distributed and shared with others:

Should be “enable”.

(And as a style point, it probably reads better if the first “that” becomes “, which”.)

• • •

A switch over a frozen enum is considered exhaustive if all cases are covered by the switch, whereas a switch over a non-exhaustive enum must always provide a default or @unknown case.

I believe “non-exhaustive” should be “non-frozen”.

• • •

This means that clients normally importing the framework remains binary compatible with a new version built for testing.

I’m not entirely sure what needs to change, but the grammar seems off here. It might be as simple as turning “remains” into “remain”, or “clients” into “a client”.

• • •

This ensures that while library evolution support can increase code size, it does impact the cache locality of data.

I suspect there’s supposed to be a “not” in there somewhere.

• • •

Currently its existence is an implementation detail, but a pitch to add modify accessors to the language is currently making its way through the Swift evolution process.

The second occurrence of “currently” seems redundant.

Formatting

The code-coloring of “associatedtype” in the blog post is weird. The first two letters are pink, while the rest of the word is black. It ought to be all pink since it’s a keyword.

• • •

The first occurrence of the Temperature example has all the lines of code indented by 2 extra spaces. (The other code examples don’t.)

• • •

I’m a bit confused the by Temperature example:

  public struct Temperature {
    public var celsius: Int
    public var fahrenheit: Int {
      get { (celsius * 9) / 5 + 32 }
      set { celsius = ((newValue - 32) * 5) / 9 }
    }
  }

The implementation of a public declaration can be changed, as long as the new implementation is compatible with existing expected behavior. For example, a function’s body might be replaced with a more efficient algorithm producing the same result. Or, a stored property can be changed into a computed property, as long as the computed property has the same observed behavior.

For example, the following implementation of Temperature is binary compatible with the one we saw earlier:

public struct Temperature {
  public var celsius: Int {
    get { ((fahrenheit - 32) * 5) / 9 }
    set { fahrenheit = (newValue * 9) / 5 + 32 }
  }
  public var fahrenheit: Int
}

Assuming we have a var t: Temperature, with the first version:

t.celsius = 2
print(t.celsius)  // “2”

But with the second version:

t.celsius = 2
print(t.celsius)  // “1”

Is this kind of observable behavior change really permitted?

• • •

I also have a general question:

• A superclass can be inserted between two existing classes.

• Changes to a protocol’s list of refined protocols are not allowed.

What’s the difference between a class hierarchy and a protocol hierarchy, which makes it possible to resiliently insert a new class into the middle of an existing hierarchy, but not possible to insert a new protocol into the middle of an existing hierarchy?

• • •

Finally, the blog post says:

An important concept is a resilience boundary .

But it never provides an actual definition for what that term means. After introducing the term, the text makes use of it repeatedly when listing things that do and don’t work across a resilience boundary, yet there is no clear explanation of where such a boundary is found.

[scanon (Steve Canon)]

@Nevin

Assuming we have a var t: Temperature , with the first version:

t.celsius = 2
print(t.celsius)  // “2”

But with the second version:

t.celsius = 2
print(t.celsius)  // “1”

Is this kind of observable behavior change really permitted?

Definitely. Library maintainers should do their best to communicate such changes to clients, but this does not change the API or ABI of the type, and preserves source and binary compatibility. If you weren't able to make changes like this, you wouldn't be able to fix bugs.

[Nevin]

@scanon

Definitely. Library maintainers should do their best to communicate such changes to clients, but this does not change the API or ABI of the type, and preserves source and binary compatibility. If you weren't to even make changes like this, you wouldn't be able to fix bugs.

Okay, then I think the text of the blog post needs to change to reflect that, because right now it says (emphasis mine):

The implementation of a public declaration can be changed, as long as the new implementation is compatible with existing expected behavior. For example, a function’s body might be replaced with a more efficient algorithm producing the same result. Or, a stored property can be changed into a computed property, as long as the computed property has the same observed behavior.

[benrimmington]

Do the following examples imply that @frozen classes are supported?

Examples of resilient changes

• Members can be added to class, struct and enum types as long as the container type is not declared @frozen. If the type is @frozen, stored properties or enum cases cannot be added. Any other kind of member can be added without restriction.

• ABI-private members can be removed from class, struct and enum types, provided the container type is not @frozen. If the type is @frozen, stored properties or enum cases cannot be removed. Any other kind of member can be removed without restriction.

Examples of non-resilient changes

• Adding or removing the @frozen attribute on a struct, enum or class is not allowed.

[Slava_Pestov]

@benrimmington

Do the following examples imply that @frozen classes are supported?

They're supported but for some reason we never documented them as part of the @frozen evolution proposal. A frozen class has a stored property layout known to the compiler, just like a struct. The vtable layout remains opaque, so you can add and re-order virtual methods (and remove non-public virtual methods).

[Slava_Pestov]

@Nevin

What’s the difference between a class hierarchy and a protocol hierarchy, which makes it possible to resiliently insert a new class into the middle of an existing hierarchy, but not possible to insert a new protocol into the middle of an existing hierarchy?

The metadata of a class type consists of the concatenation of the metadata of the "immediate members" of each of its superclasses, followed by the "immediate members" of the class itself. The members in this case are the generic requirements, field offsets for stored properties, and implementations of virtual methods.

When you call a method on a class instance, or define an extension of a generic class, the runtime has to know where to find the immediate members in the class metadata. This is not known at compile time because the superclass could add (or remove) members. So the way this is implemented is by defining a global variable for each class that stores where the immediate members begin. This global variable is initialized when the class metadata is initialized.

With this model, adding new members to a superclass is actually more or less equivalent to inserting a new superclass in the chain. The immediate members of each subclass 'slide down' and the runtime still knows where to find them.

Protocol refinement is quite different. When a type conforms to a protocol the compiler emits a witness table. The witness table has a specific "shape"; it begins by referencing the witness table for the type's conformance to each refined protocol, followed by associated type metadata, followed by the implementation of each protocol requirement. While accesses to the associated type metadata are handled by runtime functions and calls of protocol requirements are handled by dispatch thunks, the layout of the refined protocols section is known to the compiler. @Douglas_Gregor can fill in the details here since he worked on it last. I don't know if its possible to lift this restriction or not. When I originally implemented protocol resilience in Swift 4.0 or so, we couldn't add associated types either, but now we can. So maybe Doug added some affordances to add new refined protocols in the future. (And even if it does become possible you will need default implementations for every method defined in the protocol you inserted).

[Slava_Pestov]

@Nevin

Is this kind of observable behavior change really permitted?

First of all thank you for pointing out the typos and formatting errors. I'll push a new version soon.

As for the temperature example, I guess it is kind of silly. Since we're using an Int to store the temperature instead of some hypothetical exact rational type, we have this problem where the conversion between units is lossy. You could mentally substitute a more reasonable example yourself :-)

[Nevin]

@Slava_Pestov

You could mentally substitute a more reasonable example yourself :-)

Sure, but my question was about the information being communicated by the blog post.

Is the intention to say, as the text implies, “The observable behavior must not change”, or is it as the example shows “The observable behavior may change”?

[Slava_Pestov]

@Nevin

Is the intention to say, as the text implies, “The observable behavior must not change”, or is it as the example shows “The observable behavior may change”?

The intent is the former. I honestly forgot that the formulas I picked are not 1:1 over the integers.

[Slava_Pestov]

@Nevin

The code-coloring of “ associatedtype ” in the blog post is weird. The first two letters are pink, while the rest of the word is black. It ought to be all pink since it’s a keyword.

I believe this is a bug in our markdown formatter. I'm just using the ordinary code block syntax set to Swift mode, and not highlighting the keywords manually or anything.

[Nevin]

@Slava_Pestov

The intent is the former. I honestly forgot that the formulas I picked are not 1:1 over the integers.

How does that comport with Steve’s point about fixing bugs?

[Slava_Pestov]

@Nevin

How does that comport with Steve’s point about fixing bugs?

I mean, the post is about detailing what changes are API and ABI compatible. If the author of a framework wants to change the observed behavior of an API, that's their prerogative, and it might be justified if they're fixing eg, an incorrect result or a crash. Nothing in the implementation of library evolution prevents you from shooting yourself in the foot by changing observed behavior in an unsound way (whatever that might mean for any given framework and situation).

[jrose (Jordan Rose)]

@Slava_Pestov

They're supported but for some reason we never documented them as part of the @frozen evolution proposal. A frozen class has a stored property layout known to the compiler, just like a struct. The vtable layout remains opaque, so you can add and re-order virtual methods (and remove non-public virtual methods).

IIRC we (you and Ben and I) didn't bring it up in the proposal because it wasn't clear if it was needed, and I (specifically) didn't want to get into how it would interact with inheritance. (Can a frozen class inherit from a non-frozen class? How about the other way around?) It then turned out that we did need frozen classes for some very specific cases in the standard library, and those are still marked with the old spelling of the attribute: @_fixed_layout.

EDIT: We also didn't know if people would find it confusing that vtables are still laid out dynamically, i.e. @_fixed_layout doesn't currently opt out of all library evolution support for classes. I do think it's the right trade-off but someone might be unhappy with it.

I'd suggest updating the blog post for now, and even if you / the team decides to make it a supported feature with the semantics it has today, it should still get a token review.

<stdin>:1:1: error: '@frozen' attribute cannot be applied to this declaration
@frozen public class X {}
^~~~~~~~

[scanon (Steve Canon)]

Generally speaking (as someone who maintained an OS library for a decade), my viewpoint is that you should be able to depend on the following in a resilient library:

• forward binary compatibility (a binary linked today will continue to work on future OS/library versions).
• backward binary compatibility (a binary linked today can be deployed to older OS/library versions, subject to availability restrictions).
• forward source compatibility (code that compiles today will continue to compile against future OS/library SDKs, though API may be removed following a long period of deprecation).
• documented behavior is stable (unspecified behavior may change or become specified, because binaries linked in the past cannot depend on the previously undocumented behavior).
• behavior that is not documented (likely including the specific rounding behavior of conversion between temperature units) is subject to change.

Slava is focusing on the interaction of the resilient library with the rest of the system (i.e., the first three points). It's up to library authors or platform owners to document their policy for handling the last two points, however.

[omochimetaru]

Development history section is very fun.
I knew the story of ABI stability starting from Swift 3.0.
During development of large new concept,
using early implementation within standard library
before roll out as official feature seems to be practical.
Thanks to publish interesting article.

[Slava_Pestov]

Oops, I forgot we still reject @frozen when applied to classes. Alright, I'll remove the vague allusions to frozen classes from the blog post.

[Karl]

Nice post!

While we have binary-level backwards and forwards-compatibility, we're still missing a story for source-level backwards compatibility (i.e. using new functionality only if the library version >= X). I hope that's something we can tackle soon.

Currently we use Apple OS versions as a proxy for that, because Apple's libraries don't use SemVer (in fact, they can't even agree on a type for version numbers - sometimes it's a Double, other times an Int32). I don't think that situation really serves anybody well. It doesn't even serve Apple well; try to develop an App for one of the less-popular platforms such as tvOS or watchOS, and you'll find that hardly any of your dependencies bother to annotate availability on those platforms, even if they could support them.

I hope we start considering that side of things soon. If I really do depend on a binary library which is distributed and updated separately from my App, there's a good chance I might also need to support multiple versions of it with something better than the lowest-common-denominator interface.

[scanon (Steve Canon)]

@Karl

Try to develop an App for one of the less-popular platforms such as tvOS or watchOS, and you'll find that hardly any of your dependencies bother to annotate availability on those platforms, even if they could support them.

Every API in the SDK should have availability annotations; if they are missing anywhere, that's a bug. But note that for API added before the first version of tvOS or watchOS, availability can be inherited from the iOS availability information, and API may also be unannotated if it is unconditionally available. Is it possible those cases are what you're talking about?

[LucianoPAlmeida (Luciano Almeida)]

Amazing post @Slava_Pestov!
One point that may be interesting to address here is how one library developer that enables library evolution, can be able to automate the detection source compatibility breaking changes and can verify that changes across versions wouldn't break ABI for the clients?
I remember back some time ago we were having a discussion about it in an opensource project issue on how to do this and by then we even come across the swift-api-digester, but as far as I remember we couldn't find much information about it was exactly what we were looking for...

That being said, I think this is an interesting topic for bring-up on this post :))