1

Hi Swift Community,

The review of SE-0260: Library Evolution for Stable ABIs begins now and runs through May 21st, 2019.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to me as the review manager via email or direct message on the forums. If you send me email, please put "SE-0260" somewhere in the subject line.

What goes into a review of a proposal?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift.

When reviewing a proposal, here are some questions to consider:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

As always, thank you for your contribution to making Swift a better language.

John McCall
Review Manager

12 Likes
2

This is an important step for Swift; this document proposes a sensible approach.

My one concern with the design is the following:

Turning on library evolution mode will have the following effects on source code:

  • The default behavior for enums will change to be non-frozen. This is the only change visible to users of the library, who will need to use the @unknown default: technique described in SE-0192.

Ideally, a compilation flag wouldn't produce a change visible to users of the library, for the obvious reason that it may cause authors to inadvertently break user code.

Because we are introducing a new compilation mode, there's the opportunity to do what we could not do in SE-0192:

If we require every enum in library evolution mode to be explicitly @frozen or @nonfrozen, then there would be no invisible changes in default behavior for users of the library when the mode is switched on. (Implicit here would be the addition of @nonfrozen allowing enums to opt into requiring @unknown default in all modes. Likewise the same could be offered for other types to let users to opt into inlinable-to-noninlinable delegation checking in all modes.) Has consideration been given to such a design?

3

One of the conclusions I've come to since SE-0192 is that it's actually going to be very rare for someone to take an existing library and convert it over to this mode. That'll happen right now, of course, since Once this mode exists, it's the kind of thing where you'd know up front if you needed it: are you planning to ship updates in a way that needs binary compatibility, or no?

Your question's a good one. For me, though, I remain convinced that biasing towards "non-frozen" is a good idea in general. Last year I did an audit of every Objective-C enum in Apple's public SDK, and very few of them actually made sense to adopt NS_CLOSED_ENUM (the Foundation equivalent of @frozen). Swift's enums are different, of course, since they take the place of unions as well, but even then I think it's reasonable to have a default rather than to require that every enum be explicitly annotated. (Or, to put it another way, public is already an explicit annotation, so @nonfrozen public doesn't add sufficient value.)

5 Likes
4

(non-review-manager hat)

You mention in the proposal that Swift is free to give frozen types a layout that doesn't match the naive C layout. Clearly, Swift wants to be able to rearrange type layouts, but if a type's layout was frozen in a previous binary release of the library, Swift must match the layout rules that were used in that release. Agreement can be achieved simply by saying that frozen types must always use the "baseline" ABI-stable layout rules, but then frozen types can never benefit from layout optimizations, even if they are introduced/frozen long after the optimization was added. This means that (in a hypothetical but plausible future) freezing a type can create an unnecessary performance trade-off: it'll allow clients to use the type more efficiently, but the type might take more space than it did before.

Taking a non-frozen type and freezing it in a later version of a library seems like a mandatory feature in the long term. To do this without breaking ABI, you will need to have some way of communicating the freezing library version to the compiler. If we can further associate that library release with a minimum version of Swift that's necessary to use it (or at least to take advantage of any newly-frozen types in it), then we can associate the freezing with a future Swift version and therefore take advantage of the layout rules of that release.

How much thought have you given to these issues? Are you convinced that we will be able to address them adequately in a future proposal, or will we have boxed ourselves in?

9 Likes
5

I really like this resilient value type (struct) idea.
It can be placed into stack and consequence memory in array still it has dynamic size.
This feature well reflects characteristics of swift language which has both flexibility of source code and runtime performance especially compared to other compile language such as C/C++/Rust.

6

  • What is your evaluation of the proposal?
    Agree with the proposal as written. Though curious of what use cases might be foreseen for the @available(*, frozen) syntax.

  • Is the problem being addressed significant enough to warrant a change to Swift?
    Yes.

  • Does this proposal fit well with the feel and direction of Swift?
    Overall yes, having resiliency concerns limited to libraries compiled in an explicit mode falls nicely in line with progressive disclosure.

  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
    N/a

  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
    Was a part of discussions way back with SE-192. Mostly kept up since then. Read the proposal in full.

7

One other piece I'm curious about is this line "This implies that clients must manipulate fields and enum cases indirectly, via non-inlinable function calls." What are the associated runtime costs of this indirection?

8

It's great to see this proposal make it this far. I've got one small note about the document itself and then will answer the questions.

The proposal has the following sentence in the second paragraph of the Proposed Solution section:

When a library is compiled with library evolution mode enabled, it is not an ABI-breaking change to modify the fields in a struct (to add, remove, or reorder them)

This needs to be clarified, because a cursory read may incorrectly conclude that you can remove public struct fields, which you absolutely cannot. Based on the table below what is meant is that fields that are either private or internal without @usableFromInline can be removed without breaking ABI, but this could do with clarification in this section of the document to avoid confusion.

In favour. I'm pleased to see the ongoing formalisation of the Swift stable ABI process, and I think this leads to a positive change for the Swift community.

Yes. Swift has made it a core goal of the language to support a stable ABI, and the benefits of that substantial effort should be able to accrue to the community as well as to Apple. For those frameworks that need a stable ABI, this is a vital step.

By and large, yes. The flip in the behaviour of enums is a bit strange, and it remains a frustration that it is not possible to declare non-frozen enums in libraries without a stable ABI. However, the change in behaviour is clearly necessary, and I understand the desire to avoid source-breaking by flipping all enums to non-frozen in a future Swift release.

While this proposal gamely compares Swift's stable ABI evolution to a number of other languages, it's my genuine belief that there is no other language with a feature quite like this. C's stable ABI is exceedingly simple, and essentially all other languages roughly follow it where they can and simply choose not to stabilise where they cannot.

The scope of this work is breathtaking.

More directly, I've used C a great deal, and am accustomed to the indirection-based approach to library evolution that is being emulated here. This model seems easy enough to understand and hold in your head, and it's mostly fairly simple to relate it to what happens in C.

Close reading of the post, plus closely following the evolution of the stable ABI.

4 Likes
9

The compiler will gain a new mode, called "library evolution mode"

I'm not sure I like that name - it's not very descriptive. Since it introduces a level of indirection to allow binary-level changes, I would rather call it "indirect mode".

Try it out in a few contexts. I think it's a lot easier to understand:
"Libraries which are accessed indirectly may also expose direct access to other libraries in the same resilience domain", etc.

10

I don't love the name "library evolution mode" because it is hard to use in other contexts, but it exactly describes what the mode is for. Not all of the changes require or are related to indirection (such as the restriction on inlinable struct initializers), and not all indirection the compiler does is in service of evolving libraries without breaking binary compatibility.

EDIT: Also, being a library-side decision rather than a client-side decision makes a difference. To me, "accessed indirectly" sounds like there's an opportunity for "well, how do I access it directly, then?" Which is a risky door to open—if you do that for the standard library on macOS, for example, your app won't run on any version of macOS but the one whose SDK you built against. Even point releases could break you.

1 Like
11

I wish we could show some concrete numbers for this, but the answer is really "it varies drastically based on your workload". It's the moral equivalent of the "opaque struct" idiom in C:

void MyOpaqueValueCopy(MyOpaqueValue *uninitializedDestination,
                       const MyOpaqueValue *source);
void MyOpaqueValueDestroy(MyOpaqueValue *value);

On their own, these operations aren't too bad. Their real cost is that they keep the optimizer from doing higher-level transformations, since they're opaque function calls that could change the state of the world. If the client workload you're testing is mostly just calling into the library anyway, it's probably not going to be something you notice, but if you have something like "add up a bunch of CGVectors", which could just be a simple loop over addition (or even a vectorized loop), a non-frozen CGVector would be orders of magnitude slower. So it really comes down to how your types are used and what your perf testing shows.

For enums, at least, the story is usually much simpler: either you're going to add cases in the future, or you can guarantee that you won't.

4 Likes
12

To reinforce this: we spent a long time combing through the standard library and some overlays deciding which types should be frozen and which not. During this time, there was no universal rule of thumb as to whether something would benefit from being frozen or not, because, as Jordan says, it is down to the interaction between the code and the optimizer. It is sometimes surprising on both the upside and the downside whether it matters. The same goes for whether or not to make something inlinable. It's a combination of judgement and experimentation via benchmarks.

2 Likes
13

This is a complicated feature and there are always going to be places where selective quoting gives a misleading impression of the full proposal. Personally I think the explicit clarification shortly later in the table is enough.

14

I feel it would be less confusing for library authors who stumble upon this, and wonder whether it applies to them, if it somehow communicated that this concerns libraries with stable binary interfaces. Longer-form version of "library evolution mode" from the intro paragraph:

the flexibility to add to their public interface, and to change implementation details, without breaking binary compatibility

I.e. evolution for libraries with a stable binary interface. Libraries without a stable binary interface, i.e. a stable source interface (all libraries prior to ABI stability) are every bit as evolvable without this mode.

The compiler will gain a new mode, called "library evolution mode". This mode will be off by default. A library compiled with this mode enabled is said to be ABI-stable .

Missing from this is whether a library that doesn't build with this mode could be ABI-stable in the absence of changes. That is, if a library is not built with this mode, but never changes any code in it and only adds new properly-versioned types and functions, would it have a stable binary interface? My understanding is no, because struct layout in a future compiler could change and without this mode all struct layout is ABI.

So this is about building stable binary interfaces and this mode includes Swift's novel approach to making such interfaces more resilient by default. I realize that evolution is the really cool and novel feature in Swift compared with other languages, but I think something emphasizing ABI-stability communicates the "what does this do?" and "does this apply to me?" more effectively to library authors.

4 Likes
15

That's right. A few lines down in the proposal (added since the pitch where this also came up):

Binaries compiled without this mode should not be declared as ABI-stable. While they should be stable even without library evolution mode turned on, doing this will not be a supported configuration.

16

If I understand this correctly, what you are saying here would mean that any Swift library distributed as a binary but built without this flag is only valid to use with a specific version of the Swift compiler. Is that correct? If so it basically means that a library distributed as a binary has to be ABI stable because you can't prevent clients from upgrading their version of the Swift compiler.

17

I saw that tucked away, but the salient function of this mode is to build an ABI-stable library, with evolvability a (very neat!) feature of ABI-stable libraries. I feel like this is brushed aside as some minor detail, when it is in fact the point of this mode.

2 Likes
18

(non-review-manager hat)

This ties in with my post above. A binary library needs to be distributed with some sort of description of its interface. It would be reasonable to have that description say something like "this uses Swift 8 ABI rules for everything". You wouldn't be able to compile against that interface with a Swift 7 compiler, but it's (hopefully) always reasonable to ask your clients to use a newer compiler, and the opposite direction presumably has to work: unstable libraries aside, a Swift 17 compiler must still know what the Swift 8 ABI rules were because any given ABI-stable library might have a mix of types that were frozen using different ABI rules. So there are convergent benefits from planning for this kind of ABI-heterogeneity.

1 Like
19

This is also my biggest concern. I realize that this is future work, but I would like to see some thought or guidance on how this could play out. There's more to versioning and availability than just an OS version. Nothing is mentioned about it in the "Future directions" section.

20

I don't mind requiring a minimum version of Swift for a given binary release. I would actually expect this. Will the compiler validate it and refuse to build if it doesn't understand the ABI rules used by a library the user is attempting to use?

This is what I was hoping to hear. From what you have written, it sounds like a library distributed as a binary would continue to work fine with future versions of the Swift compiler. If it is distributed with an app and everything upstream is rebuilt when the app is built then there isn't a need to declare ABI stability. Is this understanding correct?

What I am trying to understand is exactly where the line is between safe and dangerous (undefined behavior) uses of a binary library that does not use -enable-library-evolution. This line is still not 100% clear to me.