Backwards-deployable Conformances

Evolution Discussion
1

This particular feature only applies to libraries shipped with Apple OSs, but that does include the Swift standard library, so it's relevant to the Swift Open Source project. And it does involve new syntax in the language.

Background

Let's say it's suddenly vitally important that Int conforms to Collection.

extension Int: Collection {
  public var startIndex: Int { 0 }
  public var endIndex: Int { self.bitWidth }
  public subscript(index: Int) -> Int { (self >> index) & 1 }
}

What's the problem? Well, to start, none of these accessors exist in iOS 13, which means that if someone tries to use this new conformance in iOS, their app will just crash when running on iOS 13. That's a limitation of shipping Swift as part of the OS.

There are two ways to resolve this today: @available with a set of minimum OS versions, or @_alwaysEmitIntoClient. But that still doesn't stop someone from writing this code...

let someInt = 42
myArray.append(contentsOf: someInt)

...and trying to run that on iOS 13, where the conformance doesn't exist. If the append(contentsOf:) is not optimized away, this will cause problems.

In theory, we have the same two options for how to deal with this: restrict the use of the conformance with availability, or emit the conformance into client binaries. I'll discuss each of those in turn and then a secret third option that I think is the best choice.

Conformances with Availability

Pros:

  • Supports witnesses (the types and members that satisfy requirements) that have their own availability constraints. Types can't be backwards-deployed at this time, so if a protocol adopter's associated type AssocImpl has newer availability than either the protocol or the adopter, the conformance necessarily can't be backwards-deployed before when AssocImpl was introduced.

  • On the implementation side: enforcing this might have been hard...if we hadn't just implemented it for implementation-only imports (still not a finalized feature). Instead, this'll "just" be another check in the same locations.

  • More on the implementation side: the only compiler changes required in SIL/IR layers are to weak-link conformance symbols.

Cons:

  • Can't backwards-deploy, of course.

  • If a client had its own implementation of a conformance, the current compiler rules will ignore the client's implementation in favor of the library's, even when backwards-deploying. This is how the standard library was able to make String conform to Collection even when third-parties had added their own conformances (have I mentioned that retroactive conformances are problematic?)

Emit-into-client

Pros:

  • Can backwards-deploy.

  • The implementation is basically just pretending there was a retroactive conformance written in the client module.

Cons:

  • Doesn't support witnesses that have availability themselves.

  • The "inlinable" problem: there might be multiple conformances floating around that are subtly different. (This is already true of explicitly-written retroactive conformances, but still.)

  • Code size implications: every client will have a copy of the conformance for all time.

"Why Don't We Have Both?"

For backwards-deployable conformances, we can actually combine both approaches, resulting in an annotation that means "if you're on a new enough OS, use the library's copy of the conformance; otherwise, use a local copy". I'd call this a "polyfill" approach.

Pros:

  • Supports witnesses with availability.

  • Supports backwards-deployment without the costs when not backwards-deploying.

Cons:

  • More complicated to implement (all of the availability work, plus extra work).

  • Slight run-time cost, to check for the canonical conformance symbol before falling back to the local copy. (I hope there are no direct references to conformance symbols in data sections?)

  • Doesn't completely solve the inlinable problem when running on older OSs (one client could have a copy of the conformance from Library v6 and another from Library v7, and it's possible they're different).

The work

1. Add attribute syntax for conformances

(needed for all three approaches)

I propose that this be written as 
extension Int:
  @foo Equatable,
  @bar Collection {
}

that is, the attributes for a conformance go before the name of a protocol in an inheritance list. Some design questions:

  • Should this be allowed for names that refer to protocol compositions? (Probably not.)

  • Should this be allowed for protocols written directly on the type, or only in extensions? (Type is probably fine too.)

  • Should we apply the restrictions from conditional conformances where parent protocols have to be written explicitly? That is, should Int have to explicitly conform to Sequence as well? (I'm leaning towards "no"; it'd get too noisy and we plan to keep improving the swift-api-digester tool.)

  • How do we disambiguate type attributes and conformance attributes in this position? (Assume conformance attribute, since you can always put parentheses around types, and type attributes are super rare today.)

This is not being proposed here, but there have been some other things we've been wanting this attribute syntax for. The main one is determining whether a conformance is "devirtualizable" or not—that is, whether the current set of witnesses is guaranteed to stay the same (or at least remain correct) from release to release. Currently this is based on whether the conforming type is frozen, which...doesn't really make sense. This actually has very similar semantics to @inlinable on functions, but I'm not sure whether that's a clear enough name to reuse for conformance devirtualization.

2. Add availability to conformances

(needed for the first and third approaches)

This has a few sub-tasks, but is mostly pretty straightforward:

2a. Parse availability attributes in conformance position, and preserve them through serialization.

2b. Validate that all witnesses used for a conformance are supported given the conformance's availability. (This is already happening based on the protocol and conforming type's availability.)

2c. Check availability at all conformance use sites.

2d. Change IRGen to weak-link conformances based on deployment-target. (This should already be happening in some form, but it'd need to check a different availability.)

3. Add "polyfill" support

(third approach only)

The main thing this needs is a spelling, which I don't want to bikeshed at the moment. We need some way to communicate "this conformance is valid all the way back to iOS 11, but clients need to keep their own copy if they're not deploying to iOS 13". Once we pick a spelling, we have some work to do:

  • Sema: make sure the compiler checks the right availability!

  • IRGen: emit references to the conformances as preferring the weak-linked symbol in the library and falling back to a copy in the client. (Or doing some kind of uniquing like the type metadata for imported C types.)

Alternative designs

Emit-into-client approach instead

The simplest thing to implement would be emitting a conformance from a library as a retroactive conformance in every client that needs it. That's implementable and has some behavior, even if it's going to have rough edges (especially around dynamic casting). However, it doesn't solve all the problems, particularly the case where a witness cannot be backwards-deployed, and so I think we'd need to do conformances-with-availability at some point anyway. At that point we'd end up with two different mechanisms to accomplish the same thing, just like we have with functions but harder to reason about. So I think it's worth doing the bigger thing now, even if we don't get to the polyfill support.

Polyfill without preferring the library version

This is roughly how @inlinable works today: there may be many versions of a conformance floating around including the one in the library. Clients would still save on code size when not backwards-deploying, and it'd be a little simpler to implement, but I like preserving the ability of the library author to change the witnesses used in the conformance on newer OSs, perhaps to fix bugs. Maybe that's too subtle, though—the new witness still has to be backwards-deployable too.

Thoughts?

25 Likes
2

I don't have an opinion on the larger tradeoff but have a couple comments.

In general this seems ok, but when you consider something like Codable I'm less sure.

Agree, but I think an argument can be made that Codable is similar, despite using a different language mechanism.

This is just food for thought. Feel free to take it or leave it.

3 Likes
3

Still thinking about it but my first impression is that I really like this idea.

4

Thanks for writing this up! At first glance, I like the idea of extending availability to cover conformances, and then doing some kind of polyfill.

I do want to address this point, from your 'cons' list:

I think this is the right tradeoff to make, because this runtime penalty should be quite small and will be acceptable in most domains. As long as it's not too easy to accidentally use polyfills without realizing it, it doesn't seem unreasonable to recommend developers of performance-critical code stick to using the set of conformances shipped in the oldest OS version they target.

5

Unfortunately I'm not sure how you'd guard against it! You could do something like an opt-in warning, but anything more precise than that (like "am I only using this within an availability context that's new") would be trickier to implement. It'd be a valid future direction, though.

6

This alternative option seems pretty appealing, being easy to understand and operating much like @inlinable. It would be nice to get a pro/con list here versus the other polyfill option because I don't think I fully understand the tradeoffs.

7

Let's see…

"Dynamic" polyfill Inlinable-style "compile-time" polyfill
Conformance emitted into client when deploying to old OSs Conformance emitted into client when deploying to old OSs
No cost when deploying to new OSs No cost when deploying to new OSs
Client checks for library symbol before using its copy of the conformance Client always uses its copy of the conformance, even on new OSs
Different clients may have different versions of the conformance when running on old OSs Different clients may have different versions of the conformance if any have old deployment targets
Library can change which implementation satisfies a requirement on a newer OS Old implementations will continue to be used, even on newer OSs, for any clients with old deployment targets

Incompatible conformances can result in strange behavior around as? in obscure cases, which makes me feel weird. (As an example, Set<Foo> in one library might be a different type from Set<Foo> in another library if they're using different implementations of Foo: Hashable.) We already have that problem with retroactive conformances, but I wouldn't want to make it be something that can happen with just a single library. On the other hand, some people (@Joe_Groff in particular) have been on the side of embracing alternate conformances and maybe eventually providing some way to be explicit about them.

They're really pretty close though.

3 Likes
8

FWIW, I am definitely on this side. Since retroactive conformances already make this possible it would be best to be explicit. My understanding is that would eliminate the potential for surprising issues. As a bonus, having the compiler better understand multiple conformances could help enable things like private conformance to a public protocol, etc.

9

I forgot to mention that conformances with availability open us up to another problem, originally described in docs/LibraryEvolution.rst in the Swift repo. I'm reworking that document to describe what is implemented instead of what could be implemented, so I'm reposting it here:

// Library, version 1
class Elf {}
protocol Summonable {}

// Client, version 1
class ShoemakingElf : Elf, Summonable {}

// Library, version 2
@available(dishwasherOS 2.0, *)
extension Elf : Summonable {}

Now ShoemakingElf conforms to Summonable in two different ways, which may be incompatible (especially if Summonable had associated types or requirements involving Self).

Additionally, the client can't even remove ShoemakingElf's conformance to Summonable, because it may itself be a library with other code depending on it. We could fix that with an annotation to explicitly inherent the conformance of Summonable from the base class, but even that may not be possible if there are incompatible associated types involved (because changing a member typealias is not a safe change).

One solution is to disallow adding a conformance for an existing protocol to an open class. I don't think people will be happy with this.

Note: Objective-C has the first part of this problem, where a subclass implements a protocol differently from a superclass, but because the conformance itself doesn't have a runtime presence beyond the protocol being listed in the class's metadata, the superclass and subclass conformances can coexist as long as they agree on the implementation.

10

This all stems from the original decision to have subclasses inherit the protocol conformances of their superclasses, if I understand correctly. I do recall that at least one core team member was mentioning that this is something that they regret.

Is it now baked into the ABI? Or is there room to evolve the design so that protocol conformances are implicitly inherited unless explicitly overridden? Then, we could say that explicit conformances to a protocol in a subclass disables (for itself and any of its subclasses in turn) any inheritance of such a conformance (retroactive or otherwise) from a superclass.