Retroactive Conformances vs. Swift-in-the-OS

Evolution Discussion
21

FWIW Swift optimizes away the wrapper type too, at least in terms of manipulating values of the type. A struct with one stored property is layout-equivalent to a value of that type.

10 Likes
22

Oh, didn't know that. Thanks. :)

23

The former doesn't account for conformances being newly introduced in OS releases, and the latter doesn't handle the String(describing:) case, which deliberately does the check/conversion far away from the definition of the type.

24

That seems like a principled version of the "fallback" annotation, yeah, and we already have something like it for @available(swift, …) (a static check rather than a dynamic check). I'm a little worried about non-iOS platforms, though. Normal @available has that * in it very deliberately, to say that the declaration is available by default on all other platforms. This one might have to be hidden by default.

25

IMO this has always struck me as an inadequate workaround. I still want to be able to work with a Set<MyIntWithCustomHashing> as if it were a Collection of Int, not of my wrapper type, but in order to do so, a wrapper type imposes a bunch of mapping in and out of the wrapper. And although Collection nicely provides pre-fab map operations to do that for you, other abstractions may not have a map operation at all. Swift's generics model was designed to accommodate "incoherent" and "orphan" protocol conformances, and in the fullness of time I'd like to see the language design embrace it.

That said, I agree with Jordan's assessment that his proposed rule seems prudent given that we have a lot of compiler implementation and language design work still to do to make such conformances practical to work with, not only theoretically possible to.

1 Like
26

To me, it seems reasonable to say that non-canonical conformances (meaning conformances that don't belong to either the type's or protocol's module, or in the fullness of time, conformances that are local/private/intentionally duplicated) don't participate in dynamic casting at all. In Swift we for the most part do a good job of modularizing dynamic behavior so that loading a library can't retroactively alter the behavior of code in the app or other libraries, and such a rule with protocol conformances would be in line with that philosophy.

It may also be worth having a marker for protocols whose purpose is intentionally to act as dynamically-queried customization points, for a number of reasons. First of all, if we adopted the rule about non-canonical conformances above, we'd want to be able to warn about conformances to them that will be ignored. We've also seen in practice that users tend to misuse these protocols as generic constraints, calling the customization points directly instead of using the intended general interfaces, and it'd be nice to be able to provide a helpful warning like "CustomStringConvertible is intended only as a customization point; did you mean to use string interpolation instead of .description?" or something along those lines.

6 Likes
27

Whether it "handles" those cases depends on your expectations. If we choose the answer I proposed, you can't ever dynamic cast to a public existential protocol with certainty that the cast will succeed. I don't think that's unreasonable, and I don't think the real effects of making such a choice are necessarily worse than giving up retroactive conformance.

Stepping back, I think you are assuming a set of answer acceptability criteria (such as "introducing a new conformance shouldn't change the behavior of code"—I don't think you hold that one specifically; it's just an example) that you haven't spelled out here. We should have a discussion about what criteria are important for evaluating solutions.

6 Likes
28

Question: why can't the version be used to retrieve the dispatch of the type? So in this case, code which linked against the framework in iOS 12.x does not see the CustomStringConvertible conformance in the library, so there is no ambiguity in use. Another system framework linked against it in iOS 13 does see the system conformance (that was expected because it was linked against that library), so it would not pick up a third party conformance.

When you attempt to build an iOS 13 build against both the system libraries and the third party conformance, it fails due to ambiguity. In other words, you have runtime resiliency but not necessarily compile time compatibility.

29

If I understand it correctly, I guess it would break the system framework. Unless the dispatch checked the source of the caller.

Example:

let myStruct = SomeStruct()
let stringConvertible = myStruct as? CustomStringConvertible

If the app is linked against iOS 15, according to what you suggest, stringConvertible should still fail on iOS 16, right?

The issue is if this code is on iOS 16 within the iOS framework, it would fail as well? :thinking: There'd need to be separate dispatch/witness tables for each loaded binary?

1 Like
30

I followed @dabrahams' advice and wrote up a list of possible goals for the final model of "retroactive conformances and resilient libraries". (Some of them are contradictory.)

  1. A library owner can update one of their types to newly conform to a protocol (theirs or another library's), and that conformance will be used within the library (including for dynamic casts). They don't have to worry about this conformance being superseded.

  2. A library owner can apply their protocol to another library's type, and that conformance will be used within the library (including for dynamic casts). Same as (1) but the library owns the protocol and not the type.

  3. A library owner can update one of their types to newly conform to a protocol, and that conformance will be used for all dynamic casts.

  4. A library owner can apply their protocol to another library's type, and that conformance will be used for all dynamic casts. Same as (3) but the library owns the protocol and not the type.

  5. A client that defines a retroactive conformance can use that conformance within their own module in static ways.

  6. A client that defines a retroactive conformance can use that conformance within their own module X and any modules that import X in static ways.

  7. A client that defines a retroactive conformance can use that conformance within their own module for dynamic casts.

  8. A client that defines a retroactive conformance can use that conformance everywhere for dynamic casts, as long as there isn't another conformance (retroactive or non-retroactive) elsewhere in the program.

  9. If a library owner provides a conformance that's only available on, say, iOS 16, a client can provide their own retroactive conformance that's used prior to iOS 16.

  10. Dynamic casts behave the same everywhere in the program. All that matters is the dynamic type of the value being casted and the type that's being casted to.

  11. There should not be two implementations of a generic type with the same generic arguments in the program (because it's additional language complexity).

  12. We should allow two implementations of a generic type with the same generic arguments in a program (because it's useful).

  13. Conformances should be able to "shadow" other conformances, meaning that a client module can use a retroactive conformance (at least in static ways) even if the library provides a non-retroactive conformance.

  14. Conformances should not be able to "shadow" other conformances, because that can affect interactions with the outside world (think about Codable).

  15. Casting a generic type up to Any and back down to the "same" generic type elsewhere in the program should always work.

  16. If a retroactive conformance behaves differently from a non-retroactive conformance, it should be annotated somehow (probably with an attribute).

Definitions:

  • "library": shorthand here for "a library with binary compatibility concerns", which today means "the stdlib and overlays"

  • "client": a module that imports a library with binary compatibility concerns. May itself be a library.

  • "static ways": relying on a conformance to satisfy generic requirements / associated type requirements, or to convert a value with a concrete type to a value with protocol ("existential") type

  • "dynamic cast": checking whether a type conforms to a protocol at run time, via is or as? or as! or some other feature that has yet to be designed

Do you (all) think this covers it? What would you like to add to the list?

I think (1) and (2) are must-have requirements -- it seems unacceptable to me for a library author not to be able to depend as as? finding a non-retroactive conformance. (10) is a strong goal for me as well; otherwise, factoring generic code out into a separate framework might suddenly change the behavior of your program. (16) is important to me because otherwise I suspect people will write code happily assuming their retroactive conformance will be valid forever and in all situations, and then when whatever compromises we make come into play, apps will break. (11) and (12) are opposite opinions, of which I'm on the (11) side (and think we can solve the motivations for (12) in other ways).

10 Likes
31

I followed @dabrahams' advice and wrote up a list of possible goals for the final model of "retroactive conformances and resilient libraries". (Some of them are contradictory.)

  1. A library owner can update one of their types to newly conform to a protocol (theirs or another library's), and that conformance will be used within the library (including for dynamic casts). They don't have to worry about this conformance being superseded.
  2. A library owner can apply their protocol to another library's type, and that conformance will be used within the library (including for dynamic casts). Same as (1) but the library owns the protocol and not the type.

Almost there, Jordan! I think you need to nail down "used within the library." For me what's important is that the conformance used is determined where the concrete type is bound to the protocol

For example,

M0.swift:

protocol P { func f() }
struct X { }

M1.swift:

import M0
import M2
extension X : P { func f() { print(1) } }

extension P {
   g() { f() }
}

X().f() // 1
X().g() // 1
X().h() // 1 <===

M2.swift:

import M0
import M1
extension X : P { func f() { print(2) } }

extension P {
   h() { f() }
}

X().f() // 2
X().h() // 2
X().g() // 2 <===
2 Likes
32

We don't allow cyclic imports, so this isn't going to come up quite as you've written it. But yes, that's the behavior I expect for non-retroactive conformances in (1) and (2); having it also be true for retroactive conformances is (5).

(Note that I'm mostly ignoring conflicting retroactive conformances for now, like the one you have here. That seems like an additional level of complexity.)

33

For me I think the goals should be:

  1. 1 - 8 as is.
  2. For 9, a conformance added in a latter version of the OS that conflicts, I would say that should be an error.
  3. 10 as is.
  4. 11 rather than 12.
  5. 14 rather than 13.
  6. 15 yes.
  7. 16 should be an error.

And I would add an additional point, number 17, that if two retroactive conformances are in conflict; this should be an error, even if they are in different libraries/modules.

I realise that my list above isn't possible with the current linking arrangement. However I feel that changing the way Swift works so that installed apps are re-linked when sdlib etc. are changed would be a good addition to Swift.

34

I still need a clear definition for what you constitutes a conformance being "used" in order to evaluate your list.

(Note that I'm mostly ignoring conflicting retroactive conformances for now, like the one you have here. That seems like an additional level of complexity.)

Could you explain that? I don't see any substantive way it's "additional;" it seems to come up as soon as you allow any retroactive conformances to exist.

35

I very much agree that it is a desirable property. Practically speaking, though, your suggestion means that retroactively adding a CustomStringConvertible conformance would be almost useless. Is that something we're prepared to accept? I think I could live with it, FWIW.

The intention "this is only a customization point and not automatically part of the public API just because I publicly conformed to the protocol" is not at all tied to the use of dynamic checks and something that one often wants to express on a requirement-by-requirement basis. I'd totally support having a way to say that, but I'm not sure whether that's compatible with what you're suggesting.

Meanwhile, I've always said that protocols meant to be used as existentials (a prerequisite for the kind of dynamic check you're talking about) should be explicitly marked, to prevent clients from being inadvertently broken by the addition of requirements that have default implementations, which is otherwise not a binary-breaking change and only very rarely a source-breaking change.

36

Right, that's why I think it'd be useful to be able to warn about this. It's fundamentally the case that, with multiple conformances in a program, some of them will be useless as dynamic customization points, since the runtime will always be in the position of having to "pick one" without context.

I understand that. I was suggesting we have an annotation for protocols whose only stated purpose is dynamic customization, like the Custom* protocols in the stdlib.

37

And I understand that :slight_smile:. I'm just afraid that adding a specific annotation for that would leave no room for the distinct annotation I described, which IMO is more useful and general.

38

This is meandering from the topic at hand, but "meant to be used as an existential" would be a temporary measure, because eventually all protocols will be usable as existentials, and adding requirements wouldn't break that ability.

2 Likes
39

Are we proposing a change to protocols defined in all libraries? Or only resilient libraries? I've got a use case in my own library that sounds like it may be broken by some of the proposed changes here.

iOS has a notion of "extension-safe APIs", meaning those APIs which are safe to call from an app extension. If a library uses only extension-safe APIs, then it can be marked as an extension-safe library. If an extension target links against a non-extension-safe library, a warning is produced at link time.

I have a library that is used by both the main app and by extensions. I want to be able to call an extension-unsafe API in the library, but only if the current process is the main process. Currently, we are doing that like this:

// --------------- //
// In library code
// --------------- //

// Extension-unsafe APIs that the sync manager
// would like to use, if available.
protocol SyncManagerAppSupport {
    func registerForPushNotifications()
}

class SyncManager {
    func foo() {
        if let appSupport = self as? SyncManagerAppSupport {
            // This only happens if we are not in an extension
            appSupport.registerForPushNotifications()
        }
    }
}

// ----------- //
// In App code
// ----------- //

extension SyncManager: SyncManagerAppSupport {
    func registerForPushNotifications() {
        UIApplication.shared.registerForPushNotifications()
    }
}

This pattern means that the library can be marked extension-safe, and still take advantage of extension-unsafe APIs when available. But if retroactive conformances do not participate in dynamic casting, then it seems like the if let appSupport = self as? SyncManagerAppSupport line would fail.

There are other ways of achieving this same functionality, but I'm trying to understand whether the proposed behavior change would affect our code.

40

To not-meander, the more useful and general annotation I was referring to is not that, but the one that means "this is only a customization point and not automatically part of the public API just because I publicly conformed to the protocol."

3 Likes