Availability checking for protocol conformances

I just finished landing a set of changes which fix a hole in Swift's availability checking model. While this was mostly intended for use by Apple, I feel that it is also worth documenting and communicating to the community since people do ship third-party frameworks that make use of OS availability.

As most of you are aware, Swift's "availability" feature checks if referenced declarations are going to be available at runtime by comparing the OS version in an @available annotation with the "availability context" of the usage location, which is either the deployment target, an @available annotation on an outer declaration, or an if #available block:

@available(macOS 100, *)
func newFeatureInMacOS100() {}

func usesNewFeature() {
  newFeatureInMacOS100() // error!
}

@available(macOS 200, *)
func newerCodeOnMacOS200() {
  newFeatureInMacOS100() // OK!
}

func conditionalCodeOnMacOS200() {
  if #available(macOS 200, *) {
    newFeatureInMacOS100() // OK!
  }
}

While we have always been able to check the availability of referenced declarations, we didn't check the availability of protocol conformances. This came up in the following scenario. Suppose a framework defines a type and a protocol:

public struct Box< Contents > {
  var contents: Contents
}

public protocol Boxable {
  associatedtype Contents

  var contents: Contents { get set }
}

Now a newer version of the framework might add a retroactive conformance of the type to a protocol:

extension Box : Boxable {}

Because we only checked availability of declarations and not conformances, it was possible to use this conformance outside of the right availability context:

func takesBoxable<T : Boxable>(_ box: T) {
  print(box.contents)
}

func passBoxable<Contents>(_ box: Box<Contents>) {
  takesBoxable(box)
}

The passBoxable() function uses Box<Contents> and Boxable, two declarations which are unconditionally available. However, by passing the Box to takesBoxable(), it was referencing the protocol conformance of Box<Contents> to Boxable, which might not exist at runtime, since it was added after the fact.

The solution to this problem is to add an @available annotation on the extension defining the conformance:

@available(macOS 100, *) extension Box : Boxable {}

Now the compiler will complain:

box.swift:17:3: warning: conformance of 'Box<Contents>' to 'Boxable' is only available in macOS 100 or newer
  takesBoxable(box)
  ^
box.swift:17:3: note: add 'if #available' version check
  takesBoxable(box)
  ^
box.swift:16:6: note: add @available attribute to enclosing global function
func passBoxable<Contents>(_ box: Box<Contents>) {
     ^

As you can see above, this problem is currently diagnosed as a warning. I had to do this because some versions of Alamofire found in our source compatibility test suite violated the new restriction. The -enable-conformance-availability-errors frontend flag, which is off by default, upgrades it to an error.

If you encounter the warning in your own code once you upgrade to a Swift compiler release that includes the new check, please treat it as an error; violating the rule can result in linker errors or runtime crashes if the conformance does not actually exist at runtime.

I would prefer to tighten up the rule and turn the warning into an error eventually; either unconditionally once we feel we can do so without causing any disruption to community projects, or perhaps conditionalize it on a new -swift-version language mode, if we ever decide to introduce a new one.

One more thing: In addition to OS version availability, the conformance availability checking feature also supports unavailable conformances, which cannot be referenced anywhere:

@available(*, unavailable)
extension Box : Boxable {}

And deprecated conformances as well:

@available(*, deprecated)
extension Box : Boxable {}

These are diagnosed just like references to unavailable declarations and deprecated declarations, respectively.

Note that references to unavailable protocol conformances are always diagnosed as errors, even when the -enable-conformance-availability-errors flag is set. This is because I did not find any example code that violates this rule in our source compatibility test suite. If this becomes a problem, please let us know.

21 Likes

This is fantastic to see, excellent work @Slava_Pestov!

Is this mistake "safe" in the sense that users will always get a linker error/runtime crash when attempting to use a missing conformance at runtime, or is it possible that existing libraries with this issue are quietly misbehaving in undefined ways? If the latter, I think the case for an (eventual) unconditional source break is pretty strong. Though I have a stronger stomach for source breaks than many, I suspect.

Are such conformances discoverable dynamically via as?, or will unavailable conformances be absent at runtime as well?

It's not guaranteed that it will always fail. For example, the optimizer might elide the call, or the call might be devirtualized to call a concrete witness that is more available than the conformance itself. For example, consider this scenario:

struct MyType {}

protocol MyProtocol {}

extension MyProtocol {
  func doStuff() { ... }
}

func doStuffWithType(_ t: MyType) {
  t.doStuff()
}

@available(macOS 100, *) extension MyType : MyProtocol {}

Swift 5.3 accepts the above code, because MyType, MyProtocol and doStuff() are all available -- but when you run it on a deployment target that does not have the conformance, it will just call doStuff(), passing in a null pointer for the witness table (since it's a weak linked symbol). This might crash, do nothing, or produce undefined behavior.

You might recall a related issue that was one of the original motivations for explicit availability checking over checking if a type responds to a selector in Objective-C is that sometimes an internal API is "upgraded" to a public one in an OS release. The symbols might exist on an old OS, but they might have different behavior.

If the latter, I think the case for an (eventual) unconditional source break is pretty strong. Though I have a stronger stomach for source breaks than many, I suspect.

Yeah, I'm also very tempted to make it an unconditional source break, especially since Alamofire is the only project I found that was affected, and the latest version of Alamofire does not even define the protocol in question. However, breaking source compatibility of downstream dependencies is something that has caused folks pain in the past; it's not just a matter of updating your own project when the compiler changes, but also anything your project depends on.

Are such conformances discoverable dynamically via as? , or will unavailable conformances be absent at runtime as well?

That's a great point. I forgot about dynamic casts. I filed [SR-13864] Conformance availability vs dynamic casts · Issue #56262 · apple/swift · GitHub to track this issue.

5 Likes

Does this mean we can add new conformances in the standard library again?

IIRC this kind of evolution was blocked because we couldn’t express the conditional availability of those conformances.

7 Likes

I think I had encountered this error about a month back.

1 Like

Yep, this was in fact the primary motivation for this feature.

6 Likes

This doesn't allow for backward deploying said new conformances to older versions of Swift though. Is there behind the scenes work or any motivation to develop backward deploying conformances?

1 Like

There's no equivalent of @_alwaysEmitIntoClient for protocol conformances yet, which is what I think you're referring to. However some machinery that could be used for implementing this does exist, because we already need it for synthesized conformances on imported Clang types -- they're emitted at each point of use, and the witness tables are uniqued at runtime.

7 Likes

This is great!

We should add all the conformances that are currently missing from the stdlib. We're currently missing some basic things -- like Equatable/Hashable conformances for String's encoding views, or sensible CustomStringConvertible conformances for index types, etc. etc. etc. ad infinitum.

This work makes it possible to add these without worrying about newly written code that may accidentally try to rely on these in places where they aren't available.

Back deployment would allow newly written code to rely on these even if it needs to deploy on previous stable releases. For new code, conformance availability would be useful even it doesn't support back deployment -- this would "just" add a (big) delay before people can generally use these. I believe this is worse than regular function availability though, because people typically can't put things like stored property declarations behind a runtime availability check -- code deploying to 5.3 and below would be practically forced to continue declaring throwaway wrapper types just to be able to, say, create a UnicodeScalarView-keyed dictionary.

(@_alwaysEmitIntoClient has the problem that it always emits the implementation into the client module (d'oh), even if it has a recent enough minimum deployment target to be able to use a symbol. I hope the eventual non-underscored variant would allow us to declare symbol availability.)

However, we will also need to decide what to do about existing code (including existing binaries) that implement some of these conformances on their own, and back deployment could be far more important for these. We tried to discourage that on this forum, but the compiler doesn't currently emit a warning when code outside the stdlib conforms stdlib types to stdlib protocols, so we're facing a potentially significant source compatibility issue.

What's the migration path for a project that currently includes code like this?

extension String.UnicodeScalarView: Equatable {
  public static func ==(left: Self, right: Self) -> Bool {
    left.elementsEqual(right)
  }
}

I expect the duplicate conformance declaration will stop compiling when we add the same conformance to the stdlib. However, the stdlib's conformance will come with an availability declaration, so without back deployment, such projects will be forced to do a nontrivial rewrite to introduce a custom wrapper type around UnicodeScalarView.

(There is also the question what to do about code whose existing conformances conflict with the ones we'll add to the stdlib:

extension String.UnicodeScalarView: Equatable {
  public static func ==(left: Self, right: Self) -> Bool {
    String(left) == String(right)
  }
}

I think it'd be okay to have these suffer a source compatibility break, but we need to make sure to keep compatibility with binaries compiled with previously released stdlib modules.)

1 Like

I should mention at least one other limitation surrounding standard library protocol conformances.

Currently, attempting to replace a default implementation of a protocol requirement with a custom implementation after-the-fact leads to an LLVM compiler error, somewhat mistifyingly at the point of use:

So if we go around adding more conformances, we're going to have to make sure that we like any default implementations we drag along, because they will not be overridable later (until we fix this bug).

2 Likes

We could downgrade this to a warning. I believe we already did something similar to deal with 'extension String : Sequence {}' at one point right?

This bug might be specific to the code in question, and not a general problem with default implementations of protocol requirements. Can you try to reduce a minimal test case?

Can you type an example?

Karoy's reply has some examples: Availability checking for protocol conformances - #9 by lorentey

I don't know if it's appropriate to ask here, but is "SR-13859: Availability check fails on macOS Big Sur even if appropriate target is chosen." unrelated?
I mean is there possibility that implementation of this feature will resolve it?

Hi @Slava_Pestov,
I know that I may be a bit late to the party, but we just stumbled over an issue with the new conformance checks on protocols in context of an XCFramework that has been compiled with module stability. As you stated

Note that references to unavailable protocol conformances are always diagnosed as errors, even when the -enable-conformance-availability-errors flag is set. This is because I did not find any example code that violates this rule in our source compatibility test suite. If this becomes a problem, please let us know.
I'd like to find out what exactly is the error root cause in my case.

Setup:

  1. Compile Reachability.swift with enabled module stability with Xcode 12.1 and prepare XCFramework archive
    2a. Use this prebuilt binary in an app which is compiled with Xcode 12.4
    => OK
    2b. Use this prebuilt binary in same app which is compiled with Xcode 12.5
    => NOK ::
    .../DemoApp/BuildProductsPath/Release-iphoneos/Reachability.framework/Modules/Reachability.swiftmodule/arm64-apple-ios.swiftinterface:92:1: conformance of 'Reachability.NetworkStatus' to 'Equatable' is unavailable

Now, the mentioned .swiftinterface, generated by a Xcode 12.1 build, contains

public class Reachability {
  public typealias NetworkReachable = (Reachability) -> ()
  public typealias NetworkUnreachable = (Reachability) -> ()
  @available(*, unavailable, renamed: "Connection")
  public enum NetworkStatus : Swift.CustomStringConvertible {
    [...]
  }
}
[...]
@available(*, unavailable, renamed: "Connection")
extension Reachability.NetworkStatus : Swift.Equatable {} // <= This is the failing line 92
@available(*, unavailable, renamed: "Connection")
extension Reachability.NetworkStatus : Swift.Hashable {} // <= Also failing

So what I'm now not sure about is the question what exactly is meant with "references to unavailable protocol conformances are always diagnosed as errors" - in our case we are nowhere referring to Reachability.NetworkStatus anywhere in own code, but is the "reference" also somehow given as part of building a module from its module interface? What is the recommended handling of this?

And even more interesting question: Where do the protocol extensions to Swift.Equatable and Swift.Hashable do come from? I was assuming they are already somewhere in Reachability.swift's sources, but apparently they are not, so I assume they are auto-generated by Swift/Xcode in case of enabled module stability when creating the .swiftinterface files?!

As a workaround I was able to build module from its interface by manually removing the referenced protocol extension from the .swiftinterface files as we are not using this anywhere in our code, but that does not feel "right", and I feel that this might happen for all builds from a framework's module interface which include some unavailable attributes ...

Sorry for now getting a bit longer, but I hope you (or anyone else) may be able to support me on this.
Many thanks and best regards!

I believe the bug you are hitting is fixed in the release/5.5 branch on GitHub: Sema: Allow unavailable conformances to be referenced from unavailable contexts [5.5] by slavapestov · Pull Request #37255 · apple/swift · GitHub. The fix should also be in the recently released Xcode 13 beta. Can you give it a shot to confirm that this is indeed the same bug?