@backDeployed is operating as an override instead of only for use in unsupported platforms versions

We have been facing a lot of headaches recently from unexpected behavior of APIs that we have @backDeployed since Xcode 15 and iOS 17 released. Long story short, I've uncovered that any API that we @backDeployed is being called not only on OSes "before" the platform deployed version, but on OSes "after" the platform deployed version too.

This is trivially reproducible with unit tests or a playground:

import UIKit

public extension Locale.LanguageCode {
  @backDeployed(before: macOS 14.0, iOS 17.0, watchOS 10.0, tvOS 17.0)
  var debugDescription: String { // <-- introduced in iOS 17
    get {
      if #available(iOS 17, macOS 14.0, *) {
        return "17"
      }
      return self.identifier
    }
  }
}

func test() {
  let locale = Locale(identifier: "en_US")
  let languageCode = locale.language.languageCode!
  
  
  let desc = languageCode.debugDescription
  // ^ This print should yield "en" on iOS 16 & 17
  // but it yields "17" on iOS 17 (proving the @backDeployed API is actually being treated as an override instead thunking to the platform deployed version).
  // You can comment out the @backDeployed `debugDescription` above to see it go back to correct behavior
}

test()

Effectively, we overriding any API we like using @backDeployed instead of just back deploying.

Is this something that can be addressed in a bug fix revision to Swift (and ideally deployed to Xcode 15.1 before it exits beta)?

1 Like

I repro the same issue on Xcode 15.1 beta 1

I think you have misinterpreted what the @backDeployed attribute does and what it is meant to be used for. @backDeployed was designed for use in the implementation of frameworks that ship with an OS. The attribute is not designed give you a way to provide your own polyfills for arbitrary functions in the SDK, so I don't think it will have the effect you're looking for if I'm interpreted what you're trying to accomplish correctly.

In your example, you have declared your own computed property debugDescription for Locale.LanguageCode. Your declaration does not "override" the one in Foundation; it just exists simultaneously in your module as a different function that could be called. Whether or not your implementation is called at any given call site depends on which declaration the compiler statically resolves debugDescription to. If the compiler resolves the call to your declaration, instead of the one from the SDK, then the @backDeployed attribute on your implementation will cause the compiler to emit a call to an intermediary thunk that calls either your original function or a copy of your function, which won't accomplish anything useful since its a function in the same module and both the original and the fallback copy will always have an identical implementation.

7 Likes

Where can I find documentation on this? I have read and re-read to enhancement request and, though it lists framework owners as a use case, it never restricts it to that usage. It also uses javascript polyfills as an example comparison, and lastly the code example of how the compiler would produce a thunk is perfectly compatible in concept with extensions outside the deploying framework adding @backDeployed. In fact, if you google it, there are many examples of doing the exact same thing we are doing as consumers of a library we want extended deployment support for the same API.

Is it possible to implement a duplicate API for an existing API that is imported already? Im pretty sure the compiler will error on that. This is circumventing that.

I can appreciate that there may be some presumed nuance that is not explicitly documented. So Im wondering if the feature should be codified to support this real world use case, or if there ought to be documentation added to make it explicitly clear what is and what is not expected. Ideally with compiler warnings when using it in a way that is not supported.

I’m open to anything, but I have not encountered anyone who interpreted this feature to be restricted the way you’ve described, so we have broken expectations (intentional or not).

Thank you for engaging in this topic, I appreciate your response!

1 Like

The proposal document is at the moment the most detailed documentation for this feature that exists. The attribute probably ought to be covered in The Swift Programming Language, where the intended use case could be clarified officially.

In hindsight, the proposal ought to have been a lot clearer about this analogy because this confusion is probably inevitable. @backDeployed allows ABI stable libraries to extend the effective availability of their own public APIs to older OSes, where the API isn't actually present in the library. This resembles a polyfill mechanically, but it doesn't support the same patterns as folks may be used to from other languages. It does not help a module to overlay a polyfill on top of an API from another module.

The attribute does not affect how this aspect of the language works. In many contexts, Swift allows you to introduce declarations that shadow other declarations. For example, function parameters can be shadowed by local variables:

func foo(x: Int) {
  let x = 2
  print(x)
}
foo(x: 1) // "2"

Swift also allows members of extensions to shadow members of a type declared in another module. For example, in the program below I've declared my own count property on the Swift.String type that always returns 1 instead of the real length:

extension String {
  var count: Int { return 1 }
}

print("string".count) // 1
print(("string" as any Collection).count) // 6

I suppose you could use this technique to transparently redirect calls to APIs from the SDK in your module if you really wanted to implement a shim without changing any of the existing call sites, but it's important to understand exactly what's happening and what the limitations of the approach are. As demonstrated by the output from the second print(), the count declaration I wrote hasn't somehow dynamically replaced String.count in the entire program; it's just an alternative function that the compiler may statically resolve specific calls to. The witness of the count requirement of Collection still calls the count implementation in the standard library, since that's the one the compiler resolved the witness to when compiling String's conformance to Collection in the standard library, and so the second .count invokes the standard library's implementation.

If you follow how shadowing operates in the example above, then you can take it a step further to understand what adding the @backDeployed attribute to the count declaration in the example does: it introduces a thunk every time that specific instance of count would be called. For callers in the same module this accomplishes basically nothing at all (and the call to the thunk would actually be optimized away). For callers in another module, it would do something, but still not something very useful since the original implementation of the function doesn't ship with the OS and therefore the thunk is still just dispatching between two copies of a function that doesn't live in the OS.

It would be interesting to discuss how a feature that allows modules to provide polyfills for declarations in other modules would work. But that would be a separate language feature to pitch and discuss, and it isn't the problem we set out to solve by introducing @backDeployed into the language. I encourage anyone who wants to see such a feature to start a discussion thread in the Evolution section of the forum.

If there are specific examples of code that ought to be considered invalid that the compiler can diagnose to help alleviate some of the confusion, I'm happy to implement them. Unfortunately, though, the original code example in this thread is a perfectly valid application of the attribute; you could copy-paste that code into the implementation of some framework in Apple's SDK and it would be a 100% valid API, if a bit weird. I think the crux of the problem is that Swift happens to allow you to use extensions to shadow the members of types, and therefore if you have this misunderstanding about the purpose of @backDeployed then nothing will stop you from writing this code and thinking it does something that it doesn't. I think the obvious next step is to ensure there's really clear official documentation for the feature, though, and I'll try to get the ball rolling on that.

4 Likes

Yes, I do understand that this is going to be based on module import shadowing the original implementation. Thank you for describing it in detail with a code example for the posterity of this post!

Sadly, with @backDeployed, there isn't a mechanism to call the shadowed API. I think that would help this feature notably if the implementer could redirect to the shadowed API when it notes the OS version being high enough. In particular, there is a pain point when @backDeployed is used with an initializer, because the semantic requirements of initializers, there's no way to "dig yourself out" (e.g. just have a different object created and returned without calling any self.init designated initializer).

Yes, agreed. Now that I understand that the feature is not as one would interpret it from the @backDeployed proposal document, I'm mostly in a learning phase of the edges of this feature -- fully requesting a new feature would be something to pursue in the future.

I couldn't agree more! Thank you!

Concluding

Thank you for the discussion and education here. I will work on undoing our @backDeployed from our codebase, which is already seeing many hundreds of usages in the code base... the tricky thing about a great perceived solution, it gets a adopted rapidly and broadly.

  1. Better documentation. It is clear from Googling usages of @backDeployed and from conversations with very seasoned Swift devs, including language contributors, that this feature is expected to be a polyfill.
  2. Better warnings from the compiler. As valid as what we're doing is in to the compiler if we were implementing it in the deploying framework, it seems pretty clear that if you are implementing a @backDeployed API when you should not be (i.e. me), the compiler should warn you that you are not using @backDeployed with its express intent and therefor beware of dragons.

Thanks again :slight_smile:

2 Likes

I filed an issue on TSPL to track documentation for @backDeployed.

1 Like

FWIW, this is exactly the reason I argued against making @backDeployed a public-facing feature. It's a good feature, but only Apple's SDK engineers have meaningful reason to use it.

We should definitely clarify its intended use in documentation. In fact, I would go even further and disallow it from regular swift code (so misunderstandings like the one in this thread result in a compiler error), unless one passes in a special compiler flag to enable it.

If users are attaching @backDeploy to their code, they're just making a mistake. Why would we let them do that?

4 Likes

I completely agree, Karl. We are in a serious technical debt bind now because of how this feature was exposed to our devs in our iOS code base (thousands of devs) and things went forward without even a warning. I am super aligned on it's usage, but it is an expensive foot-gun we have to undo -- at least it has only been since June vs any longer.

2 Likes

In reviewing the proposal, we considered this option as an alternative. However, it would have been strictly worse: the standard library and other first-party libraries will make extensive use of the feature, so users will see it whether or not it is documented; there are a number of underscored attributes meant for internal use only which users have widely adopted in this way.

A key purpose of making @backDeployed a reviewed and publicly documented feature was precisely so that the semantics are set down clearly (and so that backdeployment information about new APIs is surfaced in user-facing documentation alongside availability information). Both during review, and afterwards in the explanatory articles I've seen around the internet (for example: 1, 2), I've seen pretty accurate descriptions of who or what it's for.

However, if the proposal text is falling short in that regard, we may need to work on editorial revisions, and certainly it is expected that TSPL will have accurate information on the attribute.

1 Like

I'm not sure those example articles you linked to are really crystal clear about the intended usage.

The first article, for instance, begins as following:

The @backDeployed attribute in Swift allows you to extend function availability back to older OS versions. It’s beneficial for framework developers to make new declarations available to apps with a lower minimum deployment target compared to the framework.

I can certainly imagine developers reading that and coming away with the wrong impression. It continues:

The @backDeployed attribute allows you to back-deploy functions and make them available to apps running on older OS versions.

"allows you to back-deploy" -- No. That's wrong. It's not for you.

Then at the very end, there's a section:

Who should use Function Back Deployment?

Function back deployment is primarily helpful for SDK developers. During normal app development, we bundle our latest library version directly with our app removing the need for @backDeployed attribute usage. In other words, all new code we write will be available immediately and doesn’t require back deployment.

That's also not absolutely 100% clear. Lots of developers write libraries which they call "SDKs". There's a Facebook SDK, Google Maps SDK, Twitter SDK, etc.

There are those two sentences about bundling the latest library version, but I also wouldn't really say they are clear enough in spelling out that this feature is literally only for people who work at Apple. Especially in the context of the rest of the article, which frequently seems to suggest that this feature is something I (some Swift developer) might be interested in using.

The second article doesn't mention explicitly who the feature is for, either.

So I don't come away from those articles with the same positive impression as you do; I look at them and come away concerned. This is a very specialised feature, and these articles are explaining it from the perspective of somebody who might want to use it, rather than from the perspective of somebody who wonders what it means when they read it in an interface.


Anyway, even if it is going to be publicly documented, I don't think it should be publicly available for use. You should have to opt-in to it.

3 Likes

How far would we get just from having it warn if it's used outside of library evolution?

2 Likes

Don't XCFrameworks build with library evolution enabled? (Genuine question - I'm not sure).

If they do, then it's not precisely the same thing. Authors of XCFrameworks should also get diagnostics if they attempt to use @backDeploy. Still, it would be better than nothing, as it would give diagnostics for at least some Swift code.

1 Like

That’s certainly an option. But one problem with a requirement like that (and an accompanying diagnostic) is that some users will see the diagnostic and just think that the problem is that they haven’t enabled library evolution, rather than the problem being that they’re attempting to use a feature that isn’t appropriate for their use case. Now they have two problems.

5 Likes