(Initial) Plan for supporting C++ interoperability in Swift package manager in the Swift 5.9 release

Development C++ interoperability
,
1

NOTE: This initial proposal has been supplanted by an updated plan.

Last week in the @cxx-interop-workgroup sync-up we discussed the plan for SwiftPM support in the Swift 5.9 release. Since then I have worked on documenting and refining the plan, and filing issues for Swift and SwiftPM. I would like to present the plan to you in this post.

Overview

Swift package manager recently gained a new Swift build setting, that allows users to enable C++ interoperability for a Swift target (PR1, PR2).

This now allows us to enable C++ interoperability for a Swift target. By default, C++ interoperability is "viral", i.e. the setting propagates to dependencies of a target if such target imports C++ code or uses C++ types in its public APIs or function bodies that can be inlined.

To prevent the "virality" of this build setting, we are going to recommend users to use @_implementationOnly import when importing C++ code into Swift, and to avoid exposing C++ types in the public APIs of a Swift target that uses C++ interoperability.

This will then allow users to enable C++ interoperability in a single target, which means that they can distribute a package that exports such target without users having to opt-in into C++ interoperability. Here's a SwiftPM target diagram illustrating this scenario:

Screenshot 2023-05-16 at 12.28.48 PM
Screenshot 2023-05-16 at 12.28.48 PM2138×1306 218 KB

Notice that only the UsesCxxLib target needs to enable C++ interoperability, but not the other targets.

Exposing Swift APIs to C++ is something that we won't support in Swift 5.9 in SwiftPM, as SwiftPM right now doesn't add search paths for the generated header to C++ targets that depend on a Swift target. We think that's okay, and we can have a better build story in a future release of Swift, that will allow us to expose Swift APIs to C++ from multiple SwiftPM targets.

Proposed guidance

Here's the proposed guidance we would like to document on the Swift website:

When a package intends to vend Swift APIs from a Swift target that uses C++ interoperability, the recommended approach is to:

  • Enable C++ interoperability just for that target.
  • Import C++ targets using @_implementationOnly import.
  • Do not use any C++ types in the public interface of the Swift target.
  • Build the C++ dependencies within one package.

These guidelines ensure that the clients that consume this package and this target will not be required to enable C++ interoperability in their Swift targets, and they can still import the Swift module directly.

It’s okay to enable C++ interoperability for multiple targets within the same package, provided the targets exported by the package still follow the recommended practices above. You don’t need to bump the package’s major version if you follow the recommended approach.

Creating a package that does vend C++ APIs (it either exposes a C++ target, or a Swift target that uses C++ types in its public API and/or doesn’t use @_implementationOnly to import C++ targets) will force clients to enable C++ interoperability in their Swift targets. Therefore this approach is not recommended by default. However, if you’d like to make such a package, please follow these guidelines:

  • If it’s an existing package, bump up the major version. Forcing clients to enable C++ interoperability is a breaking change!
  • Don’t distribute such package to a wide audience. Ideally such package would be only consumed directly by another package controlled by the same package authors that doesn’t force its clients to enable C++ interoperability.
  • Explicitly inform clients that they have to enable C++ interoperability when depending on targets from such package.

Note: Using Swift APIs in C++ is not yet something that’s supported in Swift package manager. Right now you can only import C++ APIs into Swift when using SwiftPM.

Getting there

Unfortunately there's still some bug fixes that need to be done in order to make the proposed plan work in Swift 5.9. It's tracked in our C++ interoperability 5.9 tracking issue, but here's the summary:

Compiler fixes required:

SwiftPM fixes required:

Summary

We plan to provide support for using C++ APIs in Swift in SwiftPM projects in Swift 5.9, in a way that does not require package vendors to force clients to enable C++ interoperability. The C++ interoperability workgroup will provide an example of a SwiftPM package that uses C++ interoperability on Github as well, and will document the guidance to SwiftPM users and package authors on swift.org .

Please let us know what you think of the proposed plan.

13 Likes
2

Sounds good. I really hope it’s get done by the time 5.9 appears. I can‘t wait to use my C++ libraries in iOS et al. Btw., this also applies to Linux or does it?

1 Like
3

A problem with this plan is that @_implementationOnly import is not a fully supported feature, and in particular has known problems when used in code that is built without -enable-library-evolution. It relies on the ABI to hide implementation details in cases where, for example, a public struct contains a stored property whose type comes from an implementation-only import; with library evolution, we can diagnose attempts to declare such a struct as @frozen, but without library evolution, all structs are compiled as if frozen, but since the type of the stored property is hidden from clients, the compiler will likely crash or miscompile when attempting to use the struct since it can't accurately reconstruct the layout.

However, currently, to my knowledge, swiftpm does not currently support building packages with library evolution. Is this something that swiftpm could support, maybe automatically for packages that enable C++ interop support? This does not necessarily mean that we're committing to a stable ABI on non-Apple platforms, since we can take advantage of library evolution for the implementation-hiding aspects without promising the ABI is stable.

2 Likes
4

Thanks for pointing this issue out, that's something that I've overlooked.

However, currently, to my knowledge, swiftpm does not currently support building packages with library evolution. Is this something that swiftpm could support, maybe automatically for packages that enable C++ interop support? This does not necessarily mean that we're committing to a stable ABI on non-Apple platforms, since we can take advantage of library evolution for the implementation-hiding aspects without promising the ABI is stable.

Great idea! One thing that we've noticed so far is that we have no way to differentiate between a SwiftPM target that enables C++ interoperability, and requires clients to enable it versus a target that doesn't require clients to enable it. Perhaps if we were to provide a way to specify that distinction on a target basis, we could build such target with library evolution enabled when the target doesn't require clients to enable it. Having such a distinction at the SwiftPM level would also give us the ability to provide better diagnostics to the users, in case they want to consume a Swift PM package that requires clients to enable C++ interoperability, and they haven't done so yet.

The current default flag could imply that clients do not require C++ interoperability to be enabled:

.swiftSettings: [.interoperabilityMode(Cxx)]

In this mode, SwiftPM would pass -enable-library-evolution when building such target.

However, if the package author wants clients to enable C++ interoperability, they can then specify that explicitly:

.swiftSettings: [.interoperabilityMode(Cxx, requiredByDependents: true)]

I will do some testing with existing Swift packages to see how viable it is to build them with library evolution enabled to see if this plan is still viable for 5.9.

2 Likes
5

Yes, if we proceed with the proposed plan that would cover all the platforms supported by Swift and Swift package manager.

1 Like
6

Requiring library evolution mode would be a problem for us. We have to postprocess the .swiftinterface file to work around [SR-14195] Swift emits an invalid module interface when a public type has the same name as a module · Issue #56573 · apple/swift · GitHub, which isn't something that can be done when building via SPM.

7

Maybe this ends up being a different compiler mode, but the main thing we'd need from library evolution mode is just the code generation behavior that hides internal type layout details from clients. It seems to me that we could still use binary swiftmodules to propagate the interface if there are problems with textual swiftinterface support, since the binary module is still being built only to be consumed as part of the current swiftpm build and not to be distributed independently.

2 Likes
8

Just to clarify my previous post, you would be able to opt out of requiring library evolution when you force clients that use your package to enable C++ interoperability as well:

.swiftSettings: [.interoperabilityMode(Cxx, requiredByDependents: true)]

In this case, we wouldn't pass -enable-library-evolution when building such Swift target.

Is that sufficient for your use case, or do you intend to create a package you would like to distribute widely and don't want your clients to have to enable C++ interoperability?

9

I'm not entirely clear on what the negative effects of forcing downstream consumers to enable C++ interop are, but I assume there are some as this would all be pretty pointless otherwise. If switching to calling the c++ layer directly from Swift (rather than going through an obj-c intermediate as we do now) isn't a totally transparent change I don't really see us doing it.

1 Like
10

I'm definitely a bit nervous about smuggling library evolution mode into SwiftPM in this way. I understand the desire to couple it with this feature, but the two features are at least partially disjoint: I might want library evolution mode without C++ dependencies to gain access to the language dialect that allows open enums, for example.

Is it worth considering polishing up and landing @_implementationOnly as a supported feature instead, under the aegis of the C++ interop work? It is extremely broadly used across the SwiftPM ecosystem to hide C dependencies, regardless of whether it is expected to work.

10 Likes
11

I previously filed a case that covers SwiftPM support for enabling library evolution with the same-toolchain restriction as an acceptable prerequisite that would still make tremendous progress on other platforms, it definitely would be useful not only for C++ interop.

12

I think so. It seems somewhat contradictory to say that we should be considerate of dependent libraries... and therefore recommend using an unstable language feature so they now transitively depend on that.

If there is a need for this feature, we should define it and make it official instead of continuing this game where the ecosystem grows ever-deeper dependencies on features which we refuse to accept as a part of the language.

3 Likes
13

If timing isn't an issue, having proper language support for non-transitive imports would definitely be the ideal solution. However, in order to achieve the OP's stated desire to have C++ interop be non-transitive, it seems to me that such a feature would need to be able to use the library evolution implementation to some degree, even though it would otherwise have sufficed to have such a feature treat the imported declarations as semantically internal or private to the importing module while compiling normally without library evolution in all other respects. Because a struct in a module with C++ interop enabled would be able to embed a C++ type inside itself, clients would still need to have C++ interop enabled to some degree in order to understand that type's layout, even if the C++ struct is semantically internal, unless we compile the type as if library-evolution-enabled in order to completely hide its internal layout from clients.

2 Likes
14

Would it be unreasonable to require anyone using a C++ type in a private stored property to box it in an Any and (if they want for convenience) wrap it in a computed property that casts it back out?

This admittedly isn't a desirable state (and it's not the most performant), but I think it would get around the ABI issues without unleashing the entirety of -enable-library-evolution for an unrelated feature.

1 Like
15

It's possible that the Swift compiler could error on the use of C++ types in a fragile Swift type that come from implementation only imports that forces propagation of the requirement to enable C++ interop to the clients. I will investigate it how feasible that is. In that case we could avoid enabling library evolution and just have strict boundaries that govern how C++ APIs are used, and provide guidance to use patterns like you're describing (like storing the C++ value in an existential).

16

I believe that there's going to be some work happening on making @_implementationOnly a supported feature in the future. However, we would like to try to have a more usable story for C++ interop in Swift 5.9 specifically. And since C++ interop is still an actively evolving feature that didn't undergo proper evolution for its feature set yet, I think it could be okay to suggest the use of a similar not yet fully approved @_implementationOnly annotation, as long as we're still moving towards the direction where both @_implementationOnly and C++ interop itself go through evolution, and as long as we provide tools and/or diagnostics that suggest to users how to migrate their code to the officially supported language feature.

17

Not having to box C++ types (or do awful things to get an appropriate amount of scratch space in the object) is the main functional benefit of C++ interop, but I guess the reduced wrapper boilerplate would still be useful. I suspect the performance hit from downcasting from Any would often be a problem, though. It's unfortunately (quite reasonably) a pretty expensive operation and a manually written box can be much faster.

18

Good point. However, there's actually room for flexibility here - you can write a C++ box that just uses a single pointer, and use OpaquePointer stored property in your Swift type. That does not force the propagation of C++ interop to clients since OpaquePointer is a regular Swift type.

19

I've posted a new thread with the updated plan following the feedback on this thread: Updated plan for supporting C++ interoperability in Swift package manager in the Swift 5.9 release

2 Likes