1

Swift provides ergonomic interoperability with C (How Swift Imports C APIs), Objective-C (Importing Objective-C into Swift, Importing Swift into Objective-C), and Python ([1], [2], [3]).

Interoperability between Swift and C++ has been previously discussed on the forums ([1], [2], [3]). The Swift for Tensorflow project has already started to build it out under the hidden -enable-cxx-interop flag. I have been trying to understand what interop will look like across all C++ features. I have summarized possible design options in a manifesto: Interoperability between Swift and C++.

It is not a fully finalized design: many sections list multiple alternative options that we should discuss and evaluate their pros and cons. Some sections are missing, notably, calling Swift from C++. Nevertheless, this manifesto covers enough ground to show what Swift/C++ interop could look like, what will work well, and where the rough edges are.

I would like to invite everyone to provide feedback on the manifesto, discuss the design, evaluate alternative options explained in the doc, and brainstorm further alternatives that we didn’t think of. If there is lots of discussion on a particular topic, we will branch that discussion into a separate forum thread.

Interoperability with C++ is a broad topic, and the manifesto is quite long (more than 100KB!). I would like the discussion to focus on the most difficult and risky parts of the design, although you should feel free to comment on any topic. In particular, I think these are the most critical topics to discuss:

  • Goals, use cases, and constraints.
  • Mapping for C++ const reference parameters.
  • Mapping for C++ namespaces.
  • Mapping for move-only classes.
  • Mapping for non-movable classes.
  • Mapping for function and class templates.
  • Mapping for C++ enums, which are most often not annotated with NS_ENUM.

We will continue implementing Swift/C++ interoperability under the experimental flag -enable-cxx-interop. We will invoke the Swift evolution process if we need changes to the Swift language itself (as a hypothetical example, if we seriously considered adding namespaces as a feature to Swift).

I look forward to your input!

73 Likes
2

Hi @gribozavr ! This is a great and very in-depth manifesto, I really enjoyed reading it.

I have a two suggestions to offer though, regarding approach to templates and to standard types.

1. It is possible imo to bring SFINAE into Swift for C++ templates, without any added syntax to both Swift and C++.

We can do this by reverse-importing Swift protocols into C++ as proxies for type substitution.

Consider this example:

// Swift code:
protocol CustomSwiftProtocol {
    func isLess(than other: Self) -> Bool
}

func maximumValue<T: CustomSwiftProtocol >(_ v1: T, _ v2: T) -> T {
    return std_max(v1, v2)
}

// C++ code:
template<T>
T max(T v1, T v2) {
    if v1 < v2 {
        return v2;
    } else {
        return v1;
    }
}

// Auto-generated proxy header
template<T>
struct Swift::Bridge {
    Swift::TypeExistential<T> __type;
}

// User-provided bridging header
template<>
struct Swift::Bridge<Swift::CustomSwiftProtocol> {
    friend operator<(Swift::CustomSwiftProtocol, Swift::CustomSwiftProtocol);
}

bool operator<(Swift::CustomSwiftProtocol t1, Swift::CustomSwiftProtocol t2) {
    return t1.isLessThan(__type, t2);
}

When compiling maximumValue, Swift compiler will use all constraints it has on generic type and put those into generated Swift::_MangledConstraintName_Proxy class. It will then pass Swift::Bridge< Swift::_MangledConstraintName_Proxy as template argument to the C++ template. Swift::_MangledConstraintName_Proxy will have exported definitions for all protocol methods as proxies to witness table entries.

Swift compiler can then provide implementations for some standard protocols like Equatable, and users can add their own when needed. If no such implementation is available, substitution will fail at C++ site with a fairly understandable error.

This should be possible if we when generating Swift definition out of C++ headers just mark those with the source header file location and let the compiler do a special pass when those are instantiated, generating temp header and a clang pass on the result to generate specialized definitions.

2. std:: types can be wrappers for Swift types instead of the reverse how it is proposed in the manifesto.

Consider the fact that most (imo) C++ code the users want to use as libraries is seldom vendored, since C++ compilers don't guarantee proper ABI when crossing boundaries between libs and especially between compilers.

So in fact the situation where we compile C++ sources as part of the same project may be more prevalent than when we have a vendored lib. This in turn means that we have the same toolchain for both our Swift codebase and C++ codebase. And this in turn means that we may build a substitute libc++ specially for Swift bridging. Such library would provide standards-conformant API, but behind the scenes use Swift types and/or specialized abstractions to ease Swift interop.

In such case, std::string can be implemented to operate on the same backing storage as a Swift String , as well as slightly deviate and require the string to be a valid UTF-8 string. If the library designer or user knows that the string may not be UTF-8 conformant, the user may just add a -DUSE_NONUTF8_STRING to compilation flags and then a) the Swift importer will map std::string to Array<Int8>, and b) std::string will be instead using Array backing storage.

What do you think about those approaches? :slight_smile:

1 Like
3

Thank you for the suggestions!

I don't think you are describing a way to deal with SFINAE (std::enable_if constraints). I think you are suggesting two things:

  • a way to call C++ templates from Swift generics.

  • a way to export Swift types to C++.

As far as calling C++ function templates from Swift generics, the primary difficulty with your suggestion is that specializations of C++ function templates will not be picked up. Consider what happens when we call maximumValue with T=SomeCppClass. ::max could have a user-defined specialization ::max<SomeCppClass>, but instead interop will call ::max< Swift::Bridge<Swift::CustomSwiftProtocol>>.

Another difficulty is that mapping Comparable to Swift::CustomSwiftProtocol as shown above is not fully safe. C++ does not have an equivalent of Swift's Self in protocol definition. Therefore, C++ code can pass mismatched arguments to operator< and it would still compile. It would be possible to detect the error at runtime, but it is important to keep in mind that the mapping is not as straightforward.

I think this is a very interesting direction, but I think first we should discuss how to export simpler Swift types to C++, like plain structs and classes, and how to map their protocol conformances. Once we have these building blocks, we will explore how they compose with templates.

I think that would imply that std::string would have to be backed by a copy-on-write buffer, which is prohibited by the C++ standard since C++11.

Also I think that a significant number of C++ users would be unhappy about atomic reference counting and copy-on-write semantics in value types like std::string and std::vector. COW and ARC maybe could have been very reasonable ways to implement data structures in C++, but since C++ implementations have not been doing that (or even actively stopped doing it since C++11), I am certain that changing eagerly-copying data types into COW and ARC based data types will significantly degrade performance in lots of C++ code.

I'm not sure what makes you say that: many compilers and standard libraries do guarantee ABI stability. libc++ on MacOS, iOS and Linux is ABI-stable, and so is libstdc++. GCC and Clang both implement the Itanium ABI and are link-compatible. Clang also implements the MSVC ABI and is link-compatible with code produced by the Visual Studio compiler. These are not just theoretical arguments, many real systems are built on top of them (e.g., most Linux distributions, as well as MacOS and iOS). You can also look at the amount of pain Linux distributions experienced when libstdc++ had to break the ABI for std::string due to C++11 changes.

I wish we could handwave the ABI issues and tell everyone to vendor their C++ standard library. While some people can do that, a significant number of users lives in a very different world that depends on a stable ABI in the C++ standard library.

The question actually goes beyond ABI stability -- it goes into being able to rebuild from source all C++ code that the user wishes to load into the process. There are significant difficulties with loading two C++ standard libraries into one process. People will run into these difficulties if they want to depend on some system library (or a library that is distributed as a compiled binary) that loads the C++ standard library from the system, and on Swift/C++ interop that will depend on the Swift-specific C++ standard library.

I think that making Swift and C++ standard libraries cooperate is a very interesting and possibly very rewarding direction, but we must ensure that stuff that works today continues to work the same way when Swift/C++ interop is enabled. Sure, if enough users of Swift/C++ interop can vendor a C++ standard library, and breaking the ABI of the C++ standard library helps significantly enough to warrant maintenance costs (maintaining a custom C++ standard library is a huge effort), it would be an important optimization, but it won't work for every user of Swift. Therefore, we have to implement the general solution first.

5 Likes
4

Hi all, with the new year starting we'd like to share an update to the current status of the C++ Interop (also being discussed in a separate topic - How developed is C++ interoperability?). With the great help from @zoecarver and a few others, with many interesting bugs fixed, the following is implemented in general:

  • C++ structs, classes, and enums are imported (including nested types)
    • Non-virtual and static methods are imported
    • Both member and non-member operators are imported
    • Constructors are imported (copy/move constructors are not yet used by Swift)
    • Destructors are (or will be shortly) imported and used by Swift
    • Imported members respect the source visibility
  • Namespaces are imported
  • extern variables are imported
  • Static variables are imported
  • C++ modules respect @_implementationOnly visibility
  • Glibc module map is compatible with the libc++ module map
  • References are imported as UnsafePointers or UnsafeMutablePointers
  • Class/Function templates are imported
  • Fully specialized class templates behind a typedef are imported
  • It is (or will be shortly) possible to instantiate C++ class templates from Swift (nested types cannot be used across module interface)

What is currently not supported, but very likely doable:

  • Method calls work due to luck (even though we are not intentionally using the right ABI, it does not end up mattering on Itanium ABI)
  • Move constructors are not imported.
  • Copy constructors are not imported (but @zoecarver has a PR for that :)
  • Dependent types.
  • Importing the C++ standard library.
    • specifically std::string is not yet imported due to bugs; we are currently not aware of any new feature work required.
  • Importing Swift code into C++.

And of course there are more challenging but interesting and rewarding topics such as instantiating C++ templates with Swift types, or instantiating Swift generics with C++ types, which we have not pursued.

In summary, from our experience we can confidently say that the C++ Interop is a good and viable way of integrating C++ and Swift programs in an ergonomic way. The implementation is already quite powerful and has a great further potential. We haven't found any unsolvable problems.

Having said that, after what we consider the successful proof of the viability of the current approach, priorities have shifted, and we will scale down our investment in C++ interop for now. We would be happy to pass our knowledge to whoever would be interested in continuing this work, and there is already a growing community of contributors (some of which put us to shame - right Zoe? :) And we're confident that the project is in good hands.

Thank you all!

35 Likes
5

     Rather belatedly, here are some things I thought might — well? — need to get taken into account, kept in mind, and/or tracked, at least at some point, sorted roughly by the section(s) of the manifesto they pertain to:

  • 'Differences in move semantics between C++ and Swift' and related sections:
    • Recent attempts to propose 'move relocation' for C++:
      • P1029, move = bitcopies (previously move = relocates, née SG14 [[move_relocates]]) by Niall Douglas (GitHub tracking issue at cplusplus/papers#359)
      • P1144, Object relocation in terms of move plus destroy by Arthur O'Dwyer (GitHub tracking issue at cplusplus/papers#43)
  • 'Templates:'
    • Could the problems here get mitigated (on the C++ side) by, for each unspecialized template, using a type-erasing polymorphic wrapper to implement a handle bundling all of the concrete instantiations of that same unspecialized template's defaulted or custom, explicit or implicit specializations together?
  • 'Exceptions:'
    • Standards proposals for introducing 'static exceptions' into C++, as well as one other, related paper:
      • P0709, Zero-overhead deterministic exceptions: Throwing values by Herb Sutter (GitHub tracking issue at cplusplus/papers#310)
      • P1028, SG14 status_code and standard error object (previously SG14 status_code and standard error object for P0709 Zero-overhead deterministic exceptions,) by Niall Douglas (GitHub tracking issue at cplusplus/papers#405)
      • P1095R0 (also ISO C WG14 N2289,) P1095R0/N2289: Zero overhead deterministic failure — A unified mechanism for C and C++ by Niall Douglas (GitHub tracking issue at cplusplus/papers#319)
      • P2170, Feedback on implementing the proposed std::error type by Charles Salvia (GitHub tracking issue at cplusplus/papers#879)
      • /Maaay/-be P2232, Zero-Overhead Deterministic Exceptions: Catching Values by Emil Dotchevski (GitHub tracking issue at cplusplus/papers#965)

Also, when it comes to text handling, there might be some additional work WG21's SG16 for text and Unicode could do on the C++ side to make things easier for interoperating with Swift and other languages which you could coordinate with them.