Foreign reference types

Objective-C classes are imported as Swift classes with different reference counting. Objective-C blocks are imported, again, as Swift classes with different reference counting. The same is true for a variety of foreign types. I propose that some C++ types be imported as Swift classes with different reference counting: foreign reference types. And in this case, the “different reference counting” is actually “no reference counting.”

In C++ it’s very common to have types with reference semantics. Examples of such types in the Swift compiler are SILInstruction, SILBasicBlock, SILFunction, Expr, and the ASTContext to name a few. Currently, these types cannot be imported at all, because we don't have a way to materialize them or copy them around. Foreign reference types allow these C++ "reference types" to be imported with ergonomic Swift semantics, this makes them easy to use without unsafe pointers.

Foreign reference types (or custom reference types) actually have a broad set of applications beyond just C++ interop, but I won’t get into that in this post.

So what would these types actually look like? What are the rules? Foreign reference types will behave a lot like Swift classes. They will have reference semantics and be implemented as pointers under the hood. However, at least for the time being, Swift programs will not be allowed to manage their lifetime, so Swift programs will not be allowed to construct, copy, or destroy foreign reference types. This allows foreign reference types to be light-weight, trivial types. Additionally, reference types must be reference types, so we will not import any APIs that use these types as value types (i.e., a function which returns a foreign reference type by value).

If you’re like me, the easiest way to understand how this works is by looking at some code, so here you go:

// C++ API (assume non-null pointers):
struct __attribute__((swift_attr("import_as_ref"))) IntPair {
  int a = 1; int b = 2;

  static IntPair *create() { return new IntPair(); }
};

void mutateIt(IntPair &x) {
  x.a = 2;
  x.b = 4;
}
// Imported as:
class IntPair {
  var a: Int32
  var b: Int32

  class func create() -> IntPair
}

func mutateIt(_ x: IntPair)
// Swift program:
let x = IntPair.create()
mutateIt(x)
x.b = 42
print(x.a)

There are a couple of interesting things above. First, notice that we have to provide a create method to create an IntPair. We cannot construct it from Swift. Second, notice how the pointer and reference types can be used interchangeably, but IntPair is always either a pointer or reference in C++. Finally, notice how we’re able to access members of IntPair without dereferencing it, just like any other Swift class type. The above program assumes non-null pointers, but if this option were not enabled, create would return an Optional<IntPair>.

I’ve created a patch to implement these foreign reference types here. I hope to start adopting these in libSwift shortly. LibSwift will allow us to play with these types and explore how they work. After tuning the semantics, if they work well, I plan to create a proper proposal/pitch where the community can explore adopting foreign reference types as a public feature.

7 Likes

Who would be in charge of deleting the IntPair in this example (and in non-example usages)? Does this create the possibility of use after free?

The sorts of types we'd be targeting here are, in their current design in C/C++, unsafely-managed reference types. So yes, there's a possibility of use-after-free. When Swift gains the ability to express move-only types, some of these types could switch to being imported as safely-managed move-only reference types. However, others probably can never become statically managed types, because their APIs use an ownership model that's simply too dynamic to be expressed statically without major changes.

For example, the ownership model of an llvm::Instruction is that it is owned by the enclosing function. One common pattern for writing an LLVM pass is to scan the function, identify interesting instructions to replace, add the replacement instructions without touching the old ones yet, and collect all the old instructions in a list that will be deleted at the end of the pass. I'm not really sure how you would formalize that in even a very sophisticated ownership system.

1 Like

CC @dabrahams

Not to pick nits early, but a few points here:

  1. I wasn't sure exactly what you meant by “types with reference semantics,” so I went to look at examples (since @Alvae, @shabalin, @dan-zheng, @saeta, and I have been trying to formalize value semantics, how these terms are used matters to me).
  2. I think it's unlikely these types actually have reference semantics in the way we intend the term. Expr, for example, is non-copyable, and almost every non-copyable type has value semantics by our definition: given a variable of that type, you can only change its value via operations on that variable.
  3. Giving these types from inside the Swift compiler as examples of what you mean is a bit sub-optimal for the audience. For example, I took a look at SILInstruction and while it obviously has deleted copy-assignment and delete operators, many of its properties are probably a consequence of its inheritance from llvm::ilist_node<SILInstruction>… (which has four base classes of its own!), making it tough to get a read on what you're saying. It would be better if you'd give some minimal examples.

Currently, these types cannot be imported at all, because we don't have a way to materialize them or copy them around. Foreign reference types allow these C++ "reference types" to be imported with ergonomic Swift semantics, this makes them easy to use without unsafe pointers.

I'm gonna guess that what you really mean here is “non-copyable type.” But if that's not right I hope you will enlighten me.

Foreign reference types (or custom reference types) actually have a broad set of applications beyond just C++ interop, but I won’t get into that in this post.

I agree, but I doubt the specific things you're trying to handle with this initial, limited proposal should be imported as any kind of reference type. Of course, being really sure about that requires a more rigorous nailing-down of what these types have in common…

From my point-of-view, just dealing with the non-copyable subset of types that I assume you're trying to address:

  • We'll need non-copyable types eventually in Swift, so we'd better spell out their semantics in a way that isn't interop-specific.
  • C++ types with nontrivial copy constructors should be imported as non-copyable by default. All Swift types have O(1) copy/assignment operations; allowing things like std::set<std::vector<std::string>> to be implicitly copied from Swift would introduce some frightening performance cliffs.
  • Special annotation should be available to exempt things like shared_ptr that have a nontrivial but O(1) copy, so they're copyable in Swift.
  • Non-copyable types should be passable-by-value like any other type. Passing by value doesn't imply a copy or a transfer of ownership; you can think of non-copyable types as using a guaranteed calling convention, or being passed by pointer-to-immutable.
  • @escaping annotation should be available for all parameters. You can use it whenever a non-copyable parameter value might need to outlive the callee frame; typically the parameter would have to be movable for that to work.
  • Passing a non-copyable type as an @escaping parameter ends its lifetime in the calling function. If you want to keep using it, you need to use an explicit copy (if one is available).
  • Non-copyable types are passable-by-inout like any other type.
  • Imported nontrivial types that are copyable in C++ should be given an explicit copy API by the importer.

There's an implicit contract that goes with passing by & or const& in C++: unless otherwise specified, the callee is going to assume that it has a unique reference to the value. In other words, & corresponds to Swift's inout and const& corresponds Swift's pass-by-value in the common C++ programming model. The places where these assumptions do not hold are extremely rare, in part because they require awkward hoop-jumping in specification (you can see some examples in the C++ standard library). It would be a cryin' shame if we didn't take advantage of this correspondence for interop.

1 Like

It is expected that you can and will create multiple references to the objects described by these types, so no, it would not be acceptable for those references to be non-copyable. Treating these classes as defining reference types accurately captures the reality of how they are used.

Can you explain in more detail why e.g. T() (or operator new + T()) cannot be imported into Swift as init()?

I wasn't suggesting that references should be non-copyable. There is a distinction in C++ between T and T& that is sometimes significant, and to broadly interoperate with C++ code we can't always erase the distinction on import, so my baseline assumption is that a non-copyable T in C++ is non-copyable in Swift and that there's a way to form some kind of reference to it, most likely an Unsafe[Mutable]Pointer<T>.

Now, it may be the case that the types being addressed by this proposal are special in some way, and treating them as reference types and erasing the distinction of the underlying T is the best answer, but as I said in my previous post I'd like someone to spell out the qualities of code that makes such a treatment appropriate so I understand what the proposal is trying to address.

A few other points:

  • The motivation stated for the proposal is that “currently, these types cannot be imported at all.” There are two ways to address that; one is (roughly) the proposal, and the other is to solve the more general problem that preserves the distinction between T and T& on the Swift side. To me, the proposal seems like sugar that will not erase the need for a more general solution; it would obviously cause problems with the accessibility of C++ overload sets and template specializations if applied everywhere. If it is meant as an incremental move so we can get more experience with imported types and see what's left to address with a more general solution, that's OK, but someone should say so.
  • I'm a little concerned that all the information about mutation that is encoded by const on the C++ side will be dropped.
  • I'm also concerned about implicitly hiding unsafe references on import to Swift. It seems to me that at the very least the C++-side annotation that does this should include the word “unsafe.”

Dave, thank you for all those comments! There is a lot to unpack here, and I want to make sure I address it all thoughtfully, so it may take me a while to respond.

Also, I think I was in a bit of a rush when I made the original post, so it would probably be good for me to clarify a few things.

Hopefully in the next few days I'll have time to make an updated post and address some of these comments (along with other's comments).

In the meantime, I'd like to respond to two points so that I think may be causing a lot of confusion: First,

If it is meant as an incremental move so we can get more experience with imported types and see what's left to address with a more general solution, that's OK, but someone should say so.

My intention is that this "feature" is adopted in libSwift so that we can see what types have these semantics and understand if this is the best way to handle such types. As I said in the last paragraph of my original post, I'm not intending this as a publicly available feature, right now it's just something that will allow us to explore types with these semantics in Swift. There may be a better long-term solution/feature that will come out of adopting thing in libSwift and this/future forum post(s).

Second,

Now, it may be the case that the types being addressed by this proposal are special in some way, and treating them as reference types and erasing the distinction of the underlying T is the best answer

This proposal is specifically addressing types that cannot or should not be passed as a parameter as T or returned as T. That is a heuristic that is determined by the programmer/API writer by applying the attribute import_as_ref.

... More to come soon :)

1 Like

Not a problem; thanks for your work on this!

My intention is that this "feature" is adopted in libSwift so that we can see what types have these semantics and understand if this is the best way to handle such types.

OK, but I guess I'm looking for a crisper definition of what “these semantics” are, if you are referring to their C++ semantics… and if you mean their Swift semantics I think that may be a bit vacuous (see below).

As I said in the last paragraph of my original post, I'm not intending this as a publicly available feature, right now it's just something that will allow us to explore types with these semantics in Swift. There may be a better long-term solution/feature that will come out of adopting thing in libSwift and this/future forum post(s).

Oh, sorry I missed that. If you think such an exploration is useful, by all means I support it!

This proposal is specifically addressing types that cannot or should not be passed as a parameter as T or returned as T . That is a heuristic that is determined by the programmer/API writer by applying the attribute import_as_ref .

If what you're saying is that there may be a category of types that should be treated in Swift according to what's been proposed here and the only arbiter of that determination is a personal choice on the part of the programmer/API writer (so there is nothing technically or logically discernable that can help me decide what's in that category), then… sure, I guess it's an unfalsifiable claim and I can't really object.

I don't mean to give you a hard time here, but you /cc'd me and for better or worse this is how I function. To give useful input on a proposal I need to try to build a framework of understanding around it, so I tend to ask for things to be defined in terms that can be objectively evaluated.

Terms of Service

Privacy Policy

Cookie Policy