This post contains the summarized meeting notes from the Swift and C++ interoperability workgroup sync-up meeting from the 25th of July.
Attendees: compnerd, Alex_L, adlere, richard, zoecarver, tongjiew, egor.zhdan, Robertorosmaninho, dabrahams, George, mikepinkerton
Discussion
@zoecarver: The main thing I want to talk about today is the C++ interop roadmap. We have this manifesto which Google wrote some years ago now. It was very comprehensive great piece of writing, and it’s unfortunately also not representative of where we are in the project now. It’s time that we have a new manifesto or a new roadmap that’s a little bit more relevant.
This will help us define the path forward for this project as well, and will help us with evolution. When we will start finalizing the design by sending it through evolution, the roadmap will give us a way to frame our evolution posts and how they fit together. This way we won’t have to have long list of steps of how a particular proposal fits in with existing proposals, etc. It will provide a framework for all these evolution posts.
Today I started by bringing the goals to the workgroup which I’m hoping we can discuss today.
I want to work on this roadmap collaboratively with the workgroup , so far we’ve been using forum posts which we’ve been discussing (in the workgroup-specific forum section). In about two weeks we should have a more cohesive roadmap that incorporates the goals into it.
Any questions about the roadmap before we look at the goals?
@richard: I would like to see the Swift package manager managed somewhere, perhaps not on the roadmap, but it should be indicated that it’s something that should be well integrated as a real world workflow that works with C++ interop.
@zoecarver: Yes, good point. It’s not necessarily a goal for interop specially. But it can go into its own section where we list things that are important for interop, like source kit. These things should be called out in their own section, as they’re not a goal of how we import C++ APIs into Swift, right.
@richard: Right. I was thinking of the whole picture for interop, not the forward interop goals specifically.
@zoecarver: We can have something more general in the goals, like how this fits into the overall Swift ecosystem. That would be a good thing to include in the roadmap.
After that @zoecarver shared the post that listed the goals with the workgroup. We discussed some initial feedback on forum post that @dabrahams had on the forums. @dabrahams mentioned that the goals are fairly non-specific and that he doesn’t disagree with the outlined principles, but the devil is usually in the details.
@zoecarver: yes the goals are very high level. The rest of the roadmap will hopefully get into a little bit more detail. If anyone else has disagreements with the goals here please make it known. Because once this document becomes part of the roadmap it would be ideal if this roadmap becomes something we can refer back to and justify decisions with.
@zoecarver: For example , it is not a goal to have every C++ API import into Swift. If that’s a problem for one of the adopters we should try to resolve that concern now rather than down the line once we start to work out the details.
@dabrahams: Good point. I’d like to amend that slightly. I would like to say “import into Swift without any work on the part of the programmer”. I should be possible to expose whatever C++ API you want unless it’s maybe macros.
@compnerd: Completely agree. We can have it as a non-goal to import every API implicitly, but it should be possible to explicitly import that API tuning that import, with API notes, annotations, or something else there.
@zoecarver: Yes that’s an interesting point. The part that I would disagree with is that I think there’s a very big spectrum here. You mentioned macros, but what about things like type traits? And something that uses non type template parameters? There’s a lot of good arguments that we should really actually try to figure out how to import those type traits. I’m not sure we should. Figuring out that sort of boundary what’s permissible and what isn’t is going to be important.
@dabrahams: Type traits get used all over the place. Modern C++ programming is fully of concepts which are just glorified type traits. Those things should not be an impenetrable barrier (maybe a slight hurdle) to adoption. If you want to give people the experience that their C++ code can be used from Swift you need to accommodate that stuff.
@zoecarver: On the C++ side you can still use type traits and we’ll import those APIs. I was more saying if we care about a Swift programmer as a consumer of C++ using type traits in Swift, and how does that fit into the Swift model.
@George : In the document it would be great to know what are the goals for implicit import of the Swift APIs. Like what are the things that might prevent implicit import? What are the things that might require extra work to achieve implicit import. Not only for type traits, but for the vast majority of the C++ nonsense that people get up to. So that us consumers can have a mental framework for how things should be handled by interop.
@zoecarver: I think that’s a great idea. I worry because C++ is such an absolutely non-opinionated huge language that’s that’s going to be a hard thing to enumerate in the goals here. But I think it should be a goal to have a clear and well defined mapping so that a user would be able to read some kind of document and have a good idea about it.
@George: To clarify, not just an enumeration of things, but the principles of what we should think about when importing a particular API. So not like type traits will not be imported, but something like this if this feature doesn’t fit into the Swift model in this ways then it won’t work, and list some potential ways to remedy that.
@dabrahams: That sounds like a great idea. I think I hear underneath this discussion is that with the current design you’ve been going towards, class template specializations is not something that can be reflected into Swift.
@zoecarver: I think that’s not entirely correct. I would make a different statement. Specializations can be represented easily as they’re specialized . Well this is an overloaded term. Concretizations of templates can be imported (regardless of which specialization was specialized), because they’re concrete type.
@dabrahams: Sure.
@zoecarver: Specializations are a harder question. I got into it in the generics roadmap I posted 6 months ago. I think there’s a lot of development that needs to be done. I don’t think this an issue design wise. I think there are problems design wise, but we have a workable design for this. And they’re going to be constrained no matter what.
@dabrahams: Because the implementation is hard?
@zoecarver: Partially.
@dabrahams: what’s the other part.
@zoecarver: The other part is the design.
@dabrahams: I’m less certain about that. The design of Swift today necessitates a whole bunch of things that are incompatible with C++, you know no move only types. There’s been discussion in the Swift community about small democratization of certain things and as soon as you do that, then the world of C++ templates specializations opens up.
@zoecarver: It’s a question of how much we want to bring the C++ model into Swift. I guess we can make this problem simpler for ourselves if we say that specialization must exist at the call site. What I mean by that you can’t specialize through Swift generics. This makes the design a little bit simpler for ourselves. And I think it’s very doable both in terms of design and implementation. Specifically, The Swift function that calls a C++ template would need to be specialized (monomorphic) .
@zoecarver: Let’s step back towards the goals. We could have another work group meeting where we can discuss this generics issue further.
@dabrahams: There are two more things I wanted to add about the goals. The first thing is that there’s going to be interest in calling Swift APIs from C++ for existing massive C++ codebases. That’s what you’ve been calling reverse interop right?
@zoecarver: Yes. This roadmap specifically focusses on the forward interop side, using C++ from Swift. We’re going to touch on goals for the reverse interop in that roadmap for sure. But the goals for that are very different. The design decisions are going to very different for reverse interop as well. And so we realized that it doesn’t make ensue to put these into 1 roadmap together, because they’re largely different things. Alex has a huge design document for reverse interop you can look at that presents the current idea.
@Alex_L: Yes. We will make another roadmap for reverse interop, and we’ll have a similar set of meetings for reverse interop as well. We will go through the same process with the goals and roadmap the other way.
@dabrahams: Okay. The only other remark I had was relevant to reverse interop, so I don’t have anything more to say about the forward interop goals.
@zoecarver: Great. Should we talk about adoption or reverse interop in the remainder of the sync up.
@Alex_L: Maybe you can show the user manual that you’ve added, because I’m not sure if everyone here has seen that.
zoecarver showed the manual: swift/UserManual.md at main · apple/swift · GitHub
@zoecarver: This document is the current state of what’s in tree right now, not necessarily what we’re committing to. This is a sort of user manual for interop. I also spent the last two weeks adding 2 dozen diagnostics that will guide people into the right direction when something they use can’t be imported. So there are four different type categories here. Foreign reference types allow you to associate specific retain/release operations in C++ with ARC operations for reference types performed in Swift, which is something that’s an experimental feature right now. It’s all experimental of course, especially things like owned types. This is the default for something that’s getting imported. There are certain properties that that an owned type must have , like the copies and destroys must balance out.
@Alex_L: So if the type has a pointer it’s not an owned type by default , right?
@zoecarver: Right so the so that's sort of why I'm I outlining this first so, I'll get to that in a second. There’s also the iterator type categories, which we can infer using C++ iterator traits, and the begin and end methods. And then we can automatically import iterator pair as a Sequence into Swift which is very nice. We also have trivial types. And then there’s the thing that Alex mentioned, so unsafe types. So, if if there's a pointer inside of your type, um, we assume that it's an unsafe type unless you put it into 1 of these categories. And this is assumed in certain contexts, so if it's returned from a method of an owned type, we assume that it's an unsafe projection and we actually don't import that function. Because, and I think this is something that there's been a lot of confusion on. I talked about this a little bit in my towards a safe interop forum post. Returning projections from owned values in Swift is when those values are of C++ plus types that have ownership semantics. I think that that's actually substantially less safe than returning projections from Objective-C types or you know C pointers or anything like that. That's because there’s a different model for automatic lifetime management between Swift and C++. Swift does not extend those lifetimes to the end of the scope. It makes copies in places that aren’t clear. And this makes it substantially more dangerous than other APIs. That’s why I think methods that return unsafe types from owned types should not be imported by default.
@dabrahams: Could you clarify what shouldn’t they be imported by default.
@zoecarver: Imagine c_str on std::string right? That is a projection of an owned type, right? That’s what I’m talking about. Does that answer your question?
@dabrahams: So so what you mean functions that return pointers and references basically.
@zoecarver: From owned type, yes.
@dabrahams: What’s a non-owned type?
@zoecarver: For example a reference type, like immortal reference type.
@dabrahams: If you can prove that it’s immortal that’s great. But we also have retain/release FRTs as well. Almost nothing is immortal in C++. I mean it’s certainly not encoded in the type.
@zoecarver: We’re going to require ‘retain’ and ‘release’ functions on every reference type. And if you have an immortal type you must specify that it’s immortal, by not providing ‘retain’ and ‘release’ operations when they’re being specified. In that case I think it’s always safe to return projections out of that.
@dabrahams: The actual problem is that you can't tell which things are projections from the immortal type and which things are are pointers or references into other types that are not immortal.
@zoecarver: Right. Okay, what I mean is that this won’t be less safe than a free function for example that returns a projection from somewhere else.
@dabrahams: But do you think it's less safe to return a pointer from the method on a, on a non immortal type?
@zoecarver: It’s substantially less safe to do that on an owned type.
@dabrahams: Do you mean you’re more likely to get a reference pointer from a method in that case that isn’t pointing into the object you got it from, but rather somewhere else?
@zoecarver: What I mean is that there’s actually very specific pattern in Swift that makes it more unsafe than a reference type or a global function. That’s because when you’re calling a method on an owned type its ‘self’ is oftentimes copied, which immediately makes that a dangling pointer, and that is not the case for a reference type because you retain that right, and then release it afterwards.
@dabrahams: This is one of the things I worry about planning the long term for C++ interop around things that we know are deficiencies in Swift that need to get fixed. Right?
@zoecarver: Yeah, I think that's true. I guess what I would say is, I hope that once those are fixed, we can just remove this limitation and it starts working. But I agree with you. And I think it's unfortunate. I think there's a balance though, as we actually want to ship this feature, right?
@dabrahams: I guess I have to keep in mind that what you're, what you're describing here is just what's currently implemented and nothing about the future. This brings me to what we are doing in Val, um, where, where projections bind the lifetimes of things. And, and I think it would be completely reasonable to assume that that everything returned from a method is a projection unless otherwise annotated.
@zoecarver: Yeah.
@dabrahams: And then you could use that information to keep the vector alive as long as the projection was alive.
@zoecarver: I think that's a good idea. And that's something we already do today for iterator types. It’s just harder to do more generally just because of where the compiler is today. But that doesn’t mean we shouldn’t do it. I agree that it’s something we should try to do.
@dabrahams: I'm always looking to where we're where we're aiming rather than where we are right now and as, as you said, and I have to keep reminding myself, this document that you are previewing is just about where we are.
@dabrahams: I think it would actually be super helpful, just at the top of that document to put, kind of disclaimer that says, you know, this documents, the, where we are today in terms of interop requirements. It doesn't necessarily reflect the future. You know, go look at these other documents if you want.
@zoecarver: Yeah. We can move on from this document.
@Alex_L: Could you share the next steps in terms of how many posts you want to make for the roadmap? And how big is it going to be, and what are the additional things you want to have there?
@zoecarver: Right I want to have sort of an introduction where I talk about C++. Then talk about the goals, and how we derive the approach to implementation from these goals. Talk about a little about how this fits more generally into Swift with reverse interop, and the ecosystem more generally.