Well... I should probably start this by acknowledging that I put the cart before the horse on this one. I do have a patch open here to start this process. But I’m happy to adapt that patch as needed, and I’m hoping this post can serve as a description of what that patch is, how it’s implemented, and why it’s nessisary. It will also give some future direction for the clang importer that I’m hoping the community can respond to.
I should also mention this forum post which I created six months ago. It does a good job highlighting many of the problems I solve here. But at the time, I didn’t have a good solution. And unfortunately I didn’t have the time to do a major refactor.
What does it mean to requestify the Clang Importer?
The current clang importer is a visitor that walks the Clang AST. For the most part, and as far as C++ interop is concerned, this is an entirely eager processes. This means, if we have the following API
namespace N { struct B{}; struct C{}; }
struct A {
using T = N::C;
int somethingUnrelated();
N::B somethingElse();
};
A::T test();
And the user asks us to call test we actually import everything, including struct C . This is because when we visit a struct or namespace we import all of its members. When we import a function, we eagerly import its return type and parameters. Same with fields and type aliases.
All of the above problems can be solved by making five things lazy: function return types, function parameter types, type alias types, field types, and most importantly member loading.
In the above example, this means that compiler would import only the test function in the beginning. Then, when it needed the return type, it would import A, but none of its members. When it needed A’s members, it would import them all (for IRGen, maybe). But none of the referenced types would be imported. So even after IRGen we would not look at the namespace, or import any of it’s members (assuming somethingElse was never called in the program).
This can be implemented by hooking into existing requests in the Swift compiler, and creating some new requests (and a new ClangImporter request zone). The lazy type loading logic already exists in Swift, that is simply a matter of adding cases to the existing Swift requests. The more complicated and interesting part of this change has to do with lazy member loading.
Lazy member loading is done by creating three new requests. These requests compose nicely both with each other and with the existing member lookup requests in Swift.
The first of these requests is a ClangDirectLookupRequest . This request looks up a clang decl’s member in the Swift lookup table. Unfortunately the Swift lookup table may need some updates to be able to handle all C++ APIs, but this can happen in later changes. The cases where it currently falls down are when looking up unnamed members, or members of unnamed contexts and looking up members of contexts that exist in multiple ways (such as two records with the same name, namespaces that are redeclared, or forward declared records). Currently I’m working around the latter by adding an extra filter step, but we could potentially just update the Swift lookup table to work around this. Also, at some point, it would be nice if we could only materialize the Clang AST nodes that we actually use. I think this functionality exists in the Swift lookup table, but I’m not sure how much it’s already happening or what steps we can take to reduce this.
Next we have ClangRecordMemberLookup which looks up the member of a clang record (struct, class, union, etc.). This request simply issues a clang direct lookup request and then imports the results and passes them along, which as some of you may know is much simpler than the behemoth visitor it is “replacing”.
After implementing this requests, I noticed lots of tests started failing for various reasons. That is because we had a habit of validating (and potentially skipping) members in the clang record visitor, after importing them. To fix those tests, I simply moved some of these checks into their component visitors. This demonstrates one way that these visitors force us to seperate our concerns. We shouldn’t be validating a VarDecl in the record decl visitor, for example.
This new model also makes importing records much more flexible. For example, if we want to look up a member in a record’s base classes, it’s simply a matter of changing the “decl context filter” in our clang direct lookup request, and we will find members of both our class, and any base classes.
Finally we have CXXNamespaceMemberLookup . This, as the name implies, looks up a namespace’s member. However there are some interesting things here. Essentially it all boils down to the idea that namespaces should not be physical entities. That is, a namespace should only exist as a “named path” to get to a member. They shouldn’t contain members, but instead exist only as a name that can be used to get to a member. You shouldn’t be able to see a namespace or instantiate an instance of it. Given that, namespace members are imported in a similar way records, except their members are simply returned, and not added as members of the namespace decl context and there is no logic in the compiler to import all members of a namespace. These are both artificial limitations that should enforce good use of namespaces.
Justification
In a language like C++, we often have very complicated types. The (eager) model we currently have doesn’t work well with complicated types. Not only does it mean the compiler has to do a lot of work, but it is also prone to causing bugs. You run the risk of importing the same type twice because it’s referenced by itself or its child, and you run the risk of making decisions about types when they’re only half-constructed, among other things. In my previous forum post I had four example patches that fixed bugs caused by the eagerness of the clang importer. Over the past six months we’ve fixed at least four more that can be directly attributed to this eagerness as well.
Moving the monolithic SwiftDeclConverter into self-contained requests that compose well will help the people who have been working on this project for months along with those who are just starting. It will be easier to track down the source of a problem or bug and will make it easier to find where the logic for importing something is or should be added.
As you can see from the example at the beginning, this should also give us huge performance improvements. With an early version of the patch linked above, importing the standard library went from taking minutes to a few seconds (I don’t remember the exact numbers but I’ll test this again and post the improvements as a follow up comment).
If you’re not already sold, another reason for lazy type loading, especially, is correctness. C++ APIs are allowed to assume that types will be substituted on demand. For example, you’re allowed to define a function template that returns a recursive template type. With our current eager model, this creates an infinite loop (so, mainly for performance, we have a very shallow maximum template instantiation depth). We can now remove these artificial constraints and allow more C++ APIs to be imported.
Finally, this direction fits very well with the direction that the Swift compiler seems to be moving in. There are lots of ways to implement lazy evaluation, and it’s important that we follow what the rest of the compiler is doing.
Scope
As mentioned in other parts of this post, I have patches open to make record and namespace member loading lazy. I also have patches to make parameter type loading lazy and type alias type loading lazy. And I’ve already landed the patch to make function result type loading lazy.
In the future we could also make field type loading lazy, but I’m not sure how much benefit this will actually give us (if you’re interested in taking this on, it’s simply a matter of hooking into the interface type request and looking for the case where kind == var). Additionally, we could move more visitors out of the Swift decl converter and into their own requests. I probably will not have time to do this, though.