[Pitch] Package Manager Support for Custom Macros

Evolution Pitches
15

This is something I've been wondering about. The dependencies of macro targets don't necessarily have anything to do with other targets or dependencies, because they run on the host rather than the compilation target. They should really have an independent tree of dependencies, but clearly that would add a lot of complexity.

That's why I wish we'd started having discussions about build system support much earlier. IMO, the compiler implementation is actually secondary to build-system support, but it seems that's where the focus has been until now.

Anyway, since we do only have one tree of dependencies, that means every target may be built as part of a macro. I think we need an addition to BuildSettingCondition so macro builds can be customised.

For example, WebURL's parser supports reporting validation errors when it fails to parse something. This information is currently unused - I don't think this detailed failure information is really useful for the runtime parser, since that is for runtime strings. Even if I threw an Error which communicated that a URL failed to parse because it contains an unclosed IPv6 address (e.g. https://[::1), I don't expect anybody's code to actually be able to handle that. And actually throwing detailed errors has a significant cost for binary size and prohibits compiler optimisations.

But for the compile-time version of WebURL, things weigh differently - the "error handler" in this case is a human developer seeing a literal string rather than pre-written code parsing unknown data, so it is worth giving more precise diagnostics. Binary size is a secondary concern, and since the information is more realistically useful the hit to parsing performance becomes worth it.

So I want to conditionally compile some of this code based on whether the module is being built as part of a macro implementation or not. AFAICT there is no way to do this. If we had an addition to BuildSettingCondition, I could write something like:

.target(
  name: "WebURL",
  swiftSettings: [
    .define("ENABLE_ENHANCED_DIAGNOSTICS", .when(configuration: .macro))
  ]
),

If I understand this correctly, it is somewhat disappointing. It means that if I want to expose a macro as part of WebURL, I'll have to add a new "WebURLMacros" module to hold the declaration, and users will need to import that explicitly - which limits discoverability of the feature. Presumably they would also need to add the "WebURLMacros" target as a dependency of their own targets, making it even more difficult to start using.

What I want (the ideal situation) is to expose the macro declaration from the WebURL module itself, so no additional imports are necessary.

The macro implementation would be in another module, and depend on the compile-time version of the WebURL module (this would be a cycle, so the compile-time version of WebURL would need to exclude the macro declaration).

At the same time, all of the host-side stuff (the macro implementation and compile-time version of the parser) would not be built unless the client actually used the macro.

I'm not sure this is possible with just one dependency tree.

┌──────────────────────────────┐
│ WebURL (client)              │
│                              │
│- WebURL parser               │
│                              │    ┌────────────────────────────────────────────┐
│- #url(...) macro declaration ├───►│   #url(...) macro implementation (host)    │
└──────────────────────────────┘    └─────────────┬──────────────────────────────┘
                                                  │
                                                  ▼
                                    ┌──────────────────────────────┐
                                    │ WebURL (host)                │
                                    │                              │
                                    │- WebURL parser               │
                                    └──────────────────────────────┘

Why not add support for all target settings?

  • Resources: It is possible that your macro implementation might depend on a binary resource, such as a canned database, JSON/XML files, ML models, or other data.

  • Plugins: Similarly, macros may need to invoke host tools and generate some code or resources. I think this makes a lot of sense, given that macros themselves are a kind of host tool.

  • Swift settings: Has various uses, including setting defines, enabling experimental language features, etc.

  • C/C++ settings (future): We may add support for mixed-language targets, and the C/C++ side may need compiler flags.

I think we will also need to address the inherently lack of scalability at some point. If every macro, in every one of your dependencies, can pin its own version of swift-syntax, building a project could easily require building dozens of different versions of that library.

For example, if WebURL's macro support uses swift-syntax version X, and your app/library uses WebURL, you'll need to download and build swift-syntax X as part of building your project.

And if you use a library which uses WebURL, again - download and build swift-syntax X. And so on, down the chain, everybody who depends on the library in any way takes a build-time hit. It can be very far removed from the code at the end of the chain, picking up many versions of swift-syntax along the way.

And if the idea is that every macro pins to a different version, that hit multiplies for every macro you use - WebURL was written to use version X, but some other library uses version Y, and another uses version Z, and if they are anywhere in the dependency graph, you take a hit from each of them.

6 Likes