But this is necessary anyway, to support a distinction between target and host platforms, and to support building tools in release mode while building products in debug mode (which is the Right Thing). Cases where the same library build can be used for tools and products are occasional coincidences, not the norm.
In my opinion, it's wrong that targets containing a macro definition (potentially among the normal library) have to have the macro implementation as their dependency. The macro implementation is not actually a dependency of that target, because it isn't linked into it and could even be built for a different target OS in case the host and target machines are different.
So if a target has multiple dependencies, some of which are actually macro implementations, all of those dependencies are indistinguishably clumped together in the dependencies array of that target, even though they are meaning completely different things.
Maybe we could have a new attribute macroImplementations instead or just leave it out entirely, as macro definitions already contain the module of their implementations.
Nit: When I see just macro, I’m not sure if that’s for the implementation or the declarations. So why don’t we call this externalMacros instead? The module will be “called” by mentioning it in an #externalMacro expression, and the library can contain many macros so it should be plural.
Edit: How do you write a test target for MacroImpl? Macro testing is very important, and I’m not sure if we’d need special behavior to build tests for the host platform instead of the destination.
Are there also going to be resource limits or timeouts? My girlfriend asked about macros (this is what passes for pillow talk between programmers), and in the course of explaining them, I realized a malicious macro could enter an infinite loop and leave the compiler hanging as it waited for a macro expansion.
This hand-maintained list of macros is…a little gross. (So is the MyPlugin type itself, honestly, but it’s a little less offensive.) Could we add some kind of automated discovery here?
I have been going back and forth on this a bit but I think this actually the right thing to do from a build system perspective. While the macro definition is not importing any of the symbols from the implementation it is very much depending on the implementation to be present in users of the definition target.
From a build system perspective it makes total sense to model this as a dependency otherwise you would require all users to declare two explicit dependencies on on the definition target and one on the implementation target.
I am wondering if we can make this more straight forward by just calling them what they are executableTargets that are invoked by the compiler. Maybe this clears things up a bit more and also surfaces the fact that they are leaf nodes themselves more.
Don't think we should encode that implementation detail in the declaration. The prototype actually allows building macros as dylibs as well which is the way macros are shipped with the toolchain work and we may want to productize that option at some point.
Macros are being build by SwiftPM, so you're right that if you want optionality, you need to separate your library into multiple products. As proposed, swift-syntax is coming from the toolchain, but other dependencies would indeed be built even if the macro does not end up being used. Note that downloading of dependencies that are used by any products is never optional, no matter if they end up being built or used.
It wouldn't be possible without further changes to SwiftPM's build system which is rather simplistic today. So I think we should probably treat this as a future direction.
Well no, the status quo is that SwiftPM build tools don't currently work if your target and host platform are different. Xcode 14 improved things a bit, but still tries to build tools for both the host and the target and fails if your tool can't be built for iOS.
I have not tried particularly recently so it's possible I simply missed some development, but I'm under the impression that cross-compiling with swift build doesn't work with build plugins either.
Right, we need to assume that swift-syntax will continue to have breaking changes because it models the language fully, and the language's evolution should not be hampered by changes to SwiftSyntax.
In an ideal world, SwiftPM's build graph would be able to accommodate different versions of swift-syntax for different macro implementations, because each is a separate host executable. I hope we get that in the long term, but we won't have it soon.
The outcome of this is that all macro implementations will need to agree on a version of swift-syntax. This is the same old package resolution problem that we deal with everywhere, and why we ask packages to provide stability and follow semver. I think the problem rankles more here because (1) we expect to use a lot of macros, (2) we know swift-syntax cannot become stable, and (3) it's a limitation of SwiftPM, not of the overall design of macros, so we know that a better way is possible.
So how can we get a good user experience when one wants to use several macro packages in the interim? The idea behind using the toolchain version of swift-syntax is that (everyone uses the same version anyway, it's independent of any other swift-syntax version used by the program, and you can use #if compiler tricks to make macros work with the swift-syntax used with different versions of the toolchain. But the downsides are big... your macros could break with a new toolchain.
So, I think the least bad solution is for swift-syntax to adopt semver, but bump the major version every time there's a major Swift release (5.9, 5.10, etc.) that has breaking changes. Macro implementations should specify the version they need specifically. When a new Swift version comes out, macro authors would be expected to cover a range (say, 5.9 and 5.10), and we'll need some ways to conditionalize that macro implementation code (with #if something or other) to handle API differences.
I think it would be good if we had an "out" through binary dependencies. If you could have a binary package for a macro, you could compile it however you want because it's a separate build graph. As @anreitersimonnotes, there are other benefits to supporting this.
I don't think it would have changed much, other than perhaps timeline. The compiler is architected for a more general design than is expressible in SwiftPM right now, and the way the compiler works---macros in separate, sandboxed executables that can be independently built, interacting with the compiler via a versioned IPC mechanism, etc.---handles the general case. Our challenge is how to provide a reasonable user experience with the limitations we have in SwiftPM's build model and swift-syntax's necessary instability.
[EDIT: It's my fault that this is a surprise to anyone at this point. I misunderstood the nature of SwiftPM's unified dependency graph, and had assumed that this problem was already fully understood and dealt with by SwiftPM's existing build plugins.]
I agree that this is something we should do. It feels like an implementation decision that we wouldn't put into a proposal, though. We don't treat the type checker's internal limitations as something that should be part of a proposal, for example.
I suspect that would require something like custom metadata to do reliably, and is probably left until we have SE-0385 or something like it.
One of the things I've been wondering is whether a macro might want to also be a SwiftPM plugin - i.e. with the ability to run host tools and inspect the source/resources of the client in order to generate its own source/resources.
For instance, something like compile-time checked localised strings. The macro (as a SwiftPM plugin) may want to take a pass through all of your strings databases in order to generate a list of the keys which exist. Then the macro (as a compiler/language plugin) would use that list of keys to validate a custom literal such as #localized("settings.userprofile.dateofbirth") at compile time, and rewrite it as a call to a runtime function which queries the value associated with that key.
Wouldn't you be able to achieve the same result with macros that are not exposed as plugins? Since macros as pitched right now are exactly that: code generation tools that can also provide integrated diagnostics for things like validation of constants at compile time.
I think having to cut binary releases for all possible host platforms that might want to use a library would be an unfortunate burden on many users. In an ideal world a library author would have CI for macOS, Linux, Windows, and whatever else might come along that they could leverage, but even if they had that infrastructure, getting binaries from those pipelines and integrating them into their releases is non-trivial overhead.
I suppose environments like Swift Playgrounds on iPad, where binary artifacts aren't supported, would also suffer by not being able to apply that model.
Even if binary dependencies aren't a strict requirement but just an escape hatch for certain cases, I worry that those cases will be more common than we initially expect, and the macro experience and adoption of the future will really suffer because of these limitations in SPM. These workarounds feel like far more work than users would or should expect to be able to ship syntactic transformations in their code.
Posted an updated pitch here. This now proposes to uses a package dependency for swift-syntax and spells out the known SwiftPM limitations around package resolution.
)