starting this thread to collect community input and eventually end up with a recommendations about using macros in some of the more “foundational” packages.
This comes from a specific need: introducing @Traced to swift-distributed-tracing (PR, and even older PR from 2024).
For reasons of “don’t pull any dependencies” recently we started trying to split out parts of swift-java into a dependency free (including free of swift-syntax / macros) package. And the same question was posed there as now appears in tracing, and many other repositories:
Question:
Is it ok for foundational packages, like e.g. swift-distributed-tracing, which will be used by many packages in the ecosystem, to use macros?
Recent improvements:
Recently, the pre-built swift-syntax was introduced to solve some of the build-time problems. However, still, many folks have quite a strong reaction about shipping macros in core libraries. Target based dependency resolution has improved in recent years in SwiftPM, however it still resolves “not used” by a target dependencies, so e.g. doing a TracingMacros target still causes the resolver to consider it during resolution. Future work may improve this, but we’re looking at present day approaches.
Package traits also don’t yet prevent the resolution of “not used” dependencies, which may perhaps also be improved in the future, but AFAIK at present package traits are also insufficient to avoid unused dependencies from being resolved.
I’d like to collect input and thoughts about this subject, and eventually propose to make this a form of SSWG or ESG recommendation, so we can have a consistent approach in packages.
One of the most important things for foundational packages to do when adding a swift-syntax dependency is to be as permissive as possible with its version range. I know the PR's linked above are only drafts, but currently they target just SwiftSyntax 600..<601, which means if someone creates their own macro target using the Xcode template (which currently targets SwiftSyntax 602+), it will be incompatible with the foundational module.
And the more popular a package, the more important it is to support a wide range of swift-syntax versions. For the small API inconsistencies in versions of swift-syntax you can use #if canImport(SwiftSyntaxXYZ) to tweak your code depending on which version of swift-syntax was actually resolved.
I personally don't think shipping macros in core libraries is a problem ever since prebuilts shipped. And there's no better way to shake off the fear people have for macros than for Apple to ship some libraries with them.
I think macros in low-level/foundational libraries should be absolutely fine as long as there's a way to opt out of the functionality provided by macros - be it through a separate product (allowing the adopter to only use the non-macro one) or using package traits (and it's fine for the trait to be enabled by default, the adopter can turn off default traits if they want to).
Also, if Swift Syntax gets cloned as part of package resolution, that's still IMO acceptable, but the point of the above is to be able to avoid building and linking Swift Syntax if you explicitly opt out of macro features in a given package.
A policy like this would hopefully allow us to provide macro features as added value without preventing some users from fully opting out.
This is going to be the thing that’s going to hamper efforts to adopt macros in any library and something that could burn initial efforts to adopt. We need to provide clear guidance on what to do with this, and we (in this case the SSWG/ESG) need to push for a real solution to solve the second half of the macro story
Overall, I am very sympathetic to address this problem, even if that is just in fact providing some guidance from the SSWG or ESG. Since the introduction of macros, there have always been two problems:
The build time impact of building SwiftSyntax
The fact that SwiftSyntax releases a new major for every Swift release
It's great to see that the first issue has been solved by the excellent pre-built work. I can already see in the CIs of projects that use SwiftSyntax that the build time got reduced by around 4 minutes. That's an amazing achievement!
However, the build time was never really a blocker in my opinion. The second issue is far more limiting to an ecosystem that is so heavily relying on Semantic Versioning. The fact that new major versions of packages are very impactful on adopters is not only affecting swift-syntax but many other packages as well. We saw similar problems in the recent grpc-swift 2.0.0 release, which resulted in moving the new major release into a separate repository to allow adopters a gradual migration. I would love to see a general solution to this fundamental problem in SwiftPM and potentially swift-build, but this is a complicated feature that requires careful design.
So with both a solution to the build time problem and the understanding that a solution to the versioning problem is not trivial, I am personally in favor of providing guidance to package authors on how they can start providing macros as safely as possible. This guidance should include details on how to best define the version range of Swift Syntax, how this relates to a package’s supported Swift version, and how supporting different versions of Swift Syntax can be tested in CI.
Can you explain further? Since SPM doesn't really support unbounded versioning, unlike CocoaPods, I don't see why the existence of a new major version of a package would have any impact on consumers at all, or why a separate repo would help anything.
You are correct that by just creating a new major, no consumers are impacted directly. However, the impact is in the secondary effect. The gRPC package was used by many other packages since it is a foundational networking protocol. There are Swift Server applications out there with many dependencies where some of those transitive dependencies have a dependency on gRPC itself. Since currently a package resolution must resolve to exactly one version of a package, this meant that such large applications had no path for an incremental migration to gRPC 2.0 until all transitive dependencies adopted the new gRPC release. This is very limiting, and often it is outside the control of the application author if and when a transitive dependency migrates. The secondary repo meant that gRPC 1.0.0 and gRPC 2.0.0 are treated as separate packages by Swift PM and opens up such a migration path.
Package traits also don’t yet prevent the resolution of “not used” dependencies, which may perhaps also be improved in the future, but AFAIK at present package traits are also insufficient to avoid unused dependencies from being resolved.
I did some work in this area, and you’re correct in that package traits don’t entirely handle the case for omitting unused dependencies – the current criteria for traits to omit a package dependency is that the dependency has to be:
Guarded by an un-enabled trait in each target dependency reference, e.g.:
The dependency is not used in any other case (i.e. there is no target dependency declared that uses this package dependency without a trait condition).
To specifically address whether SwiftPM handles unused dependencies: there is an experimental flag that I did some work on –experimental-prune-unused-dependencies that should omit package dependencies that aren’t used, but since it’s experimental it’s not something that I would say is 100% reliable at the moment. I’m hoping to see some development here in the future, and if folks try it out I would love to know your thoughts!
Did you also rename the library so there wouldn't be linking errors? I thought SPM couldn't support multiple versions since they'd conflict at build time. It sounds like you practically created a whole new library.
What might also be helpful for the community is some clear guidance — which TBF could already be out there and please point that out if it exists — about the goals of achieving "devx" parity between Xcode and VS Code when building packages that contain macros. I believe that VS Code recently added the ability to preview and inspect the codegen from macros inline which is great! But moving forward the community might also like to know if the project lead has committed to investing engineering time in maintaining and improving the VS Code support or if the VS Code support is internally thought of as more of a "community driven" project that is not on the direct scope of engineers working inside the project lead.
And vice versa, where Xcode doesn't properly support module aliasing, traits, or registries. Updating Xcode's package DX to better match what you get in VSCode would go a long way towards ensuring expectations match across platforms.
I don't think there's a reasonable path to adoption of macros whilst we require a single swift-syntax version for the whole graph. (Unless swift-syntax stopped releasing a new major for every Swift version)
There are a number of solutions to this problem but before it's solved, I don't think foundational libraries can adopt macros. Adding a swift-syntax dependency current risks being or suddenly becoming incompatible with a huge amount of code.
