Macros are one of the most celebrated new features of Swift, and many of us are excited to adopt them in our projects. Many members of the core team are also excited to suggest macros as a solution to many problems. We’d love to hit the ground running and adopt macros in our projects, but the decision to adopt them raises some questions, particularly around the dependence on the swift-syntax package.
Introducing SwiftSyntax to a project immediately incurs an additional 20 second debug build cost to a project, which may not seem like much, but that’s an extra minute for every 3 cleans. Things get much slower when building for release with whole module optimization: over 4 minutes just for SwiftSyntax. This issue from January highlights the problem, but there isn’t a lot of discussion around how it might be addressed. With Swift 5.9’s release on the horizon, can we expect these build times to improve in the near future, or will they be the cost of adopting macros for now?
Beyond build times, SwiftSyntax is a complex project to depend on, and it’s unclear how to version a project that depends on it. I started this discussion a few days ago and was hoping for guidance there, but I’ll restate the problem here. Because SwiftSyntax is a moving target and versioned alongside Swift releases, how can a library adopt macros and be compatible with multiple Swift versions at the same time? Is there a straightforward solution out there that isn’t documented? Beyond library maintenance there’s the question of library adoption. If an app depends on several packages that all ship macros, what happens when these libraries target different Swift versions? While this is a more general problem that comes with dependencies, SwiftSyntax would seem to exacerbate the issue.
And finally, SwiftSyntax’s API seems to be in constant flux. I’ve used the library sporadically over the years and whenever I return to a project that uses it, it rarely still builds. While it’s nice to see a new DocC tutorial on the 5.9 development release, there isn’t a ton of up-to-date documentation that’s easy to navigate in general, the examples in the repo are using old APIs that don’t seem to build any more, and many online resources one finds via a web search are also outdated. Is there a plan to stabilize these APIs? Should folks writing macros be concerned about maintaining SwiftSyntax code as its API continues to evolve?
Those are the three main issues that it would be nice to have clarity/guidance on. I believe there’s been discussion in the past about including SwiftSyntax alongside Swift, so maybe that would address all of the above concerns, but is it actually planned? Should folks be cautioned against adopting macros at this early stage?
I guess this is about the SPM distributed version? There is a bundled with the XCode's toolchain prebuilt version: /Applications/Xcode15.0b5.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/host/
The SPM-distributed version is what Xcode 15's macro template uses, and there's no way to simply import SwiftSyntax from Swift right now otherwise. One of my questions above less directly asks if this is to change by release.
@dmt Unless I'm mistaken, you can't distribute an SPM package with custom import paths. I also don't think Apple is advising folks link to this bundled version, nor do they document this option.
If number 3 continues to be an issue, it could be an option to update SwiftMacroToolkit to wrap the SwiftSyntax API more comprehensively to hide away the constantly changing API in a simpler, more stable, and more macro-specific wrapper for macro devs. That of course doesn’t help with the first two options at all though.
Huh, I guess you're right. I've found this in the example repo, but it seems like this approach has been abandoned, according to this commit.
Also I've found out that the toolchain for linux doesn't contain prebuilt SwiftSyntax for some reason.
Sorry for misleading.
On #2) It would be great to see it standard practice to import the package corresponding to targeted Swift language version. If you are targeting Swift 5.9 then use the 5.9 branch of swift-syntax. There is nothing stopping that now, but SPM isn't great at detecting updates to a branch.
I think it would be great to include the recommended version for each Swift language mode as a shared library with Xcode to speed up compilation. Just don't include it as a dependency if you want to use the Swift built-in version. I think a shared library would only be available to macros or build plugins and not other uses of swift-syntax.
Seems like auto importing the version corresponding to the Swift version in the package would be a good first step. Then allow an override of that auto import with some sort of manual version so people can track development versions.
I encountered another example of the "constant flux" when I updated the swift-syntax dependency specified by Xcode's macro template (509.0.0-swift-DEVELOPMENT-SNAPSHOT-2023-07-10-a auto-updated to 2023-08-07-a). I got a bunch of warnings when upgrading a relatively small project:
init(file:source:) is deprecated: Use init(fileName:tree:) instead
init(leadingTrivia:_:openDelimiter:_:openQuote:_:segments:_:closeQuote:_:closeDelimiter:_:trailingTrivia:) is deprecated: renamed to 'StringLiteralExprSyntax(leadingTrivia:_:openingPounds:_:openingQuote:_:segments:_:closingQuote:_:closingPounds:_:trailingTrivia:)'
rawStringDelimiter(_:leadingTrivia:trailingTrivia:presence:) is deprecated: renamed to rawStringPoundDelimiter
rawStringDelimiter(_:leadingTrivia:trailingTrivia:presence:) is deprecated: renamed to rawStringPoundDelimiter
argumentList is deprecated: renamed to arguments
replacing(childAt:with:) is deprecated: Use .with(\.[index], newValue) instead
argumentList is deprecated: renamed to arguments
replacing(childAt:with:) is deprecated: Use .with(\.[index], newValue) instead
appending is deprecated: Create a new array of elements and construct a new collection type from those elements
While some of these are simple fixes, others are not.
Yes, that is cost of building a macro at the moment and unfortunately I don't have any suggestion of how to improve this situation at the moment. Installing swift-syntax into the toolchain will only work in cases where that installed version happens to match the one that SwiftPM resolved, which won't match if we need to create a patch release of swift-syntax or the Swift 5.10 compiler is used with swift-syntax 509.
swift-syntax has matured significantly over the last few months, the main driver being the drastic increase in our user base from macro authors. This has involved a significant number of rapid changes, with the goal of delivering a solid release alongside Swift 5.9. The latest changes you comment on are due to Significant changes in the latest swift-syntax 509 prerelease, which we decided to land now to reduce the churn of the Swift 5.10-aligned release of swift-syntax.
We are anticipating that swift-syntax's API will become a lot more stable after its Swift 5.9-aligned release, though there will always be some API breakages in swift-syntax as the swift language itself evolves. We will do our best to mitigate these changes through API deprecations, which we have already been doing between the latest snapshot releases.
Regarding (3b) Lack of Examples
The swift-syntax repository contains a couple of examples that are guaranteed to always be up-to-date by a CI job. We'll also incorporate @Douglas_Gregor's list of example macros into there as well (#2026). Similarly swift-format is a sizable client of swift-syntax that's always compatible with the latest swift-syntax release. Beyond these, are there any particular examples you think are missing?
Regarding (3b) Lack of Documentation
With the two WWDC presentations on macros, Expand on Swift macros and Write Swift macros I think we have two good ways of getting started with macro development. swiftpackageindex.com allows browsing the swift-syntax API and swift-ast-explorer.com allows interactive exploration of the syntax tree. I agree that it would always be good to have more documentation, but I think we aren't in a terribly bad spot at the moment. If there's anything you feel is missing, pull requests and/or issues are always welcome.
It sounds like the answer to "how do you support multiple versions of Swift at once?" is "you don't". Updating to 510 requires dropping support for 509, and failing to upgrade means you block updating other packages which have upgraded. This isn't a problem in the short run since of course only 5.9 will be the only version of Swift with macros for a while, but it really isn't a viable long-term answer.
@ahoppen Thanks for replying! I appreciate the time and context.
While it's a bummer that there's no official way to address the build times, especially when it comes to release builds, I do hope awareness leads to a solution in the future.
Thanks for taking the time to work on the article! Having guidance is helpful, but I think it also highlights just how precarious versioning macros may be:
Should a client depend on another macro, which hasn’t released a new version that depends on swift-syntax 510 yet, then SwiftPM will continue to select version 1.2 for the macro. In order for the client to update to version 1.3 of the macro, all macros need to release a version that is compatible with swift-syntax 510. The macro can continue to deliver updates for those clients by creating patch releases such as 1.2.1.
@tgoyne already responded with my main unanswered question, which this guidance unfortunately doesn't help with. It's impossible to version a library by minor release this way and support multiple versions of Swift at the same time. Major versioning by Swift release (508.x, 509.x, 510.x, etc.) is a more flexible solution, and certainly works for executables like swift-format, and macro libraries that simply ship a macro and do nothing else, but this doesn't seem to be a feasible option for more general libraries that happen to ship macro-based APIs. More general libraries have the goal of supporting multiple versions of Swift at the same time while also iterating on their own features and APIs, and these libraries need to be able to introduce major, minor, and patch versions outside of Swift releases and swift-syntax versions, so I think we still lack guidance on how to handle this common use case.
(b) The examples out there are indeed helpful when they're up to date! I think it might be a good idea to prune this list in the README, though. Many of these projects haven't been updated in years.
This is precisely why an example library would be nice. While it won't be possible to show a macro that supports multiple versions of Swift concurrently till 5.10 is available, we could have a demo that shows how a more general library can use SwiftSyntax and ship versions that are compatible with several versions of Swift.
