i’ve fully understood this and its implications for a while now, i think that the point that @tgoyne is getting at is that this is not a sustainable way to develop macros.
imagine if macro A ships a critical bug fix and bumps its swift-syntax dependency requirement at the same time. users of that macros would have to dump all other macros in the same build tree in order to obtain that bug fix.
in a perfect world, macro libraries would also maintain compatibility branches for older swift-syntax majors and cherry pick these kinds of changes into those branches. but most existing (non-macro) libraries don’t provide multiple parallel series of releases and expecting macro libraries to start doing so is not very realistic.
There also isn't a straightforward answer to how macro library authors should do this kind of parallel versioning. The suggestion of minor releases for major swift-syntax bumps doesn't work for libraries that ship multiple macros or features and want to ship releases that are more typical semantic bumps: major, minor, or patch. The parallel series of releases may want to introduce new features to both the swift-syntax 510 release, as well as the swift-syntax 509 release, but there's no way to wiggle a minor release in between the ones that were used to bump the dependency.
@ahoppen Are macro libs loaded to the same address space or each one lives in its own sandbox?
If they are isolated from each other I guess it shouldn't be a problem to mix several versions of swift-syntax
Macros are independent executables that the compiler spawns (macros distributed with Apple's SDKs are dylibs, but they don't apply to this discussion). So there's no requirement that all macros use the same version of SwiftSyntax from that lower level point of view.
The broader issue, as I understand it, is that Swift Package Manager is limited to a single version of a dependency across the entire package graph of a build.
Though if that is solved then the build time issue may feel more pronounced. If 4 packages in your project depend on different swift-syntax versions, that'll add 16 minutes to a release build.
As discussed during the initial pitch for macros, lifting this limitation in SwiftPM's design is a large undertaking. Solving it will eventually be necessary for sure, but it isn't really a viable solution for the foreseeable future.
I think it's been clear for a while that the way we've implemented macros has some advantages, but scaling isn't one of them.
It works, but if you have, say, 20 packages in your dependency graph, all using macros for various cool features, written against whatever version of SwiftSyntax was latest for them at the time, you're going to end up downloading and building a lot of copies of that library. Every design has strengths and weaknesses, and this is ours.
The simplest short-term solution I can think of would be to cache SwiftSyntax build artefacts, so that "clean builds" aren't quite totally clean builds. They would be more similar to the kind of clean build you get when using binary dependencies -- your code would be built from scratch, but some parts (the various copies of SwiftSyntax) would be precompiled.
this wouldn't be an issue if SPM supported distributing pre-built target binaries for select platforms. why must building from source always be all or nothing?
Or less of an issue if it cached build artefacts more broadly (e.g. for all builds on a given Mac, at least).
Thankfully I don't have too many projects using SPM and there's not a huge amount of overlap in what packages I use, but even so I still notice Xcode sometimes lovingly building an artisanal version of common packages for each of my projects. I appreciate Xcode's commitment to a delightful and personalised build experience, but I'd rather it just give me my build products promptly.
Language changes may require breaking swift-syntax API changes, hence the major version update.
swift-syntax 510 will not use 5.10 language features, just as 509 does not use 5.9 language features. We support the latest release + previous major version. So 510 would support back to Swift 5.8.
SwiftPM currently doesn't pass through the user module version and as you noted, it's also underscored right now. There's PRs up for the first, though I don't expect them to get into 5.9.0 at this point.
Having said that, there's another similar solution - swift-syntax can include a new (empty) SwiftSyntaxVersion<version> module. So swift-syntax 509 would have SwiftSyntaxVersion509 and 510 would contain SwiftSyntaxVersion510 as well as SwiftSyntaxVersion509 . Macros plugins can then declare compatibility with both swift-syntax 509 and 510 by specifying the swift-syntax version range as "509.0.0"..<"511.0.0" . The macro can then use #if canImport(SwiftSyntax510) to check if it is building agains swift-syntax 510 and provide swift-syntax 509 compatible code in the #else branch. This should avoid the need for the parallel series of releases mentioned in a few posts, though still has the issue that all macros being used in a project will need to have updated for 510 to be used.
Alex has put up a PR on swift-syntax for this purpose.
Thanks for the clarification. That is an interesting workaround with the empty modules (fake defines), but I think just two major releases back will be too short for wide compatibility between packages. I think this will eventually cause issues in the wild, but I guess the community could always come up with a bridge API (or high level API) to centralize fixing backwards compatibility issues.
I would like to suggest using the "module" backwards compatibility for as long as reasonable even if it is only guaranteed for a major release back.
I think this approach is also going to result in people doing something like "509.0.0"..<"599.0.0" for the dependency and just use those modules to present a compile time error if it doesn't support the language version they are looking for. I'm not sure if this is a good situation or too much of a hack.
Alternatively, is there a plan to allow different compiler plugins to target different versions of swift-syntax with SPM?
We may want to come up with clearer language here, as the previous major release of Swift would be version 4, by semantic versioning at least. If nothing else, Swift needs to clarify its versioning vocabulary across projects and how it compares to semantic versioning.
Whoops, thanks for catching! This was really just meant to mean the last couple releases of Swift, ignoring patch releases (ie. either major or minor).
It still feels like this doesn't address semantic versioning though... I'll be quiet after this since I realize this can be an issue for another day and the marker-modules fix some of the issues, but I think one would want to target future Swift point releases too so all of your Macro packages can be updated gracefully.
This would essentially make all your macro packages stuck at Swift 5.11 even if one of them wanted to use newer language features from Swift 5.12 and the other macros were still targeting a backwards supported release of 5.11.
I understand the point that SwiftSynax is unstable at the moment. However this doesn't mean it cannot be stabilized and adopt SemVer. IHMO, it absolutely must be stabilized as it's now a part of a publicly available feature in the language. No one would be happy supporting their macros for a variety of incompatible versions of SwiftSyntax even with compiler conditionals.
We could release the first stable version of SwifSyntax together with Swift 5.9. Let's suppose it will be 5.9.0 to mach the language features it supports. Then for Swift 5.10 there will be a 5.10.0 SwifSyntax release backward compatible with 5.9.x etc. So a macro with upToNextMajor: "5.9.0" won't stop using libraries with a newer SwiftSyntax up until Swift 6.
Am I missing something here?
Just encountered another example of the issue. We have a package that we wanted to add an optional module to that introduced some swift-syntax code, but the act of depending on swift-syntax can cause downstream dependency resolution issues even if the optional module isn't explicitly depended on.
This means its impossible for a package to introduce a macro or other swift-syntax code without potentially breaking apps and libraries that depend on it, since they may already be depending on another version of swift-syntax. Instead we are forced to break out the optional code that needs to touch swift-syntax into its own package, which is quite a bit of work and quite laborious, both for us and for folks that want access to this new functionality.
This is really starting to feel like an impending nightmare with how pervasive swift-syntax will be in the package ecosystem, so it'd be nice to have some hope for the future of introducing dependencies on swift-syntax.
(I know this is a daft question but…) Is swift-syntax strictly necessary, ie is it possible to construct a macro without it and if so does anyone know of any examples?
I appreciate swift-syntax covers many scenarios but given the input and output to the compiler plugin have to be valid I’d (maybe) rather have a simpler coder which I can inline to the project.
You could write a compiler plugin which does not use swift-syntax, but it's not a very practical option. There's a lot of complexity being handled for you on the officially blessed path. Could be a fun project, but I wouldn't want to ship anything relying on it.
