A precompiled swift-syntax and optimized macro plugins won't fix the issues you've quoted, just make them somewhat less visible, as the macros would execute more quickly. We'd still pay the cost of a slow emit modules step that waits for all of the macro expansions to finish, among other things.
If it's swift-syntax building, then yeah. But even then, they shouldn't be building incrementally. This is probably worth its own Issue report.
Hi folks, looks like there was some work on this issue. @dschaefer2 thank you so much for looking into this. Could you please tell us if there are any updates? Did your solution work or what is the plan here?
Yup, working on a solution now. Essentially the plan is to prebuild the swift-syntax modules needed for macros for all our supported host platforms for the last two major releases of swift-syntax and for the last two releases of the swift compiler. Swift Package Manager will resolve the dependencies for macros and if the required version of swift-syntax has binaries available it'll download them into the cache and extract them into the scratch directory so we can build against them. If it's in the cache, we of course skip the download. And if binaries are not available, we build from source as we currently do. As such, there's no changes needed to your packages for this to work.
One interesting advantage we get from this is that for macros to be performant we really need swift-syntax to be built in release mode. And that can take at least 3 minutes of build time. So this should be good all around.
I'll post again once I start pushing PRs up to swift-package-manager so you can follow along. That should probably be right after the US thanksgiving holiday next week. Being in Canada, that'll give me a nice quiet week to get some code done :).
69 Likes
@dschaefer2 it’s great to hear that these longstanding compile-time issues are being tackled.
Curious about the following:
-
Is this a swift-syntax specific optimization (i.e. hardcoded for swift-syntax usage with macros)?
-
Would I be correct in understanding that these SwiftPM improvements wouldn’t be usable in Xcode until the Xcode team updates the version of SwiftPM being shipped? Last I recall selecting a custom toolchain made no difference to the version of SwiftPM being used in Xcode.
It’s great that these issues are being tackled, and I’d love to get an understanding of when iOS developers (a majority of them tied to Xcode) will be able to take advantage of these improvements.
1 Like
This is actually macro specific and specific to swift-syntax. Basically that's being done to keep things simple for now since the place I'm adding this doesn't know what target platform you're building for. With macros, they're always the host platform so I can cheat a little. I do think we need a more general strategy for build caching but that's a much bigger architectural change.
Being an open source forum we don't talk about futures of commercial product. But let's say I know where said team sits.
5 Likes
@dschaefer2 got it, thanks for clarifying!
I'm personally in favor of anything that makes it simple to improve the situation right now, really looking forward to this being shipped with Xcode.
2 Likes
How is this work proceeding? Are you able to manage peoples expectations on when this important fix might make it into Xcode?
2 Likes
It was merged last week and should hopefully go out with Swift 6.1. If Apple keeps to the same release schedule as the last several years, that should mean the spring Xcode release (likely 16.3 or 16.4). The big remaining question is whether Xcode itself will properly integrate it into its builds or whether it will only work in SPM directly.
9 Likes
Can someone give an answer to this? Thanks.
1 Like
It's unlikely anyone who knows would tell us, we'll have to wait for the first beta to ship in a month to six weeks.
2 Likes
First Xcode 16.3 beta is here with Swift 6.1, but I don't don't see anything different in regards to swift-syntax (still builds from source, still builds universally), and there aren't really any Swift changes in the release notes (aside from C++ interop). So @dschaefer2, anything you can tell us here?
7 Likes
It looks like the work is still underway, with another pull landing just last week. Doug seems to be aiming for the Swift 6.1 release, as those prebuilt library pulls now only go into the release/6.1 branch, not even pushing them to trunk yet.
I'm sure they'll let you know once they have it working for you to try out, as unlike previously, you can see there is work on this ongoing in the OSS SwiftPM repo, though obviously nobody is guaranteeing any deliverable or release date yet.
8 Likes
The sheer delay on this and the lack of public commitment to when this will actually ship for the majority of Swift developers right now (i.e. folks practically confined to Xcode), an issue that disrupts the adoption of a critical language feature, is severely demoralizing.
19 Likes
No pressure, just curious 
but have improvements been released in 6.1 or will be aimed for 6.2?
3 Likes
Happy New Year folks. As a binary solution for swift-syntax didn't seem to turn up with Swift 6.1, I've refreshed the repo I mentioned upstream and it works a lot better this time as the support for binary frameworks seems far more robust in Xcode 16.3.
The usage instructions are as before. Clone the repo, and rename it:
git clone https://github.com/johnno1962/InstantSyntax -b main --single-branch swift-syntax
Then, drag the directory of the resulting binary version of swift-syntax onto the top level your project and it will take the place of the source dependency in the sidebar. No more random minute long pauses while swift-syntax compiles and macros become as lightweight as they should be to add to your project. Someone more expert than me will probably be able to work out how to incorporate this package into CI but I don't know if something like it could work on Linux.
There are two known problems: SwiftUI previews of a Mac app don't work and sometimes your project builds but you can see distracting errors in the source editor where you use a macro. If you encounter this, there is a script to run passing the path to the macro executable mentioned in the error which makes this problem go away next time you make an edit. If you'd prefer to roll your own version directly from Apple's source on site there's a script for that.
If you encounter problems please raise an issue on the repo. If you don't, share the love and let people know here there is a form of solution to this long standing problem.
3 Likes
FWIW, although SwiftSyntax is a huge problem, it's far from the only problem. For example, we have warnings turned on for long compilation times, and just adopting swift-testing and adding 3 #expect calls to a test function, was enough to hit a 200ms compilation time warning for that one test function. It does depend a bit on how you word the test assertions (simple is better, unsurprisingly!), but it's not good.
We have ~18000 unit tests; we could not accept 200ms/test — that's a full hour of additional build time.
5 Likes
If you added more test methods you'd likely find that not all #expect statements produce that warning. They might, but those warnings often show up once for similar code, as it's the first time the compiler has to solve the typing for it, which it can then reuse in later statements. If not, it might be worthwhile for Swift Testing to produce more fully typed code in its macros, as it's one place we can hide that overhead while maintaining the benefits.
1 Like
Comparing notes with other teams who have looked at this problem I've been able to make the binary frameworks for swift-syntax simpler to use. It seems you can include static libraries in binary frameworks and this has been possible since Xcode 11. The means it is now possible to publish a binary replacement for swift-syntax that can be simply added as a SPM dependency to your app.
To try it out, add the following URL as a top level Swift Package dependency of your project that is using macros: GitHub - Concoction/swift-syntax: Binary .xcframeworks for swift-syntax. and it will replace the version with the same "identity" (last path component) that was compiling from source all the time used by your macros. Resolution is easier if you just use the branch main. If you check in your modified project it will also work for CI/CD to make it faster. I've built the new .xcframeworks using Xcode15.4 and they seem to work back to Xcode 15.0!
I'm kicking myself I didn't realise this was possible sooner and if you ask me someone at Apple should be feeling the same way though it seems they were caught in the thinking error that "no solution" was better than a "solution that didn't work for all platforms" all this time (despite the vast majority of developers using one platform).
I'm a firm believer that conversion of large dependencies to binary frameworks offers a way to ease the load on Xcode which is struggling to scale as reuse of large source repos increases. As ever, there is a short script in the repo which allows you to completely rebuild the binary frameworks from source for peace of mind which should generalise to generating binary equivalents of other repos.
13 Likes
As a general rule, there's very little type information we can add to the expansion of #expect that the compiler won't have already. That said, #840 will change the calculus.