[Preview] Swift-Syntax Prebuilts for Macros

Development Package Manager
, ,
1

The introduction of macros to Swift has given developers the ability to create more expressive libraries that can be distributed as a Swift package. Macros can greatly improve the developer ergonomics making Swift packages an even more appealing environment in which to work.

One issue developers run into when adding dependencies on these packages is that build times all of a sudden get significantly longer. This is especially true when building in release mode and on CI machines with limited CPU and memory. There has been much discussion here: Compilation extremely slow since macros adoption, where people have reported extended build times of 5 minutes or more.

The key contributing factor to this build performance is the need to build the swift-syntax library that is the foundation of macro implementations. It’s a fairly large amount of Swift code required to create and present the Swift syntax tree to the macro.

As discussed at length here Distribute a prebuilt swift-syntax binary · Issue #2421 · swiftlang/swift-syntax · GitHub, one possible solution would be to provide a prebuilt library for swift-syntax that could be downloaded by Swift Package Manager and integrated into the build for macros. The good news, if you follow along with the SwiftPM PRs, is that’s what I’ve been working on for the last few months. With one final fix in Swift 6.1.1, it’s now ready for the community to try out.

Supporting prebuilt binaries for Swift can be challenging but SwiftPM takes care of that for you. It will download the prebuilt binary for the resolved version of swift-syntax, for the current running host platform and for the current version of the compiler and integrate them into the build. If prebuilts aren’t available for that combination, SwiftPM will build from source as it does today. Note that currently we have prebuilts for swift-syntax versions 600.0.1 and 601.0.1.

I encourage you to give it a try. I have added a flag to swift build and swift test to enable the feature:

swift build --enable-experimental-prebuilts

You should see an extra message when SwiftPM is downloading the zip file containing the prebuilts, and of course, your build should be faster, especially if you use -c release . You may also see your macros run faster since the swift-syntax prebuilts are built in release mode to help improve performance.

It is also available in Xcode 16.4 by enabling a user default.

defaults write com.apple.dt.Xcode IDEPackageEnablePrebuilts YES

Set it to NO to disable it or delete that key.

defaults delete com.apple.dt.Xcode IDEPackageEnablePrebuilts

Please report any problems you have to the SwiftPM Github Issues, GitHub · Where software is built. And, of course, I'd also love to hear if this is helping you. It has been fun to test it and I hope you find the same.

107 Likes
2

I'm glad to finally see this, thank you for the work you put into this feature!

7 Likes
3

Looks great! Nice to get that time back, and it'll be interesting to see if it helps with overall macro performance. Is there a message we should see in Xcode's log when it's downloaded or used? And in the future, is there a release cadence for prebuilts of new swift-syntax versions? Where should we report the need?

2 Likes
4

There isn't one right now. You just don't see Swift Syntax things getting built. Definitely something I want added in.

1 Like
5

It is very nice not to see swift-syntax built, especially the x86 version!

2 Likes
6

I'm hoping the prebuilts get done for every swift-syntax release. I've made it fairly easy to add new ones to the generator script (which is actually in SwiftPM).

2 Likes
7

Is this experimental feature supposed to work when a target has both macros and a plugin? Seeing the following error in an open source project that has both:

/Users/dfed/Library/Developer/Xcode/DerivedData/ExampleMultiProjectIntegration-ezjrqaajoaqjyhgouvuvughguadc/SourcePackages/prebuilts/swift-syntax/600.0.1/6.1-MacroSupport-macos_aarch64/include/_SwiftSyntaxCShims/module.modulemap:1:8: error: redefinition of module '_SwiftSyntaxCShims'
module _SwiftSyntaxCShims {
       ^
/Users/dfed/Library/Developer/Xcode/DerivedData/ExampleMultiProjectIntegration-ezjrqaajoaqjyhgouvuvughguadc/SourcePackages/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include/module.modulemap:1:8: note: previously defined here
module _SwiftSyntaxCShims {
       ^

Edit: This similar Package seems to compile on command line using swift build --enable-experimental-prebuilts, but there's no compile-time performance gain from using prebuilt binaries.

I think there are two issues here:

  1. The performance speedup seems to only be for macros, not plugins or other utilizations of SwiftSyntax
  2. In Xcode, utilizing both macros and plugins (or other utilizations of SwiftSyntax) can lead to compilation failure with the IDEPackageEnablePrebuilts user default enabled
8

This is great! Thanks a lot for doing this.

Are prebuilts only available for Swift syntax, or can any Package create prebuilts for its consumers? I could not spot where in swift-syntax/Package.swift at main · swiftlang/swift-syntax · GitHub the pre-built was described.

9

Packages can already do binary dependencies by publishing such.

10

Looks like this knocks our incremental build penalty from 40s to 16s in the Emit Swift Module phase, so the release build certainly helps with the overhead. On to the next bottleneck I guess.

2 Likes
11

#1. Yes, this is only for macros right now. That was done because we calculate where to use the prebuilts at package load time where we don't know what the target platform is. Macros are guaranteed to be host only where plug-in tools it is not.

I'll take a look at #2. So this is a plugin that depends on an executable (called a plugin tool) that also uses swift-syntax. Thanks for all your help finding that.

4 Likes
12

Ahh… interesting!

From the perspective of what happens after the build… do there exist any public artifacts in or adjacent to the compiled products that would indicate this product was or was not built with a swift-syntax prebuild dependency? Or is the intention for the prebuild and postbuild dependencies to produce the same identical outputs?

If there was no public artifact indicating this was from prebuild… are there going to be any kind of safety valves at all if the security of these binary prebuild artifacts are ever compromised?

Would we know if Apple would have any announcement to make as far as whether App Store submissions might have any restrictions on prebuild dependencies?

1 Like
13

This is only used for macro targets. The prebuilt library itself is not used in the product code.

And artifacts are signed to allow SwiftPM to ensure they are what was published.

3 Likes
14

Ahh… makes sense. Is the plan then for supporting "transitive" macro dependencies? Where MyLibraryPackage depends on YourMacroPackage that depends on swift-syntax? If I build MyLibraryPackage with enable-experimental-prebuilts do we plan to respect the prebuild option down through the dependency graph of MyLibraryPackage?

15

It already works like that. The prebuilt replaces the source build of swift-syntax if the resolved version has a prebuilt.

1 Like
16

Ah, I see @dfed. You have a library of extensions to swift-syntax that's used both in the macros and build plugin tool. That's causing the two environments to mix. Sorry about that. Thanks again for finding this. I'll work on a solution.

3 Likes
17

Is this limited to macOS or is there a solution for other platforms as well?

1 Like
18

Aha! That is indeed what I'm doing. Is the issue @dschaefer2 that I have an intermediary module with a dependency on SwiftSyntax, or is it that the intermediary module is shared with another executable that is not a macro? (The more I know, the more I can be clever and work around the problem in the interim)

Edit: looks like it was the intermediary module being shared that was the issue! Managed to work around it using symlinks rather than dependencies. My build times just improved a ton – thank you!

4 Likes
19

This whole feature seemed to be delayed due to building cross platform support, so yes, it supports many other platforms. You can see the full list of supported platforms in the manifest at https://download.swift.org/prebuilts/swift-syntax/601.0.1/6.1-manifest.json

6 Likes
20

Great! Saves nearly 3 minutes for a simple build on Linux!

Please note that it only works if using "swiftlang/swift-syntax", but does not if the older repo name "apple/swift-syntax" is still present in Package.swift.

Is there any way to pass --enable-experimental-prebuilts via environmental variable ? OTHER_SWIFT_FLAGS does not seem to be used.

Is there any plans to add iOS builds? For the use case when Xcode is used to build an XCFramework starting with:

xcodebuild archive -scheme SpmPackage -quiet -configuration Release -skipPackagePluginValidation -skipMacroValidation -destination 'generic/platform=iOS Simulator'
5 Likes