Compilation extremely slow since macros adoption

Great!

Have you considered publishing these as binary dependencies for Swift Package Manager that can be used directly without downloading the repo locally? Simplest option is to have a thin layer secondary repo with the same Package.swift file that you have. Or the static linker patch issue prevents that?

Also, have you tried non xcodeproj path?

git clone https://github.com/apple/swift-syntax.git
cd swift-syntax
git checkout 509.1.1

xcodebuild archive -scheme SwiftBasicFormat -quiet -configuration Release -destination 'generic/platform=macOS' -archivePath /private/tmp/swift-syntax/SwiftBasicFormat-macos.xcarchive -derivedDataPath /private/tmp/swift-syntax/SwiftBasicFormat-macos.derived SKIP_INSTALL=NO BUILD_LIBRARY_FOR_DISTRIBUTION=YES SWIFT_SERIALIZE_DEBUGGING_OPTIONS=NO
xcodebuild archive -scheme SwiftBasicFormat -quiet -configuration Release -destination 'generic/platform=iOS Simulator' -archivePath /private/tmp/swift-syntax/SwiftBasicFormat-iossimulator.xcarchive -derivedDataPath /private/tmp/swift-syntax/SwiftBasicFormat-iossimulator.derived SKIP_INSTALL=NO BUILD_LIBRARY_FOR_DISTRIBUTION=YES SWIFT_SERIALIZE_DEBUGGING_OPTIONS=NO

Cannot use this command to create as the dir just contains object files SwiftBasicFormat.o SwiftSyntax.o SwiftSyntax509.o are not accepted:

xcodebuild -create-xcframework  -library /private/tmp/swift-syntax/SwiftBasicFormat-macos.xcarchive/<cannot use object files above> -output SwiftBasicFormat.xcframework

Can link this manually:

clang -shared -v -L/Applications/Xcode-15.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx  -o SwiftBasicFormat.dylib *.o
➜  Objects ls
SwiftBasicFormat.o  SwiftBasicFormat.dylib SwiftSyntax.o       SwiftSyntax509.o

Now it can create Xcframework but here the linking is not correct as SwiftSyntax and SwiftSyntax509 are embedded:

➜  Objects xcodebuild -create-xcframework  -library $PWD/SwiftBasicFormat.dylib -output /tmp/SwiftBasicFormat.xcframework 
xcframework successfully written out to: /tmp/SwiftBasicFormat.xcframework
1 Like

"Proxy/Local" binary frameworks are a potent idea that could be applied to many repos. I'm not familiar with SPM's repo but you could probably use an old version of Xcode 13 to generate a .xcodeproj from the Package.swift and it's largely RTFM from there. I singled out swift-syntax as it seemed to be causing huge discomfort in the macro community and I was looking at it recently for a PR. The static linking etc I had to get up to was only because macro plugins are sandboxed. It'd be far easier for other repos.

My take on why SPM coupling comes up specifically is these issues where people are having to have a dependency on the sourcekit-lsp repo (which depends on SPM) when all they are after is the Codeable data model for the LSP protocol so they can write clients. That really should be separated out into its own package.

Edit: I used .xcodeproj path largely because I had experience with it.

2 Likes

Ok have managed to also build a simple framework; the missing step is that in Package.swift each library needs to be defined as dynamic library :

diff --git a/Package.swift b/Package.swift
index b35a1999..56792aa8 100644
--- a/Package.swift
+++ b/Package.swift
@@ -43,7 +43,7 @@ let package = Package(
     .macCatalyst(.v13),
   ],
   products: [
-    .library(name: "SwiftBasicFormat", targets: ["SwiftBasicFormat"]),
+    .library(name: "SwiftBasicFormat", type: .dynamic, targets: ["SwiftBasicFormat"]),
[...]

And then this builds:

xcodebuild archive -scheme SwiftBasicFormat -quiet -configuration Release -destination 'generic/platform=macOS' -archivePath /private/tmp/swift-syntax/SwiftBasicFormat-macos.xcarchive -derivedDataPath /private/tmp/swift-syntax/SwiftBasicFormat-macos.derived SKIP_INSTALL=NO BUILD_LIBRARY_FOR_DISTRIBUTION=YES SWIFT_SERIALIZE_DEBUGGING_OPTIONS=NO

xcodebuild -create-xcframework  -framework /private/tmp/swift-syntax/SwiftBasicFormat-macos.xcarchive/Products/usr/local/lib/SwiftBasicFormat.framework -output /private/tmp/SwiftBasicFormat.xcframework 
xcframework successfully written out to: /private/tmp/SwiftBasicFormat.xcframework

but the linking here is wrong, it doesn't create a link to SwiftSyntax but it is fully embedded:

otool -L /private/tmp/SwiftBasicFormat.xcframework/macos-arm64_x86_64/SwiftBasicFormat.framework/SwiftBasicFormat 
/private/tmp/SwiftBasicFormat.xcframework/macos-arm64_x86_64/SwiftBasicFormat.framework/SwiftBasicFormat (architecture x86_64):
	@rpath/SwiftBasicFormat.framework/Versions/A/SwiftBasicFormat (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 2202.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.157.0)
	/usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 5.9.2)
	/usr/lib/swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
/private/tmp/SwiftBasicFormat.xcframework/macos-arm64_x86_64/SwiftBasicFormat.framework/SwiftBasicFormat (architecture arm64):
	@rpath/SwiftBasicFormat.framework/Versions/A/SwiftBasicFormat (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 2202.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.157.0)
	/usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 5.9.2)
	/usr/lib/swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

It seems like providing a pre-compiled version of Swift-Syntax is a pretty simple solution that Apple should provide to third-party macro creators.

Otherwise it really puts a wet blanket on the otherwise incredible capabilities that macros unlock. It would be awesome to see Apple support this directly.

13 Likes

@Sarunas, is it possible to have frameworks that are generated from Packages using SPM include the .swiftinterface files? Is there the equivalent of the Xcode setting "Build for distribution"?

Yes, it can be copied over:

name="SwiftSyntaxMacros"
platform="macos"

xcodebuild archive -scheme ${name} -quiet -configuration Release -destination 'generic/platform=macOS' -archivePath /private/tmp/swift-syntax/${name}-${platform}.xcarchive -derivedDataPath /private/tmp/swift-syntax/${name}-${platform}.derived SKIP_INSTALL=NO BUILD_LIBRARY_FOR_DISTRIBUTION=YES SWIFT_SERIALIZE_DEBUGGING_OPTIONS=NO

mkdir -p /private/tmp/swift-syntax/${name}-${platform}.xcarchive/Products/usr/local/lib/${name}.framework/Modules

find /private/tmp/swift-syntax/${name}-${platform}.derived/Build -type d -name ${name}.swiftmodule | xargs -I {} cp -r {} /private/tmp/swift-syntax/${name}-${platform}.xcarchive/Products/usr/local/lib/${name}.framework/Modules

xcodebuild -create-xcframework -framework /private/tmp/swift-syntax/${name}-macos.xcarchive/Products/usr/local/lib/${name}.framework -output /private/tmp/${name}.xcframework

Creates:

/private/tmp/SwiftSyntaxMacros.xcframework
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Resources
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/A
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/A/Resources
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/A/Resources/Info.plist
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/A/SwiftSyntaxMacros
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/Current
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/SwiftSyntaxMacros
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/x86_64-apple-macos.private.swiftinterface
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/arm64-apple-macos.swiftinterface
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/arm64-apple-macos.swiftdoc
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/x86_64-apple-macos.swiftdoc
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/arm64-apple-macos.abi.json
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/x86_64-apple-macos.abi.json
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/x86_64-apple-macos.swiftinterface
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/arm64-apple-macos.private.swiftinterface
/private/tmp/SwiftSyntaxMacros.xcframework/Info.plist

As for correct dynamic linking in SPM, this is not supported: Inter dynamic library dependencies · Issue #7327 · apple/swift-package-manager · GitHub

SwiftSyntax does have experimental Bazel build rules; and they do work but there is a limitation that they can only generate xcframeworks for macOS as Linux is statically linked as well: How to build shared libraries using rule_swift (.so)? · Issue #1141 · bazelbuild/rules_swift · GitHub

2 Likes

Thanks @Sarunas, that's useful information.

Highlighting this for the folks coming in new to the thread since there's been quite a bit of activity, and I've had a couple of folks message me off-forums asking me about the team's response. This is my attempt at summarizing the key points (please correct me if I'm wrong).

TLDR:

  • The Language Steering Group is aware of the build-system issues causing pain for macro adopters.

  • No timeline for a fix as of Feb 20th, 2024.


My two cents for Swift developers considering adopting macros:

  • Be aware of the fact that you're adding ~38,000 LoC of Swift source code to your project. This cost is forwarded to anyone consuming your Swift packages.
  • Be aware of the fact that SPM currently fetches the entire git history of the SwiftSyntax repo. This is a separate but, in my view, not unrelated issue as it directly factors into the cost of adopting macros.
  • Just adding SwiftSyntax can potentially add up to 12 minutes to your build time on Xcode Cloud as noted by @Ignacio_Soto.

It's peculiar enough that a core language feature is tied to a still-maturing package manager riddled with performance/build-systems issues, but what I find extremely disappointing is that these issues aren't mentioned anywhere in any official documentation/release notes.

While it's great that we can discuss this here on the forums, I don't think we should expect this to serve as a discoverable resource on the serious issues blocking folks on the adoption of macros.

If the LSG is aware of these issues as issues hindering adoption, one would expect that awareness to be reflected somewhere other than a forums thread. Perhaps an update/addendum to the Developer Experience section of the Swift 5.9 Released blog post is warranted?

If not, I'd love to know if there's an appropriate documentation article/resource that I can PR to!

46 Likes

Got some inspiration from @stephencelis and @mbrandonw swift-dependencies package and released swift-foundation-extensions@0.5.0. Macros are extracted to a separate target now, it will still affect package resolution, however it won't be compiled/linked by default :white_check_mark:

But I still would recommend to use .upToNextMinor(from: "0.x.x") binding if any package has major version equal to 0 :new_moon_with_face:

4 Likes

A fix is still pending as of April 27th, 2024.

I’m wondering why not push for adding swift-syntax to let’s say Swift 6 tollchain, but for distribution of precompiled swift-syntax itself :thinking:

3 Likes

A fix is still pending as of June 7th, 2024.

1 Like

will explicit modules help at all with the compilation time of swift-syntax?
it does mention that modules can be shared across targets so maybe that results in less swift-syntax rebuilds :hand_with_index_finger_and_thumb_crossed:

I haven't watched the videos about it yet, so I'm not sure what my expectations should be, but the explicit module option actually regresses build performance on one of my small projects by almost 50% (23s to 33s on an M1 Ultra). Xcode 16b1 generally regresses 10 - 12% over Xcode 15.4 (23s to 26s).

AFAIK there's no upcoming solution to macro build performance.

1 Like

@fmCarlos @Jon_Shier I'm also looking for any announcements/updates to binary syntax distribution for swift-syntax, and was unable to find them.

@vatsal

1 Like

Also hope to have fast compilation with macros usage

Still don't understand why people push for precompiled swift-syntax standalone distribution when it would be much more convenient to have it as a part of the toolchain, yes, it has some drawbacks like locking swift-syntax updates delivery to swift updates, but you'll have problems with one of those anyway, at least the usage would be much more convenient :tipping_hand_man:

However I came up with another idea that should fix the following issues

  • compilation time
  • swift-syntax version updates

What if macros could be precompiled? :exploding_head:

The problem with hidden malicious code could be solved with a combination of a few checksums, artifacts should be pretty small so they could be saved right in the repo. Client side won't even need to compile anything, just recalculate the checksum and use precompiled macros. And since they are precompiled, as macros dev you won't have to depend on .exact swift-syntax version to avoid conflicts on updates and as macros user you won't have to struggle from increased compilation times or care about fetching right swift-syntax version (which may cause conflicts btw), sounds like win-win to me :new_moon_with_face:

And it also should be relatively easy to design :thinking:

@Kyle-Ye has the (inferring here) decision to not utilize the support for xcframeworks to solve this for Apple platforms been discussed explicitly anywhere?

I repeatedly have asked this on forums and GitHub issues, and have not received a reply acknowledging linked community projects nor a straight answer to why it cannot be solved incrementally. There very well may be a good reason, but me along with others engaged in the discourse of this issue are left wondering.

I agree that new threads will not help accelerate the process, and can only reiterate my question hoping that we’ll receive some meaningful reply to at the very least glean insight on why xcframeworks cannot be used — even say, gated by an opt-in build flag, something that Package.swift is flexible enough to allow for without forcing it as a solution for every client of swift-syntax.

1 Like

Macros being precompiled has been tackled and solved IMO by a few folks (see CocoaPods). The practical blocker is meaningful integration with Xcode, and the fact that it is married to SwiftPM.

Heck, I’d be able to solve this issue if Xcode allowed me to use my own fork of SwiftPM, but AFAIK it uses a private fork of SwiftPM and selecting a custom toolchain breaks a lot of essential IDE features that are needed for developing on Apple platforms.

So here we are left unable to solve the practically adopt a fix interim despite being motivated enough to solve it ourselves, and without a clear answer on as to why xcframeworks cannot be used (which again there might be a good reason for, I’d really love for it to be articulated).

2 Likes