Compilation extremely slow since macros adoption

Great!

Have you considered publishing these as binary dependencies for Swift Package Manager that can be used directly without downloading the repo locally? Simplest option is to have a thin layer secondary repo with the same Package.swift file that you have. Or the static linker patch issue prevents that?

Also, have you tried non xcodeproj path?

git clone https://github.com/apple/swift-syntax.git
cd swift-syntax
git checkout 509.1.1

xcodebuild archive -scheme SwiftBasicFormat -quiet -configuration Release -destination 'generic/platform=macOS' -archivePath /private/tmp/swift-syntax/SwiftBasicFormat-macos.xcarchive -derivedDataPath /private/tmp/swift-syntax/SwiftBasicFormat-macos.derived SKIP_INSTALL=NO BUILD_LIBRARY_FOR_DISTRIBUTION=YES SWIFT_SERIALIZE_DEBUGGING_OPTIONS=NO
xcodebuild archive -scheme SwiftBasicFormat -quiet -configuration Release -destination 'generic/platform=iOS Simulator' -archivePath /private/tmp/swift-syntax/SwiftBasicFormat-iossimulator.xcarchive -derivedDataPath /private/tmp/swift-syntax/SwiftBasicFormat-iossimulator.derived SKIP_INSTALL=NO BUILD_LIBRARY_FOR_DISTRIBUTION=YES SWIFT_SERIALIZE_DEBUGGING_OPTIONS=NO

Cannot use this command to create as the dir just contains object files SwiftBasicFormat.o SwiftSyntax.o SwiftSyntax509.o are not accepted:

xcodebuild -create-xcframework  -library /private/tmp/swift-syntax/SwiftBasicFormat-macos.xcarchive/<cannot use object files above> -output SwiftBasicFormat.xcframework

Can link this manually:

clang -shared -v -L/Applications/Xcode-15.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx  -o SwiftBasicFormat.dylib *.o
➜  Objects ls
SwiftBasicFormat.o  SwiftBasicFormat.dylib SwiftSyntax.o       SwiftSyntax509.o

Now it can create Xcframework but here the linking is not correct as SwiftSyntax and SwiftSyntax509 are embedded:

➜  Objects xcodebuild -create-xcframework  -library $PWD/SwiftBasicFormat.dylib -output /tmp/SwiftBasicFormat.xcframework 
xcframework successfully written out to: /tmp/SwiftBasicFormat.xcframework
1 Like

"Proxy/Local" binary frameworks are a potent idea that could be applied to many repos. I'm not familiar with SPM's repo but you could probably use an old version of Xcode 13 to generate a .xcodeproj from the Package.swift and it's largely RTFM from there. I singled out swift-syntax as it seemed to be causing huge discomfort in the macro community and I was looking at it recently for a PR. The static linking etc I had to get up to was only because macro plugins are sandboxed. It'd be far easier for other repos.

My take on why SPM coupling comes up specifically is these issues where people are having to have a dependency on the sourcekit-lsp repo (which depends on SPM) when all they are after is the Codeable data model for the LSP protocol so they can write clients. That really should be separated out into its own package.

Edit: I used .xcodeproj path largely because I had experience with it.

2 Likes

Ok have managed to also build a simple framework; the missing step is that in Package.swift each library needs to be defined as dynamic library :

diff --git a/Package.swift b/Package.swift
index b35a1999..56792aa8 100644
--- a/Package.swift
+++ b/Package.swift
@@ -43,7 +43,7 @@ let package = Package(
     .macCatalyst(.v13),
   ],
   products: [
-    .library(name: "SwiftBasicFormat", targets: ["SwiftBasicFormat"]),
+    .library(name: "SwiftBasicFormat", type: .dynamic, targets: ["SwiftBasicFormat"]),
[...]

And then this builds:

xcodebuild archive -scheme SwiftBasicFormat -quiet -configuration Release -destination 'generic/platform=macOS' -archivePath /private/tmp/swift-syntax/SwiftBasicFormat-macos.xcarchive -derivedDataPath /private/tmp/swift-syntax/SwiftBasicFormat-macos.derived SKIP_INSTALL=NO BUILD_LIBRARY_FOR_DISTRIBUTION=YES SWIFT_SERIALIZE_DEBUGGING_OPTIONS=NO

xcodebuild -create-xcframework  -framework /private/tmp/swift-syntax/SwiftBasicFormat-macos.xcarchive/Products/usr/local/lib/SwiftBasicFormat.framework -output /private/tmp/SwiftBasicFormat.xcframework 
xcframework successfully written out to: /private/tmp/SwiftBasicFormat.xcframework

but the linking here is wrong, it doesn't create a link to SwiftSyntax but it is fully embedded:

otool -L /private/tmp/SwiftBasicFormat.xcframework/macos-arm64_x86_64/SwiftBasicFormat.framework/SwiftBasicFormat 
/private/tmp/SwiftBasicFormat.xcframework/macos-arm64_x86_64/SwiftBasicFormat.framework/SwiftBasicFormat (architecture x86_64):
	@rpath/SwiftBasicFormat.framework/Versions/A/SwiftBasicFormat (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 2202.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.157.0)
	/usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 5.9.2)
	/usr/lib/swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
/private/tmp/SwiftBasicFormat.xcframework/macos-arm64_x86_64/SwiftBasicFormat.framework/SwiftBasicFormat (architecture arm64):
	@rpath/SwiftBasicFormat.framework/Versions/A/SwiftBasicFormat (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 2202.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.157.0)
	/usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 5.9.2)
	/usr/lib/swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

It seems like providing a pre-compiled version of Swift-Syntax is a pretty simple solution that Apple should provide to third-party macro creators.

Otherwise it really puts a wet blanket on the otherwise incredible capabilities that macros unlock. It would be awesome to see Apple support this directly.

11 Likes

@Sven, is it possible to have frameworks that are generated from Packages using SPM include the .swiftinterface files? Is there the equivalent of the Xcode setting "Build for distribution"?

Yes, it can be copied over:

name="SwiftSyntaxMacros"
platform="macos"

xcodebuild archive -scheme ${name} -quiet -configuration Release -destination 'generic/platform=macOS' -archivePath /private/tmp/swift-syntax/${name}-${platform}.xcarchive -derivedDataPath /private/tmp/swift-syntax/${name}-${platform}.derived SKIP_INSTALL=NO BUILD_LIBRARY_FOR_DISTRIBUTION=YES SWIFT_SERIALIZE_DEBUGGING_OPTIONS=NO

mkdir -p /private/tmp/swift-syntax/${name}-${platform}.xcarchive/Products/usr/local/lib/${name}.framework/Modules

find /private/tmp/swift-syntax/${name}-${platform}.derived/Build -type d -name ${name}.swiftmodule | xargs -I {} cp -r {} /private/tmp/swift-syntax/${name}-${platform}.xcarchive/Products/usr/local/lib/${name}.framework/Modules

xcodebuild -create-xcframework -framework /private/tmp/swift-syntax/${name}-macos.xcarchive/Products/usr/local/lib/${name}.framework -output /private/tmp/${name}.xcframework

Creates:

/private/tmp/SwiftSyntaxMacros.xcframework
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Resources
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/A
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/A/Resources
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/A/Resources/Info.plist
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/A/SwiftSyntaxMacros
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/Current
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/SwiftSyntaxMacros
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/x86_64-apple-macos.private.swiftinterface
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/arm64-apple-macos.swiftinterface
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/arm64-apple-macos.swiftdoc
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/x86_64-apple-macos.swiftdoc
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/arm64-apple-macos.abi.json
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/x86_64-apple-macos.abi.json
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/x86_64-apple-macos.swiftinterface
/private/tmp/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Modules/SwiftSyntaxMacros.swiftmodule/arm64-apple-macos.private.swiftinterface
/private/tmp/SwiftSyntaxMacros.xcframework/Info.plist

As for correct dynamic linking in SPM, this is not supported: Inter dynamic library dependencies · Issue #7327 · apple/swift-package-manager · GitHub

SwiftSyntax does have experimental Bazel build rules; and they do work but there is a limitation that they can only generate xcframeworks for macOS as Linux is statically linked as well: How to build shared libraries using rule_swift (.so)? · Issue #1141 · bazelbuild/rules_swift · GitHub

2 Likes

Thanks @Sven, that's useful information.

Highlighting this for the folks coming in new to the thread since there's been quite a bit of activity, and I've had a couple of folks message me off-forums asking me about the team's response. This is my attempt at summarizing the key points (please correct me if I'm wrong).

TLDR:

  • The Language Steering Group is aware of the build-system issues causing pain for macro adopters.

  • No timeline for a fix as of Feb 20th, 2024.


My two cents for Swift developers considering adopting macros:

  • Be aware of the fact that you're adding ~38,000 LoC of Swift source code to your project. This cost is forwarded to anyone consuming your Swift packages.
  • Be aware of the fact that SPM currently fetches the entire git history of the SwiftSyntax repo. This is a separate but, in my view, not unrelated issue as it directly factors into the cost of adopting macros.
  • Just adding SwiftSyntax can potentially add up to 12 minutes to your build time on Xcode Cloud as noted by @Ignacio_Soto.

It's peculiar enough that a core language feature is tied to a still-maturing package manager riddled with performance/build-systems issues, but what I find extremely disappointing is that these issues aren't mentioned anywhere in any official documentation/release notes.

While it's great that we can discuss this here on the forums, I don't think we should expect this to serve as a discoverable resource on the serious issues blocking folks on the adoption of macros.

If the LSG is aware of these issues as issues hindering adoption, one would expect that awareness to be reflected somewhere other than a forums thread. Perhaps an update/addendum to the Developer Experience section of the Swift 5.9 Released blog post is warranted?

If not, I'd love to know if there's an appropriate documentation article/resource that I can PR to!

9 Likes