Macros and XCFrameworks

Development Macros
, , ,
1

As those who have started adopting macros have found out, swift-syntax takes a while to build. On my M1 Pro, that's around 15 - 18 seconds. For smaller apps, I feel like this is an unacceptable wait - it's the vast majority of the compile time in fact.

I did try to search these forums and the web at large, but it seems like nobody has asked the question yet, so I will - is it possible to build a XCFrameworks of the various swift-syntax products and use them as dependencies to macro packages?

I'm starting to lean toward no based on my experiments so far. What I've tried:

  1. clone swift-syntax
  2. edit product types of SwiftSyntaxMacros and SwiftCompilerPlugin to .dynamic
  3. create XCFrameworks from them
  4. add these as binary targets to an example macro package, replacing the dependency on swift-syntax

That didn't work, because (I think) the dependencies of those two frameworks are not included in them. I then repeated the process, but with all the products in swift-syntax, putting that in a wrapper package and then importing that into the macro package. The macro target compiles, but with this warning, which leads me to believe it won't actually work:

<unknown>:0: warning: compiler plugin not loaded: 'path/to/macro/deriveddata/Build/Products/Debug/CaseMacroMacros; failed to initialize

I won't pretend I'm an expert on dependencies or XCFrameworks, so I'd like to ask here - is what I'm attempting even possible? This is for internal project use, there is no issue with having to upgrade this for new swift-syntax as there are no downstream clients.

What I find interesting is that searching the web didn't really yield any discussions of this. I don't understand how people are OK with adding almost 20s of build time to their projects just to use macros...

10 Likes
2

As an alternative, I was looking into putting the actual macros inside an XCFramework, but I can't make that work either.

Though that has to work somehow, since SwiftUI has the #Preview macros as part of the framework :thinking:

1 Like
3

Macros run in a sandbox, so they won't be able to link any external dynamic libraries. You may have more luck with building your XCFrameworks with static libraries which should at least work in theory.

I believe during the pitch phase, we did consider macros themselves to be shipped as statically linked binaries, but I don't remember the specifics of why we didn't do that.

1 Like
4

Thanks for responding!

Do you mean to somehow compile swift-syntax statically? If so, how would one do that? I heard about using XCFrameworks to wrap static libraries, but I'm not sure how to even get the libraries in the first place.

5

Yes, if you change the product type to .static you should get static libraries at least when building with swift build -- not 100% sure what the behavior of Xcode is. But you should be able to take the built products from swift build into an XCFramework as well.

6

I thought that is the default behavior - I specifically had to add type: .dynamic to get a framework.

The documentation around generating XCFrameworks says

To include a static library file (.a file), replace -framework with -library in the command above.

But building products of swift-syntax (e.g., SwiftCompilerPlugin), I can't seem to find any such files in the .build directory - is there something I need to specify to get those?

7

The default is .automatic which only produces .o files. You only get .a files when there's an explicit .static product type.

2 Likes
8

Ok so the issue was that I was also trying to build just one target using --target. Once I used just swift build, all the .a files were present :+1:

However, I can't say I made it work. Creating XCFrameworks worked fine, but using them in the macro package did not. At first I was getting

No such module 'SwiftCompilerPlugin'

I thought that was because a .swiftinterface file was missing from the XCFrameworks, because once I added those (by building with -Xswiftc -emit-module-interface -Xswiftc -enable-library-evolution), that error was replaced with

/path/to/derived/data/Build/Products/Debug/SwiftSyntaxMacros.swiftinterface:5:8: error: no such module 'SwiftDiagnostics'
import SwiftDiagnostics
       ^
/path/to/derived/data/Build/Products/Debug/SwiftCompilerPlugin.swiftinterface:5:8: error: failed to build module 'SwiftSyntaxMacros' for importation due to the errors above; the textual interface may be broken by project issues or a compiler bug
import SwiftSyntaxMacros
       ^
/path/to/package/CaseMacro/Sources/CaseMacroMacros/CaseMacroMacro.swift:2:8: error: failed to build module 'SwiftCompilerPlugin' for importation due to the errors above; the textual interface may be broken by project issues or a compiler bug
import SwiftCompilerPlugin
       ^
/path/to/derived/data/Build/Products/Debug/SwiftSyntaxMacros.swiftinterface:5:8: error: no such module 'SwiftDiagnostics'
import SwiftDiagnostics
       ^

Sure enough, SwiftSyntaxMacros.swiftinterface contains things like

import SwiftDiagnostics
import SwiftSyntax
import SwiftSyntaxBuilder

And something tells me it wouldn't end there if I made frameworks for those. This feels like I'm doing something wrong, but can't tell what it is :sweat_smile:

9

I think that's right, you will likely need to make individual XCFrameworks for most of the modules in swift-syntax.

10

Alright, after some more experimentation, I was able to make this work!

It looks like only about half of the products of swift-syntax are required to actually build and use the macro. Making XCFrameworks out of all of them, sticking them in either the macro's package, or a wrapper so that only one thing needs to be specified in the macro package, allows me to build the macro package and use the macro.

The only remaining issue is that I get a warning about duplicate linked libraries for those swift-syntax products:

ld: warning: ignoring duplicate libraries: '-lSwiftBasicFormat', '-lSwiftCompilerPlugin', '-lSwiftCompilerPluginMessageHandling', '-lSwiftDiagnostics', '-lSwiftIDEUtils', '-lSwiftOperators', '-lSwiftParser', '-lSwiftParserDiagnostics', '-lSwiftRefactor', '-lSwiftSyntax', '-lSwiftSyntaxBuilder', '-lSwiftSyntaxMacroExpansion', '-lSwiftSyntaxMacros', '-lSwiftSyntaxMacrosTestSupport'

I don't understand where these are coming from, but even just silencing the warning would be great - is there a way to do this?

1 Like
11

Might have spoken too soon. When I try to use a macro package with these dependencies in another package, I get the following error in Xcode:

Failed to receive result from plugin (from macro 'CaseExtractor')

And now it works for some reason :thinking:

Must have been my changes to the package manifest (trying to get rid of the warning), but it turns out the warnings don't show when building the consuming package anyway :smile:

12

Alright, so I tried to integrate this into our actual app. The app uses Pointfree's excellent swift-snapshot-testing library, which happens to also depend on swift-syntax, which results in the following error during package resolution:

multiple similar targets 'SwiftBasicFormat', 'SwiftCompilerPlugin', 'SwiftCompilerPluginMessageHandling' and 11 others appear in package 'swift-syntax' and 'swiftsyntaxwrapper', this may indicate that the two packages are the same and can be de-duplicated by using mirrors. if they are not duplicate consider using the moduleAliases parameter in manifest to provide unique names

Can module aliases actually help in this case? I looked over the proposal and the only place I can think of using them would be the macro package. I tried declaring aliases for some of the XCFrameworks, which lead to a bizzare error:

cannot refer to module as 'SwiftSyntaxMacrosAlias' because it has been aliased; use 'SwiftSyntaxMacros' instead

I guess I don't understand, because from where I'm sitting, it's telling me that I can't use the alias I declared... because the alias exists... and to use the original (clashing) name instead :exploding_head:

13

Before alias, perhaps try forking the library and see if you can get it to depend on a binary swift-syntax at all?

If you can get this working it would be a great boon to the community, even if you have to vend the binaries outside swift-syntax.

1 Like
14

I'm pretty sure it could work, but this approach doesn't scale. I don't want to (have to) maintain a fork of every library that we adopt that happens to use the package just to be able to use macros without building swift-syntax. Concerns around swift-syntax are already being discussed:

But that's not the point of this thread.

For now, since aliasing doesn't seem to be working for me (possibly because I'm using it wrong), I'm probably going to try... a different kind of aliasing by renaming parts of the swift-syntax package before building, hopefully eliminating the clash that way...

In case anyone is wondering why I care about building macros fast if the app already depends on swift-syntax, the app is heavily modularized and it is rare devs need to build the whole thing - rather, the typical workflow is opening small feature packages and working on those. And in such cases, building swift-syntax would be the majority of compile time.

15

My point was, if you can prove that a binary swift-syntax can work at all, we may be able to cut off this build time explosion before it gets out of hand, either by having swift-syntax itself ship binaries, or you yourself shipping them. If you're not interested, can you at least post your build scripts / commands for the binary swift-syntax? At least then others could step into the void.

1 Like
16

Hmm ok I’ll try to put what I have on GitHub once I’m done with these aliasing experiments.

1 Like
17

Binary libraries are only a thing on Darwin platforms, so using binaries would require every package that uses swift-syntax to offer a Darwin and a non-Darwin version. Since ABI stability doesn't exist on non-Darwin (at least I think that's still the status quo), it won't be possible to ship binary Swift libraries on non-Darwin platforms any time soon.

When we were designing macros, I argued for macros themselves to be shipped as statically linked executables when releasing a package (basically remote dependencies would have been required to offer binaries, local development would be the same as it is today). We ultimately didn't go that route, but I think that's still the most viable path to not having to build swift-syntax as a mere user of macros.

1 Like
18

@NeoNacho I know SwiftUI is an Apple framework, so technically not part of Swift as such, but the #Preview macro is somehow shipped as part of that framework - how is that done? Is this something we could do as well?

19

Is this solvable by letting users have a conditional dependency (if they support non-Apple platforms at all) that uses the binary on Apple platforms and compiles from source on others?

20

I don't believe so since both libraries would be using the same module names. Since SwiftPM uses the module names to reference to targets, there's no way I can think of to declare these two variants. Potentially #if os() could be used with the caveat that it breaks cross-compilation (which at least today isn't a huge concern since macros are not supported when cross-compiling with SwiftPM, but may be problematic for the future).