I've been working on various new areas for Swift, and as part of those efforts I've increasingly ran into the limitation that SwiftPM cannot compile languages other than Swift, C and C++.
In my particular use case, I want to run AI/ML inference on-device with GPU acceleration (CUDA). My preferred tool for the job is mlx-swift. This does since recently support CUDA, but only through CMake... and SwiftPM does not support importing CMake products.
The MLX example is interesting for Apple platforms too. Here, MLX relies on being built through xcodebuild. SwiftPM itself does not know how to compile metal shaders, therefore MLX-Swift does not work through SwiftPM on Apple platforms.
There are more examples where this becomes a visible limitation.
Other language libraries that need to be wrapped in Swift - but expose a C FFI header
Mixed-language projects on non-apple platforms. This could potentially even enable some Objective-C code to run on Linux (through a community plugin).
To enable this, I envision a new SwiftPM plugin. It has the ability to compile a source file (unknown to SwiftPM) into a .o on SwiftPM's behalf. The exact API would need to be thought out carefully of course. We'd need to pass along and consider things like:
Exposing the compiled symbols (.h)
Registering relevant file types
What if multiple provides register the same file extension?
Letting SwiftPM know of the intended parallelisation (or vice versa)
Probably a lot more I'm unaware of
I would love to hear your thoughts on this possible avenue. I think it would unblock a lot of use cases.
Thanks @Joannis_Orlandos! I think this would solve a number of use cases where developers are trying to use different languages that can generate object code instead of Swift/C/C++ source.
To help me understand what the plugin shape should look like, do you have an example of the command that generates the object files for your scenario? In particular, what would be needed in its plugin context to be able to generate that command.
We should also consider sandboxing requirements for those platforms that support sandboxing. Do we need to change anything from the current build plugin tools implementation. And more generally, are there security concerns that need to be mitigated, or at least described.
We then need to consider the build system implications. How does it find out about the object files and how does it integrate those into the product builds, e.g. archiving, linking, etc. That gets much deeper into what the implementation would look like. The timing is interesting since we haven’t got rid of the Native build system yet for SwiftBuild. Do we need to add this to Native?
I think we can do something really good here. Are there other use cases that people have run into that could benefit from this?
I had a similar need for the swift-temporal-sdk where we are now distributing pre-compiled binary artifacts. I think this is great way to extend plugins but we probably need to give the plugin a lot more context about the build. At least about the target triplets that we are building for but probably more like the build configuration etc.
Part of the problem with supplying the build request, which contains the triple(-ish), in the plugin context is that the plugins run before we know what the build request is.
In particular with SwiftBuild, the PIF that describes the targets in the build is created once and then reused for different build requests that may target different platforms and architectures, for example.
Once we’re fully on SwiftBuild, we should be able to take advantage of its build setting macro support to be able to pass the triple in the args for the Command the same as is done with clang and swift compiler calls.
I'm very supportive of extending plugins to support this - IMO allowing them to directly provide linker inputs generalizes well to support a lot of different use cases. A few miscellaneous notes from our earlier discussion on this, in no particular order:
Exposing more information about the current build request:
There are a few things which are likely required to be exposed in the plugin context to allow a plugin to produce valid linker inputs:
target triple (or triples, when building a universal binary)
SDK/sysroot
Others may not be strictly required, but beneficial to expose. For example, which sanitizers are enabled, the clang resource dir, etc.
Registering relevant file types
I think this could largely work the same way it does for source-generating plugins today, by allowing plugins to inspect the project model and choose which inputs to process. It may be worth formally documenting the order multiple plugins are applied in so it's clear which wins in the case multiple are capable of consuming some input.
Exposing the compiled symbols (.h)
This opens up a fairly large design space and may deserve being split out and discussed separately. A few things which likely need to be sorted out:
How are headers exposed to downstream targets? Is a plugin allowed to impart search paths on direct and transitive dependencies? Should they be covered by an automatically generated modulemap?
Are headers exposed to Swift/C compiles in the target which generates them (since this can create unexpected ordering dependencies)? Swift Build does do this for Swift/ObjC interop on Apple platforms by using a VFS to exclude the Swift-generated header from clang modules before it's been produced (when compiling the Swift code itself). However, it's not clear this would generalize to arbitrary compilers.
Header file generation is a separate issue that people have brought up. My previous post on that provides a short term incomplete “fix” to at least unblock them. It is indeed wrought with issues so we definitely need to revisit that.
I could turn the .a into a .o, but it would be more convenient if SPM allowed the .a directly.
That is another use case we should look at as well. In this case the plugin should be able to provide the output for the target. We would still need to know that this is a static library target so it could be consumed properly.