C/C++ Interop improvements with SwiftPM for libraries

Evolution Discussion
, ,
1

Hi,

this should just be a discussion around possible future improvements around C/C++ Interop in SPM for libraries.
Basically every library for common technologies in the server-side space has a C++ library available. With C++ Interop now there, in theory, these libraries could also be available for Swift users, with (if at all) just a small wrapper library necessary. This would boost the possibilities for adoption for Swift on Server massively as all the technologies would be available.

But this is, at least in my experience, only theory. Many of the bigger libraries don’t have a nice /include directory with all the headers but use a rather complicated build system, mostly done with Cmake, also including dependencies. Replicating the build in SPM is nearly impossible for such systems which limits the possibilities for the use of C++ libraries tremendously.

Currently there are 2 options I know of available:

  1. Using unsafe options to statically link the libraries. This works but nobody can use your package as it includes unsafe options.

  2. Completely refactor the library to bring it into a format SPM can work with. This has the drawback that it takes a. much time to do and b. you have to port all the changes over to your version of the package manually everytime there is an update.

Both options are not really usable. 1. is literally unusable and 2. can take so much time that you could ask yourself why not just stick with C++ and/or Java instead of refactoring possibly dozens of libraries.

My wishes

First I have to say I‘m not a C/C++ developer, nor Cmake user and not an expert in the Swift build system, nevertheless I would suggest some points we can at least discuss to make the situation better for such usecases.

1. Directly support Cmake with SPM

Possibly the easiest solution from a Package Developer‘s view would be to directly support Cmake or at least make builds. This support could provide SPM with all the necessary public headers and the exact way on how to build and link the C/C++ target. While I can imagine that this is a complicated step from SPM perspective, it would make the usability nearly perfect

2. Allow the linking args as a safe flag

This would allow statically linking C++/C binaries into the Swift Package. This is currently possible with stating the direct compiler options but only as an unsafe flag which makes the package not usable as a dependency. If we would find a way to allow some advanced linker options as safe flags, that would make it possible to rather easily work with C++ libraries in Swift. If the user doesn’t trust the binaries, they can just rebuild from source and switch out the binaries for their own.

3. Extend the build system of SPM massively

This idea is related to 2 but goes into all directions. The problem now is that we are not really able to replicate the Cmake files in SPM‘s build system. If we were to massively extend the options for SPM C/C++ builds, it would probably be possible, at some point, to just build these libraries with native options.

These would be some ideas of mine to make the C++ Interop for libraries much better to support my suggested use case. Hope we can discuss my ideas/bring new ideas from you into the mix. Hopefully the one or another proposal will come out of this thread.

Happy holidays!

6 Likes
2

I vaguely remember build plugins being created for this purpose. I've never written one though, so I could be wrong. @NeoNacho or @abertelrud, what do you think?

3

Build plugins currently cannot process C/C++ sources or headers last time I tried that. Xcode (not SPM) did support yacc and bison files strangely enough but doesn’t compile the produced C sources.

4

I vaguely remember talk of using them to run CMake and other external build tools before the SwiftPM build began. That way they could be used to enable 1. from his options above, ie run CMake and ninja on the C/C++ source, then have SwiftPM link Swift code against those freshly-built C/C++ libraries.

I have no idea if build plugins enable all that though, hence my question for SwiftPM devs.

1 Like
5

I’m not entirely convinced that extending Swift Package Manager (SPM) can address all use cases effectively, given the complexity and lengthy process involved in implementing such improvements.

In my experience, integrating Bazel with Swift Package Manager worked quite well when I tested it, using the following tool: rules_swift_package_manager. This integration extracts the necessary third-party package metadata, downloads them via https/git and generates proper Bazel BUILD files automatically.

For projects relying heavily on C++ libraries, using CMake as the primary build system simplifies external library management significantly. However, the real challenge arises when working with SPM. If the community could develop robust CMake integration for SPM, it might provide a viable solution. Given that CMake is a Turing-complete programming language, this seems like a feasible approach. :slight_smile:

6

fwiw, @stackotter has enabled SwiftBundler to integrate upstream projects and use custom builders (defined in Swift files) to build and bundle those libraries and executables alongside your SwiftPM project.

These custom builders make it possible to build existing C++ projects with CMake and support other build systems as well.

7

and for anyone trying to figure out the new Bundler.toml config with these updates, here is what works for me:

8

This is very work in progress, people should probably at least wait until I've ironed things out and documented the configuration format. Hopefully within the next week or so.

For anyone interested though, here's an example configuration file for how you can pull in the sentry-native library (built with cmake) using Swift Bundler from commit 741487a255d03d74e5668a013988b758ea251abb. With this configuration, you can import GitHub - stackotter/swift-sentry: A Swift wrapper on the native Sentry SDK as a dependency of your Swift package and all the cmake stuff should just happen automatically whenever you run swift bundler run or swift bundler bundle.

format_version = 2

[apps.Example]
product = "Example"
version = "0.1.0"
identifier = "dev.stackotter.Example"
# Dependencies get built first and get used during various
# phases of your main app's building and bundling steps depending
# on the product type of each dependency.
dependencies = ["sentry.sentry", "sentry.crashpad_handler"]

[projects.sentry]
source = "git(https://github.com/stackotter/sentry-native)"
revision = "1206a3ad0f0fc09fa3c8a5c6a71db216130c3db9"

[projects.sentry.builder]
name = "CMakeBuilder.swift"
type = "wholeProject"
# This defines the version of Swift Bundler that the builder gets
# built against. The only requirement is that the CLI is newer than
# the revisions required by your builders.
api = "revision(741487a255d03d74e5668a013988b758ea251abb)"

# The product's filename is determined based off target platform.
# E.g. on macOS it's inferred to be `libsentry.dylib`.
# Libraries get copied into the products directory (e.g. `.build/debug`)
# so that SwiftPM can find them without any special linker flags.
[projects.sentry.products.sentry]
type = "dynamicLibrary"

# Executables get copied into your app bundle next to your main
# executable. Generally they're assumed to be helper executables.
[projects.sentry.products.crashpad_handler]
type = "executable"
output_directory = "crashpad_build/handler"

And here's the generic cmake builder used in this example. Just put
it at the root of your project (builders are referred to by path.

import SwiftBundlerBuilders

@main
struct CMakeBuilder: Builder {
    static func build(_ context: some BuilderContext) throws -> BuilderResult {
        try context.run("cmake", ["-B", context.buildDirectory.path])
        try context.run("cmake", ["--build", context.buildDirectory.path])
        return BuilderResult()
    }
}

Future plans

  • The default SwiftPM build will act as a synthesised default project which you can override in the configuration file
  • The product field of your app will be able to refer to any executable product from any subproject. E.g. you could bundle up the crashpad handler as a standalone app with product = "sentry.crashpad_handler". product = "Blah" will just be shorthand for product = "default.Blah"
  • The builder API will be expanded upon so that all functionality of the current default SwiftPM builder can be defined as an external builder
  • Dependencies of your SwiftPM package will be able to include Bundler.toml in their repositories that configure dependencies built with external tools, to allow packages like thebrowserco/swift-sentry to work out of the box if consumers of the package are using Swift Bundler (without affecting regular SwiftPM users of the package, who already have to manually build sentry-native and copy it into their products directory before building their app).
  • Builders will be able to define their own configuration fields so that they can be configured centrally via the Bundler.toml to make them more reusable
  • Builders will be able to be pulled in from external sources so that reusable builders (e.g. a flexible cmake builder) can be provided in git repositories

I've probably forgotten a few plans here, but those plans should give a general idea of the direction at least.

5 Likes
9

I believe that build plugins only handle the case of generating Swift source files for now. They can't produce compilation products such as dylibs or anything like that (and can't even generate c/c++ sources as Christophe mentioned).

I remember being quite excited about the prospect of build plugins when they were first pitched, because I thought they might allow me to integrate Rust code into SwiftPM builds, but that sorta support never materialised.

1 Like
10

I guess the "easiest" fix for now would be to allow .a files to get linked without calling -L and -l as an unsafe flag. So a mix between systemLibrary and target, where we can just put our library inside the target‘s directory together with the headers and modulemap and SwiftPM links it.

1 Like