This is a pitch based on work I've been doing this summer and posted about earlier. This focuses on extending the capabilities of build tool plugins and used to build external targets and new build products. I'd love your feedback on the direction of this and on any details that seem cumbersome. The power this offers is important but so is the developer ergonomics for those trying to wield it.
Introduction
SwiftPM build tool plugins were introduced with SE-0303 as a strategy to allow package providers to extend the capabilities of SwiftPM. The original focus was on integrating source code generators into the build of targets. The capabilities were pretty limited.
This proposal extends build tool plugins to allow integration of more complex build tools beyond source file generation and to allow for targets to produce new types of build products for targets. One useful application of these extensions is to integrate complete build systems that can be used to build libraries from other ecosystems and leverage the resulting build artifacts in Swift packages.
A proof of concept PR has been created as a proving ground as work progressed to build the example package and test out the ergonomics of this and other features. https://github.com/swiftlang/swift-package-manager/pull/10374
This includes an example that integrates the build for SDL, a cross platform hardware integration library, and builds a simple cross platform SDL application and builds for host and for Android. This example is described in a section below.
A new PR is underway for this feature explicitly and will include an update version of the example as this PR evolves. Extending plugins to customize targets. by dschaefer2 · Pull Request #10667 · swiftlang/swift-package-manager · GitHub
Motivation
Swift's language interoperability with C and Objective-C were critical to it's early success. It allowed for incremental adoption of Swift allowing developers to take advantage of Swift's safey and ergonomics while continuing to interface with existing code. However, the capabilities of Swift packages have lagged behind the language and building mixed language systems has been awkward.
SwiftPM has been key for building up a rich ecosystem of Swift library packages that provide common components needed to build applications and services. To integrate these libraries into real applications that can are deployable on devices, developers usually have to fall back to alternative build systems to complete the task. Ideally they should be able to accomplish all their build and deployment needs without leaving packages behind. To put it simply, swift build should just work. Developers should be able to checkout a complex package and SwiftPM should be able to do what is needed, with the help of plugins as necessary, to produce deployable artifacts for that package and it's dependencies.
Proposed Solution
The build tool plugin API is extended to allow build commands to play a larger role in the build graph. This includes designating some of it’s output files as build products.
A build variable mechanism is added to allow build tools to gain more knowledge about the build request to ensure they are creating binaries in the correct format.
A new target type, external target, is added that enables the integration of non-Swift package sources. Build tools plugins on these targets are responsible for calling into the natural build system for these sources and make the build products available.
These capabilities are intended for the SwiftBuild build system only. Use of these features with the old Native build system is not supported.
Detailed Design
Build Command Extensions
The following extensions are added to build commands provided by build tool plugins. Note that prebuild commands are not changed in this proposal.
Build Variables
The new SwiftBuild system used by SwiftPM operates under the assumption that it serves multiple different build requests using the same project description. This is managed using macro expressions in the build settings which may have different values derived from the individual build requests. This allows the project description to be cross platform.
Since the build commands produced by build tool plugins are also part of the project description, these commands need to offer the same capability.
A build variable system is introduced for all strings and paths in build commands. These variables start with the dollar sign, $, and the name surrounded with parenthesis, e.g. $(SYSROOT). SwiftPM will replace these variables with values that take into account the parameters of the build request. The initial set of variables include:
CONFIGURATIONthe current build configuration, Release or Debug.TRIPLEthe compiler target tripleSDKROOTthe root of the SDK directorySYSROOTthe clang sysrootCLANG_RESOURCE_DIRthe clang resource directory for locating standard C libraries.SWIFT_RESOURCE_DIRthe Swift compiler resource directory for locating the Swift standard library.BUILD_SUBDIRa unique name that can be used to create a subdirectory in the plugin output directory with platform and build configuration specific binaries. This ends up being the same name as used in build product directories.
Other variables may be added overtime as needed.
Note that the underlying SwiftBuild macros are not directly exposed through this capability to ensure we keep the ergonomics clean and avoid the complex history of SwiftBuild's macros.
Un-deprecation of Path and adoption of FilePath
A number of the build variables resolve to absolute paths. As such, it's difficult to construct a URL that includes these variables as the prefix for the path, i.e. without a leading slash or drive letter. The original Path type specified in the plugin API is better suited for this. As a result, the deprecation notices for these types are removed.
As SE-0529 calls for Swift System's FilePath to be brought into the Swift standard library. As it supports paths as relative or absolute, support for FilePath will be added to the plugin API. The Path type is type aliased to FilePath and any functions needed to maintain backwards compatibility with Path will remain as extensions on FilePath.
Build Products
public static func buildCommand(
...
outputFiles: [Path] = [],
buildProducts: [BuildProduct] = [],
...
)
In SwiftPM, targets produce one or more files as output of the build of that target. It’s these files that are shared with targets that depend on that target. They are placed in a directory called the build products directory. Currently there is one shared build products directory for a given build request, though that may change in the future to help differentiate targets that produce files with the same name.
public struct BuildProduct {
public var outputFile: Path
public var productSubdir: String? = nil
public var conditions: CommandConditions? = nil
}
Build products specify the output file that is to be copied to the build products directory, an optional sub-directory of the build products directory to store it in, and conditionals on the build product. Note that it is not necessary to add the output file for the build product to the list of output files for the command as it is automatically added.
public struct CommandConditions {
public var platforms: [Platform]?
public var traits: Set<String>?
}
extension CommandConditions {
public static func when(platforms: [Platform], traits: Set<String>? = nil) -> Self {
.init(platforms: platforms, traits: traits)
}
public static func when(traits: Set<String>? = nil) -> Self {
.init(platforms: nil, traits: traits)
}
}
The command conditions are the same as target conditions in thepackage manifest and list constraints on platforms and/or traits. The Platform type used here is shared from the package manifest.
Always Out Of Date Flag
public static func buildCommand(
...
alwaysOutOfDate: Bool = false
...
)
When a build command has no input files, that command currently is only run once to produce the output files. When the build tool is another build system that is managing it’s own input file dependencies, such as Ninja, the build tool should be run on every build. A flag is added to specify that.
Conditions on Build Commands
public static func buildCommand(
...
_ conditions: CommandConditions? = nil
...
)
Conditions are made available on build products. However, the commands themselves may also need to be filtered to certain traits and platforms. Conditions are added to the build command as a whole as well. This could be used to have different command arguments based on the condition.
Target Extensions
To accommodate more powerful build tool commands, targets are extended in the Package Manifest as follows.
“No” Language Targets
With SE-0541, package targets were extended to allow source for both Swift and Clang-supported languages in the same target. This proposal adds the concept of a target with no Swift/Clang sources. The build for these targets are completely managed by the build commands.
Multi-language source code generation
To fully complete SE-0541, build tools may also contribute Swift and Clang source files to a target. This capability allows for generation of Swift bridge code to C libraries using source analysis tools that produce module maps for external C libraries with API notes and Swift binding code to simplify interfaces. Module map files as well as their header files and API notes are automatically detected and added to the build. This feature was previously available using the experimentalCGen feature tag on the manifest. It is now graduated to fully supported.
External targets
public final class Target {
public enum TargetType: String {
...
case external
...
}
public enum Location: Sendable {
case path(String)
case remoteArchive(url: String, checksum: String)
}
public var location: Location?
...
}
One of the powers with the extended build commands is the ability to call out to an external build system to build a set of sources. Since SwiftPM doesn’t manage those sources, a new type of target, an External Target is added to allow for specification of the location of the sources for that target. That path is provided to the plugins to find that source.
In this proposal, it is assumed that the location for those sources are relative to the package path, for example, as a git submodule or in a sibling directory.
public static func externalTarget(
name: String,
dependencies: [Dependency] = [],
location: Location? = nil,
cSettings: [CSetting]? = nil,
cxxSettings: [CXXSetting]? = nil,
swiftSettings: [SwiftSetting]? = nil,
linkerSettings: [LinkerSetting]? = nil,
plugins: [PluginUsage]? = nil
) -> Target
External targets have no path. The path property for the target is determined by the end location for sources.
public static func binaryTarget(
name: String,
location: Location
) -> Target
The Location type is also added for Binary Targets which have similar requirements.
Build Settings Header Path Usage
public enum BuildSettingUsage: Sendable {
case `private`
case `public`
case interface
}
public structure CSetting {
...
public static func headerSearchPath(_ path: String, usage: BuildSettingUsage = .private, _ condition: BuildSettingCondition? = nil) -> CSetting {
public static func define(_ name: String, usage: BuildSettingUsage = .private, _ condition: BuildSettingCondition? = nil) -> CSetting {
...
}
public structure CXXSetting {
...
public static func headerSearchPath(_ path: String, usage: BuildSettingUsage = .private, _ condition: BuildSettingCondition? = nil) -> CSetting {
public static func define(_ name: String, usage: BuildSettingUsage = .private, _ condition: BuildSettingCondition? = nil) -> CSetting {
...
}
To support package targets that depend on external targets, build settings are added to the external target. This may include the header search path to find header files for the external target as well as any preprocessor defines that are need to properly include those header files. A build setting usage property is added to those settings for C and C++ to specify how the setting is propagated.
If the usage is set to public, it is set on the target as well as any targets that depend on that target. If the usage is set to interface, it is only set on targets that depend on that target, not on the target itself. If the usage is set to private, it is only used in the build for that target. (Note: these concepts should be familiar to users of CMake).
Example: A Cross Platform SDL Application
An example Swift package is provided that builds a cross platform application using the popular SDL library and targets host and Android platforms. It served as a driver for the capabilities in this proposal that enables SwiftPM to help developers leverage Swift's language interoperability features. It has the following components.
SDL External Target
.externalTarget(
name: "SDL",
location: .path("SDL"),
cSettings: [
.headerSearchPath("include", usage: .public),
],
plugins: ["CMakeBuilderPlugin"]
),
This target assumes SDL is added as a git submodule at the root of the package. A public header path is added to expose the SDL header files to consuming targets. It use the CMakeBuilder build tool plugin to build SDL using CMake and Ninja.
Traits for Shared Versus Static
traits: [
"SDL_static",
"SDL_shared",
.default(enabledTraits: ["SDL_shared"])
],
Both static and share versions of the SDL library are supported. This is controlled by traits with the default being shared.
CMakeBuilder
let arguments = [
"--output-dir", buildDir.path,
"--triple", "$(TRIPLE)",
"--sysroot", "$(SYSROOT)",
target.directoryURL.path,
]
return [
.buildCommand(
displayName: "CMake Build",
executable: builder.url,
arguments: arguments,
environment: environment,
buildProducts: [
.init(buildDir.appending(path: "libSDL3.dylib"), .when(platforms: [.macOS])),
.init(buildDir.appending(path: "libSDL3.so"), .when(platforms: [.android])),
.init(buildDir.appending(path: "SDL3.lib"), .when(platforms: [.windows])),
.init(buildDir.appending(path: "SDL3.dll"), .when(platforms: [.windows])),
.init(buildDir.appending(path: "SDL3.jar"), .when(platforms: [.android])),
],
alwaysOutOfDate: true,
.when(traits: ["SDL_shared"])
),
.buildCommand(
displayName: "CMake Build",
executable: builder.url,
arguments: arguments + ["--static"],
environment: environment,
buildProducts: [
.init(buildDir.appending(path: "libSDL3.a"), .when(platforms: [.macOS, .android])),
.init(buildDir.appending(path: "SDL3-static.lib"), .when(platforms: [.windows])),
.init(buildDir.appending(path: "SDL3.jar"), .when(platforms: [.android])),
],
alwaysOutOfDate: true,
.when(traits: ["SDL_static"])
),
]
The CMakeBuilder plugin add two commands each gated on one of the two traits. The commands use build variables to determine the triple and sysroot for the platform requested in the build request. It defines the library as platform specific build products. For Android it also provides the Android bridging Java jar file.
SDL Swift Binding Target
.target(
name: "SwiftSDL3",
dependencies: [
"SDL",
],
linkerSettings: [
.linkedLibrary("SDL3", .when(platforms: [.macOS, .android])),
.linkedLibrary("SDL3-static", .when(platforms: [.windows], traits: ["SDL_static"])),
.linkedFramework("CoreMedia", .when(platforms: [.macOS])),
.linkedFramework("CoreVideo", .when(platforms: [.macOS])),
...
],
plugins: ["SwiftSDLGenPlugin"]
),
This target exposes the SDL3 library in a multi-language module. It uses a code generator to produce a module map, umbrella header files, API notes and Swift code that provides a clean Swift-y interface to the SDL API.
AndroidApp
.target(
name: "AndroidApp",
dependencies: [
.product(name: "MyAppCore", package: "swift-sdl"),
.product(name: "SwiftSDL3", package: "swift-sdl"),
],
plugins: ["ApkBuilderPlugin"]
),
This target produces an APK of the application deployable on Android devices using the APKBuilder plugin.
APKBuilder
let libs = ...find shared libraries in build products of target dependencies...
let javaFiles = ...find java files in the sources of the target...
let jarFiles = ...find jar files in build products of target dependencies...
let manifestFile = ...the AndroidManifest.xml file from the target sources...
let apk = ...build product for the generated APK file...
return [
.buildCommand(
displayName: "Building apk",
executable: builder.url,
arguments: [
"--name", target.name,
"--output-dir", outputDir.path,
"--swift-resource-dir", "$(SWIFT_RESOURCE_DIR)",
"--sysroot", "$(SYSROOT)",
"--manifest", manifestFile.url.path
] + javaFiles.flatMap { ["--java", $0.path] }
+ jarFiles.flatMap { ["--jar", $0.path] }
+ libs.flatMap { ["--native-lib", $0.path] },
inputFiles: javaFiles + jarFiles + libs,
productFiles: [apk]
),
]
The APKBuilder executes a series of command line tools provided by the Android SDK to produce a signed APK file. It fetches the location of build products from the target dependencies. The APKBuilder also copies the Swift and C++ runtime libraries into the APK from the swift resource directory and clang sysroot. It also builds any Java files in the target and adds the AndroidManifest.xml file.
Security
There are no new security issues introduced with this feature. As plugins become more powerful, it becomes more important to provide sandboxing solutions on all host platforms. Investigations are continuing on solutions.
Impact on Existing Packages
The new features are gated on the swift-tools-version of the next Swift release. Previous APIs will be marked deprecated in that release.
The APIs provided in the early plugin API that give access to build settings such as header paths and link libraries breaks the architectural philosophy of one project description for many build requests. While we continue to support these APIs, the intention is that build variables for commands should eventually replace them.
Alternatives Considered
Early in prototyping, external targets were modelled as complete packages with the plugin manifest supplied by consuming packages. That started down the path of supporting direct builds of the external packages with SwiftPM, but the ergonomics of overlaying the package concepts were cumbersome. Instead we focused more on using the natural build system for the external packages and providing plugin capability to call those build systems to produce the build products.
Future Directions
The proposal for generating C/C++ header files only supports files included from a module map. It may be desirable to add directories containing these header files to the header path for the target. This would require a new setting that would point at these directories while maintaining
While external targets will only support local paths to the source for the target initially, in future releases support will be added for the sources to be made available from remote locations, either as a source archive, or checked out from source control.
When fetched from an external location, it should also be possible to apply patches to the downloaded source.
Many external packages also provide binary distributions of the libraries they provide. It would help build times and make integrations easier by supporting integration of these prebuilt binaries. As we learned form swift-syntax prebuilts, SwiftPM’s current platform conditions isn’t sufficient to properly identify what platforms a given binary supports. Concepts like version of libc, CPU architecture, and potentially even triple and SDK need to be considered.
It should be possible to find external sources or binaries in alternative package managers. For example, we should be able to add a dependency on jars managed by Maven central and then consumed by plugins that use swift-java to produce Swift bindings to those libraries and that product Java applications that use Swift native libraries. Also, package managers like nuget or vcpkg are able to serve up C/C++ libraries that could be incorporated into Swift applications. A new plugin capability should be added to allow Swift package manifest to express dependencies on these external packages and to invoke the appropriate tool to make those packages available.