Evolving SwiftPM Builds with Swift Build

Development Swift Build
1

Today, the new Swift Build open source project was announced with the goal of delivering a new set of powerful build technologies as part of the Swift toolchain. One opportunity presented by this new project is unifying SwiftPM’s build engine implementation across all the platforms Swift supports to deliver a more consistent cross-platform experience and enable powerful new features for package developers. I wanted to write up some initial thoughts on what this evolution of SwiftPM could look like to share with folks. I’m looking forward to hearing your feedback! You can feel free to reply here, and there’s also a new Swift Build forums category for other questions and discussions regarding the new project.

-- Owen

One of SwiftPM’s primary functions is to build packages and produce binaries and other products requested by the user. The component responsible for performing these builds is the SwiftPM build system. The build system plays an important role in providing a great developer experience for Swift users on all of its supported platforms. Its capabilities enable higher level features which determine how users architect and configure their projects, and its performance and reliability have a direct impact on developer productivity.

This document proposes a new technical foundation for the SwiftPM build system based on Swift Build. In the short term, this would be designed as a transparent migration for SwiftPM users that improves the cross-platform feature parity and behavioral consistency of SwiftPM. In the long run, this work can provide the infrastructure necessary to lift longstanding limitations of the current SwiftPM build system.

Goals

There are a number of technical goals I think are worth considering when evolving the technical underpinnings of the SwiftPM build engine. At a very high level, SwiftPM’s implementation should be well positioned to support desired end-user features, adopt new compiler advances, and deliver new developer productivity improvements. Specific proposed (but not exhaustive) goals include:

Providing a great build experience on (and targeting) all of Swift’s supported platforms

It’s worth explicitly stating that as it evolves, the SwiftPM build system must continue supporting all of the platforms it does today, while also laying the foundation for future expansion. SwiftPM should have the same high-level feature set on all platforms, and build behavior should be predictable and consistent.

Adopting the library-based SwiftDriver

The libSwiftDriver library allows higher level build systems to merge Swift’s module-level build graph with their own, higher level build graph. This allows a higher level build system to schedule tasks more efficiently by avoiding overcommit of system resources. Merging the granular tasks needed to build a single Swift module into the higher-level graph of an entire build request is also an important enabler of further build graph optimizations.

Providing a technical foundation for more expressive build configuration

Users are currently limited in the types of build configuration they can express in a Package.swift manifest. The SwiftPM build system’s implementation should become more flexible that it’s possible to design more advanced configuration options and a wider array of specialized product types (e.g. GUI applications, web services).

Supporting new build graph optimization techniques

libSwiftDriver adoption allows for more granular scheduling of individual swift-frontend jobs. This gives build systems the ability to partially interleave the compilation of dependent modules, allowing for greater parallelism. For example, the build system may prioritize the frontend invocation responsible for emitting a module’s swiftmodule file, allowing compilation of dependent modules to run in parallel with the frontend jobs responsible for producing object files. Other build graph optimizations may take advantage of unique characteristics of a target platform. For example, on Apple platforms TBD files can be used to shortcut linking in incremental builds where exported symbols don’t change.

Supporting global scheduling of explicitly built modules

Explicit compilation of Clang and Swift modules has a number of benefits. By eliminating the locking which occurs in implicit module builds, it allows for better build system scheduling decisions which reduce contention and allow for additional parallelism. Explicit module builds also improve the overall correctness of module builds by eliminating unsound sharing of content in the global module cache. The debugger can leverage modules generated during the build to accelerate expression evaluation. Errors encountered when building module dependencies are clearer and easier to understand. And, by precisely capturing the dependencies of a translation unit, explicit modules make it easier to soundly share and reuse build outputs across incremental builds.

Adopting Swift Build as SwiftPM’s new build engine would make it possible to work towards achieving the goals laid out above. In the short term, Swift Build’s extensibility makes it easier to implement and organize domain specific functionality like platform support and product types. It already has support for key compiler technologies like the library-based SwiftDriver and explicitly built modules, meaning those technical investments do not need to be replicated in the SwiftPM build system to unblock work on the end user features they enable. Finally, it contains a number of performance optimizations which build on these investments to go beyond what SwiftPM supports today, including the build graph optimizations described above.

Work Needed to Adopt Swift Build as SwiftPM’s Low Level Build System

The work needed to successfully adopt Swift Build as a replacement for SwiftPM’s current build system can be divided into two broad categories: feature parity, and platform support parity.

First, SwiftPM builds using Swift Build must have full feature parity with the current build system. Today, SwiftPM has basic support for building packages on Apple platforms using the build system bundled inside Xcode via the --build-system xcode option. However, this option does not support the complete SwiftPM feature set, and is not currently supported on non-Apple platforms. To achieve feature parity, we’d like to more tightly integrate Swift Build with the SwiftPM codebase, and build out support for missing manifest features like package plugins and macros. Higher-level features relying on information about the build will need updates to work with Swift Build as well. For example, SourceKit-LSP’s background indexing can be taught to use Swift Build’s index preparation and indexing info API.

SwiftPM builds using Swift Build must also support all of SwiftPM’s currently supported platforms. Currently, Swift Build has mature support for building on and targeting all Apple platforms, and has preliminary support for building on and targeting Linux, Windows, and Android. Further development is required to achieve Linux/Windows parity with the current SwiftPM build system. Swift Build will also need to add support for using Swift SDKs for cross-compilation by bridging them to its existing internal SDK and toolchain abstractions.

A Note on Platforms Support

Swift Build contains support for building software using a number of Apple-specific tools and product types. Now that it’s been contributed to the Swift project, we’d like to take a more principled approach to how this platform-specific support is organized as part of our efforts to provide first class support for additional non-Apple platforms. We’ve moved support for a number of tools, like the Asset Catalog and Core Data compilers, into Swift Build’s SWBApplePlatform plugin, and we intend to continue this process of separating support for Apple platform technologies from the core build engine implementation. Even though this platform support is moving into plugins for organizational purposes, it will remain a part of the open source Swift Build distribution, in order to ensure that open source clients like SwiftPM can continue leveraging it.

This same plugin-based approach has been used to successfully develop initial support for building projects targeting Linux and Windows using Swift Build. We’re excited to continue this work in open source to ensure the new project provides full feature parity and a great experience when building for any of the platforms in the Swift ecosystem.

Next Steps

The process of integrating Swift Build with SwiftPM and implementing the required features to allow adopting it as the SwiftPM build system is just getting started. We’re sharing our ideas early so that the Swift community can participate throughout the process and provide feedback. You can check out the initial pull request to start integrating Swift Build as an alternate build system for SwiftPM here. We’re looking forward to hearing your thoughts and discussion, and excited to work with you all to improve the experience of building Swift code!

59 Likes
2

Are there any specific types/configurations of SwiftPM projects that would be particularly helpful to test? (known unknowns)

Any that are known not ready (as of Feb 1, 2025) that would be premature to test and just add noise to report a problem?

2 Likes
3

Since the new build system is using CAS for caching build outputs to speed up incremental compilation, can the cache be also used across machines? If so, will it be possible to plug in a custom backend like other build systems, such as Bazel?

3 Likes
4

And, by precisely capturing the dependencies of a translation unit, explicit modules make it easier to soundly share and reuse build outputs across incremental builds.

Here, I'm primarily referring to two areas of improvement:

  1. Soundness improvements through use of explicitly built modules versus the implicit module cache. This avoids issues related to false sharing and cache pollution through more precise tracking of modular dependencies, leading to more reproducible builds and more precise error messaging.

  2. Sharing modules between build products and lldb to accelerate expression evaluation and other operations which otherwise are required to rebuild modules implicitly from scratch.

3 Likes
5

It is early days so there's lots to do, so it'll be a bit rocky out of the gate. We know we need to make some changes and additions to the PIF builder to catch up with the latest SwiftPM features but the basics should work. But once the initial PR lands and you can try it out, please do raise issues for anything you see broken. That way we can make sure we don't miss anything.

And if anyone one wants to help with the effort, PRs are open :). We should co-ordinate though to make sure we aren't duplicating effort, so please leave notes on Issues if you're looking at them. I'm following all the SwiftPM Issues so will help steer the traffic there.

7 Likes
6

Just to add to what Doug said, yes! We'd very much appreciate hearing from anybody who is willing to try out the integration and report issues they run into, even in these early stages of the project where there are some known gaps in functionality. As I said above, maintaining compatibility with existing packages is a requirement of this type of transition, and issues, forums feedback, and PRs are all very valuable contributions towards that goal.

For folks looking to try out Swift Build as an alternate SwiftPM build system, @cmcgee1024 's PR here is a great place to start. Currently, this requires building the branch from source, but the intent is that once this lands it will be as easy as passing --build-system swiftbuild on the command line.

3 Likes
7

Are there any plans in having a central system directory where dependencies, and their build outputs, are stored to achieve better performance? Re-compiling the same version of swift-syntax in multiple projects is not funand a waste of system resources.

5 Likes
8

In the SwiftPM PR there's a README that gives a small overview of what I expect could work and what probably will not work at this moment.

2 Likes
9

@cmcgee1024 We should probably take that and raise Issues for things that don't work. We have a tag "swift build" that we can use to track these. That will give us a real time look at what's left to do. I'll start raising ones for things I see.

3 Likes
10

This is really great news, thanks so much for open sourcing this.

One thing that I'm quite interested in is cross-compilation and remote debugging support - I think that is something that Xcode does extremely well for iOS devices, and you really wish for that kind of experience when targeting other platforms, too (such as Linux or embedded devices). Is that included in the goal to provide a great experience targeting all supported platforms?

Otherwise, I think the area where the current Swift build system falls short the most right now is macros. You mention that Swift Build does not currently support build-time products like package plugins or macros - can we expect that this will incorporate some lessons learned from SwiftPM and perform better, or is the initial support just focused on feature parity?

But yeah, very exciting!

9 Likes
11

Cross-compilation is something we've already begun to make some progress towards, as there is some initial support for targeting Android and QNX in the repo. WebAssembly also comes to mind. All of this also ties in with Swift SDKs support as well, which is something we're tracking: Swift SDKs Cross Compilation Support · Issue #6 · swiftlang/swift-build · GitHub

EDIT: To add some additional clarification, Swift Build is generally much more natively architected on the idea that all targets are cross-compilation targets, so we're already in a pretty good place here.

Debugging wise, I think there are definitely some interesting opportunities there. For example, perhaps swift-run could learn how to delegate to adb to allow seamlessly running a built command line tool on a connected Android device, or on similar devices running Linux or other embedded platforms.

6 Likes
12

This is great news! I am especially curious about one possible implication of this in particular: Could this lead to XCFramework modules (or equivalent) becoming available on non-Apple platforms? Being able to specify a pre-compiled library in the Package.swift file on Linux would be tremendously helpful for server-side Swift.

13

There’s the XCFramework format itself, and there’s the SwiftPM binary dependencies feature which happens to be built on top of XCFrameworks on Apple platforms.

When SwiftPM binary dependencies were originally proposed, the consensus landed on deferring support for non-Apple platforms due to various complexities especially on Linux.

Now that the reference implementation of XCFramework is Open Source, it may be worth revisiting the topic, but I could see it going either way - extending XCFrameworks could be the path forward, or perhaps SwiftPM’s binary dependencies feature could use one or more different formats for the other platforms, especially if there are existing formats that might be more appropriate.

I’d encourage reading up on [PITCH] Support for binary dependencies - #74 by jakepetroules if you haven’t already, which contains a lot of great background information and some of the difficulties on Linux in particular due to it having such a vast number of possible ABIs and no clear and canonical way to identify them.

Personally I think it wouldn’t be too hard to add at least Windows and Android, but we should go into more detail in a dedicated thread if you or someone else would like to kick off that discussion.

3 Likes
14

Is my understanding correct that SWBProtocol (Swift Build's binary protocol) is essentially a subset of XCBBuildService's binary protocol (i.e. Xcode's built-in build service's binary protocol)?

15

are there any implications for projects that use asset catalogs in packages that build via xcode but also standalone spm?

for example, will compiled assets and code-generation like in xcode be possible (at least on macos) in the future?

16

Also very happy to see this happen, thanks for open sourcing it!

Just would like to reply to this one shortly (and can follow up in the other thread on RLP linked to below as it may be more suitable).

Quoting from that link:

I would like to see a compelling argument for why we absolutely need binary packages on Linux when we have the system package manager to handle the vast majority (if not close to 100%) of those use cases, including for closed-source proprietary packages (of which there are few to begin with).

I just want to clearly state that there is a clear and important use case for those of us who wants to support customers building plugins vs. a resilient API in a controlled enterprise environment (where we as an ISV vend an API that our customers makes use of, but with strict constraints and control on runtime platform and toolchains used/supported). Happy to elaborate as needed, but it is a proven need with a few decades of background experience in various shapes and forms.

Some background here:

and a resulting warm and fresh pitch for a limited solution that supports this use case (instead of dismissing Linux as too complex and impossible to solve).

4 Likes
17

Not a subset, all of the SWBBuildService binary protocol is part of the Open Source Swift Build code.

3 Likes
18

As you note, Xcode projects have this behavior today via the current implementation in Swift Build (you can see some of the relevant code here).

Bringing that to Swift packages is certainly possible in principle. That would require an evolution proposal to add requisite new package manifest APIs to hook into and activate that existing functionality in Swift Build when building in a package context.

1 Like
19

That was my opinion 6 years ago. :slight_smile:

I agree there are opportunities for some fresh ideas in this space.

5 Likes
20

That's great to hear, we are very keen on enabling support for plugin architectures in controlled linux environments, waiting for a "generic solution" seems to let perfect be the enemy of good as the general problem is hard.

We hope that the related pitch can be a way forward, but let's follow up there.

5 Likes