[Pitch] Replaceable Library Plugins

Evolution Pitches
1

hi Swift Evolvers!

this is the Swift Evolution Pitch for the feature previously discussed in Can You (Dynamically) Link Swift Libraries on Linux?

if you want to test-drive this feature at home, clone the implementation PR of SwiftPM and build the modified package manager, and try it out with the end-to-end example projects here:

Example Producer: swift-rlp-example
Example Consumer: swift-rlp-example-client

or, alternatively, you can try your hand at creating your own Replaceable Libraries! :sunglasses:

i would like to again thank Ordo One for their generous support in making this work possible! :gift_heart:

Replaceable Library Plugins

Introduction

SwiftPM currently has no support for non-system binary library dependencies on Linux. This proposal adds support for Replaceable Library Plugins, which are a type of dynamic library that is shared across a fleet of machines and can be upgraded without recompiling and redeploying all applications running on those machines. We will distribute Replaceable Library Plugins through the existing .artifactbundle format.

Swift-evolution thread: Discussion thread topic for that
proposal

Example Producer: swift-rlp-example

Example Consumer: swift-rlp-example-client

Motivation

Many of us in the Server World have a Big App with a small component that changes very rapidly, much more rapidly than the rest of the App. This component might be something like a filter, or an algorithm, or a plugin that is being constantly tuned.

We could, for argument’s sake, try and turn this component into data that can be consumed by the Big App, which would probably involve designing a bytecode and an interpreter, and maybe even a whole interpreted domain-specific programming language. But that is very hard and we would rather just write this thing in Swift, and let Swift code call Swift code.

While macOS has Dynamic Library support through XCFrameworks, on Linux we currently have to recompile the Big App from source and redeploy the Big App every time the filter changes, and we don’t want to do that. What we really want instead is to have the Big App link the filter as a Dynamic Library, and redeploy the Dynamic Library as needed.

Proposed solution

On Linux, there are a lot of obstacles to having fully general support for Dynamic Libraries. Swift is not ABI stable on Linux, and Linux itself is not a single platform but a wide range of similar platforms that provide few binary compatibility guarantees. This means it is pretty much impossible for a public Swift library to vend precompiled binaries that will Just Work for everyone, and we are not going to try to solve that problem in this proposal.

Instead, we will focus on Replaceable Library Plugins (RLPs). We choose this term to emphasize the distinction between our use case and fully general Dynamic Libraries.

Organization-Defined Platforms (ODPs)

Unlike fully general Dynamic Libraries, you would distribute Replaceable Library Plugins strictly for internal consumption within an organization, or to a small set of paying clients.

The organization that distributes an RLP is responsible for defining what exactly constitutes a “platform” for their purposes. An Organization-Defined Platform (ODP) is not necessarily an operating system or architecture, or even a specific distribution of an operating system. A trivial example of two ODPs might be:

  1. Ubuntu 24.04 with the Swift 6.0.3 runtime installed at /home/ubuntu/swift
  2. Ubuntu 24.04 with the Swift 6.0.3 runtime installed at /home/ubuntu/swift-runtime

Concepts like Platform Triples are not sufficient to describe an ODP. Even though both ODPs above would probably share the Triple aarch64-unknown-linux-gnu, Swift code compiled (without --static-swift-stdlib) for one would never be able to run on the other.

Organizations add and remove ODPs as needed, and trying to define a global registry of all possible ODPs is a non-goal.

To keep things simple, we identify ODPs by the URL of the Artifact Bundle that contains the RLP.

Creating RLPs

To compile an RLP, you just need to build an ordinary SwiftPM library product with the -enable-library-evolution flag. This requires no modifications to SwiftPM.

You would package an RLP as an .artifactbundle just as you would an executable, with the following differences:

  • The info.json must have schemaVersion set to 1.2 or higher.
  • The artifact type must be library, a new enum case introduced in this proposal.
  • The artifact must have exactly one variant in the variants list, and the supportedTriples field is forbidden.
  • The artifact payload must include the .swiftinterface file corresponding to the actual library object.

Because SwiftPM is not (and cannot be) aware of a particular organization’s ODPs, this enforces the requirement that each ODP must have its own Artifact Bundle.

The organization that distributes the RLP is responsible for upholding ABI stability guarantees, including the exact Swift compiler and runtime versions needed to safely consume the RLP.

Consuming RLPs

To consume an RLP, you would add a binaryTarget to your Package.swift manifest, just as you would for an executable. Because ODPs are identified by the URL of the Artifact Bundle, there are no new fields in the PackageDescription API.

We expect that the logic for selecting the correct RLP for a given ODP would live within the Package.swift file, that it would be highly organization-specific, and that it would be manipulated using existing means such as environment variables.

Deploying RLPs

Deploying RLPs does not involve SwiftPM or Artifact Bundles at all. You would deploy an RLP by copying the latest binaries to the appropriate @rpath location on each machine in your fleet. The @rpath location is part of the ODP definition, and is not modeled by SwiftPM.

Some organizations might choose to forgo the @rpath mechanism entirely and simply install the RLPs in a system-wide location.

Detailed design

Schema extensions

We will extend the ArtifactsArchiveMetadata schema to include a new library case in the ArtifactType enum.

public enum ArtifactType: String, RawRepresentable, Decodable {
    case executable
+   case library
    case swiftSDK
}

This also bumps the latest schemaVersion to 1.2.

Artifact Bundle layout

Below is an example of an info.json file for an Artifact Bundle containing a single library called MyLibrary.

{
    "schemaVersion": "1.2",
    "artifacts": {
        "MyLibrary": {
            "type": "library",
            "version": "1.0.0",
            "variants": [{ "path": "MyLibrary" }]
        }
    }
}

The artifact must have exactly one variant in the variants list, and the supportedTriples field is forbidden. An RLP Artifact Bundle can contain multiple libraries at the top level.

Below is an example of the layout of an Artifact Bundle containing a single library called MyLibrary. Only the info.json must appear at the root of the Artifact Bundle; all other files can appear at whatever paths are defined in the info.json, as long as they are within the Artifact Bundle.

đź“‚ example.artifactbundle
    đź“‚ MyLibrary
        ⚙️ libMyLibrary.so
        đź“ť MyLibrary.swiftinterface
    đź“ť info.json

A macOS Artifact Bundle would contain a .dylib instead of a .so. RLPs will be supported on macOS, although we expect this will be an exceedingly rare use case.

Security

RLPs are not intended for public distribution, and are not subject to the same security concerns as public libraries. Organizations that distribute RLPs are responsible for ensuring that the RLPs are safe to consume.

Impact on existing packages

There will be no impact on existing packages. All Artifact Bundle schema changes are additive.

Alternatives considered

Extending Platform Triples to model ODPs

SwiftPM currently uses Platform Triples to select among artifact variants when consuming executables. This is workable because it is usually feasible to build executables that are portable across the range of platforms encompassed by a single Platform Triple.

We could extend Platform Triples to model ODPs, but this would privilege a narrow set of predefined deployment architectures, and if you wanted to add a new ODP, you would have to modify SwiftPM to teach it to recognize the new ODP.

Supporting multiple variants of an RLP in the same Artifact Bundle

We could allow an Artifact Bundle to contain multiple variants of an RLP, but we would still need to support a way to identify those variants, which in practice makes SwiftPM aware of ODPs.

We also don’t see much value in this feature, as you would probably package and upload RLPs using one CI/CD workflow per ODP anyway. Combining artifacts would require some kind of synchronization mechanism to await all pipelines before fetching and merging bundles.

One benefit of merging bundles would be that it reduces the number of checksums you need to keep track of, but we expect that most organizations will have a very small number of ODPs, with new ODPs continously phasing out old ODPs.

Using a different ArtifactType name besides library

We intentionally preserved the structure of the variants list in the info.json file, despite imposing the current restriction of one variant per library, in order to allow this format to be extended in the future to support fully general Dynamic Libraries.

13 Likes
2

My general reaction here is a tepid -0.

This pitch barely changes any of the infrastructure of Swift, so it’s very hard for me to actually be meaningfully opposed to it. Adding a new schema version to artifact bundles costs us little. Given that Ordo-One believes it has a use-case, I can’t in good conscience stand in the way of that.

However, I am concerned that this is an extremely expert-level feature that doesn’t resolve the actual problem that the ecosystem needs solved. In fact, I worry that it helps those most able to solve the general problem, reducing their incentive to try to tackle the bigger issue.

Deferring the concerns about managing ABI stability onto the user seems to me to be likely to produce mystifying issues for end users. Many users are not equipped to understand the problems they will face with deploying ODPs. Keeping track of what changes have an ABI impact is not trivial, and this pitch proposes no strategy to help users on this front.

I will note that there seems to be a perception in the community that the general solution is hard. It is not. The prior art for the general solution is in the Python community, and has been for many years. The majority of that solution is not about building new infrastructure. Instead, it’s about building tooling to streamline the production of manylinux wheels, and to help tell users when their wheel is not meeting the requirements for distribution. Defining the requirements is the hardest part.

I do think it’s a bit of a shame that we as a community keep deciding that “just copy Python” is too hard. But that is not a reason to oppose this pitch, so I don’t.

This part of the proposal worries me. Side-effects in Package.swift files are typically to be avoided, as the general premise of the format is that they’re supposed to be declarative, not dynamic.

I suppose it’s not out of step with the rest of the proposal, but I’m nervous about further encouraging dynamism in Package.swift files.

9 Likes
3

I like this pitch a lot. It is a core idea that I planned to incorporate into an operating system orientated around Swift (SwiftOS, a proposal).

However, I don't think the pitch's implementation is the one we want. I was more aligned with how products already work with the SwiftPM (.static and .dynamic), so developers can choose which linking strategy they want to support/use on a case-by-case basis. Some patching would be necessary (or not?) to fully incorporate the correct usage of dynamic products, which is noted in your other post, but I see it as a net-positive and an absolute necessity.

It is worth noting that I haven't experimented/tested with dynamic linking yet, so my opinion might change on whether or not the existing SwiftPM system should be used (and patched if necessary) instead of adding a new one, but I don't think it will.

1 Like
4

Hi Cory,

Thanks for the thoughtful feedback, I'd like to respond to a few of your points, elaborate a little bit on the rationale and perhaps suggest a possible improvement to the proposal that perhaps may alleviate your major concern.

I do agree, it is an expert-level feature - but it does facilitate solutions with Swift on the server side that is not possible today and can be viewed as a stepping-stone to facilitate adoption of Swift for a wider range of such enterprise-style deployments in the short- to medium time frame.

A wider adoption there will provide more incentive and resources for a solution to the general problem over time (at Ordo One, we'd support such further improvements too, but the bigger issue is a bit too large for us to take on at this point and would like to take a first step at enabling this).

ABI management is tricky, but as library evolution mode is tricky and have a significant piece of documentation on the dos and don'ts (and even the digester tool to try to help out), we don't think this is much worse - it is largely a tool intended for the same audience as would use library evolution which does require significant understanding of ABI concerns.

It'd be an interesting approach to consider as a future direction but could coexist with the existing proposal - I don't think it solves the exact same problem either, as we need to use the same toolchain on Linux to guarantee ABI stability - to quote from that blog article:

However, stable module interfaces and library evolution can be used on all platforms supported by Swift. So on non-Apple platforms, you can still use multiple versions of the same library without recompiling a client application, as long as all binaries were built with the same version of the Swift compiler.

This is something we can enforce/guarantee using ODPs, not sure how that would be solved with the manylinux approach (but it is perhaps a different discussion)

I just realised that one potential way to address this could be by leveraging SE-0450 (Package Traits).

This would give us:

  • Declarative RLP Selection – Instead of dynamically modifying Package.swift, we could define traits corresponding to different Organization-Defined Platforms (ODPs). The correct RLP could then be selected via build configuration rather than imperative logic.

  • No Side Effects in Package.swift – Traits would allow developers to specify the correct RLP at build time using:

swift build --traits ODP1

This removes reliance on environment variables while keeping SwiftPM manifests declarative. Looking at SE-0450 is fits well with one of the first motivations of the traits feature, to quote that part:

Pluggable dependencies
Some packages want to make it configurable what underlying technology is used.

Would that be a reasonable approach? (WDYT @taylorswift / @FranzBusch / @Max_Desiatov)

Overall, we are very keen on an integrated solution to solve the use case of enabling library evolution in controlled enterprise environments as a first step - I think it also could be of interest for e.g. environments like AWS (@sebsto ?)

Joakim

6 Likes
5

I'm an enthusiastic +1 for this proposal.

I work at a company that also uses Swift extensively for server-side code running on Linux. We waste an unfortunate amount of time (both wall clock and compute time) on our CI constantly re-compiling dependent libraries that haven't changed but whose build artifacts cannot be shared via the current Swift package manager.

That said, I do share basically all of Cory's concerns. They are all valid. But given how long this subject has languished (at least since 2019) I think any forward progress is good. I don't think there's anything in this proposal that stands in the way of a more generalized solution in the future.

I also like Joakim's addendum of leveraging Package Traits.

3 Likes
6

Neat usage of SE-0450! Seems like a perfect fit

2 Likes
7

i’m not sure if SwiftPM Traits would actually reduce complexity, and it might actually make workflows much harder than they need to be.

the first problem is that Traits don’t really help with defining, identifying, or selecting ODPs; they are really just “first-class” environment variables. they are a bit easier to pass from the command line, and avoid the need to import Foundation in the manifest, but beyond that, you would still need some switch/case-like logic in the manifest to compute the correct artifactbundle URL. so this wouldn’t really do anything to “lift logic” out of the Package.swift.

if you really hated the idea of computing URLs in the manifest, you could spend some effort on the packaging side to merge all those bundles into the same archive, at the same URL. but that would create a lot of work for the distributor — you would need some kind of staging area in the cloud to collect intermediate artifacts, and some kind of task that monitors the status of the various CI/CD pipelines that upload those artifacts so it can download, merge, and re-upload them when all pipelines are complete, and cleans up after itself when it’s done.

since we expect that the producer and consumer of most RLPs will be the same entity, i’m not convinced this would be a net win at all.

finally, i think we are overstating the actual amount of “dynamism” in the manifests when using environment variables. the switch statement in the example project (which has four ODPs) is just this:

and i would expect a more realistic use case to have fewer than four ODPs. i regularly encounter Package.swifts in existing code bases that have far more complex control flow in the manifests.

SwiftPM products are not involved in the consumer side of a binary dependency.

today, you declare a binary dependency as a binaryTarget in the targets: section of the manifest; it never appears as a “proper” package dependency, it does not need to have any associated Git repository or published source code, and i don’t think we would ever want it to.

i recognize that this is a bit off topic, but i just want to respond to this part.

i would argue that Swift today has already copied Python, and that Swift’s equivalent of Python Wheels are SwiftPM Packages. Python Wheels get a Good Rap for being highly portable, i think, for two reasons:

  1. they (purelibs at least) distribute textual Python source code, and compile that source to .pyc lazily on the client machine. the source code is far more stable than the bytecode, so runtime compatibility becomes a non-issue for Wheels.
  2. almost every system that someone would want to install Wheels on already has a Python interpreter and the Python Package Manager (pip) installed, and if they are not, they are readily available through apt, yum, etc. Wheels are very challenging to set up without pip.

Swift source code, like Python source code, has excellent source compatibility across platforms and runtime versions, and this is why i constantly hear success stories from people who are using Swift libraries i wrote on platforms (e.g. Android) that i never even intended for them to work on. this means if you can get SwiftPM and the Swift toolchain running on some esoteric platform, you can reasonably expect to use any Swift “purelib” regardless of whether that library officially supports that platform.

but the catch of course is that you need to run SwiftPM and compile source code on the client machine. if the client is a developer’s workstation with plentiful RAM, this is a non-issue, although it’s a bit more painful with Swift than with Python since Swift is an AoT language. but if the consumer is, for example, a server instance that is close to the “embedded” side of the spectrum, Wheel-like source distribution just isn’t a realistic deployment strategy.

Python Wheels do support shipping pre-compiled binary code, but this is heavily dependent on static linking to achieve portability. that’s not great for resource-constrained systems like servers. now, this doesn’t matter for simple one RLP + one Big App architectures, but it could be limiting if you wanted to do anything fancier. Wheels in practice mitigate a lot of the static linking overhead by defining that “manylinux” ODP that has a lot of the common dependencies like Glibc. but this brings us back to the problem of privileging one or a handful of ODPs at the expense of other configurations people might be using.

binaries in Python Wheels also lean heavily on C and C++’s ABI stability to mitigate the dimensional curses of ABI compatibility, and we just don’t have that in Swift.

ultimately, RLPs are an advanced tool intended for fleets that are intentionally trying to avoid the overhead that comes with a generic Wheel-like distribution model, or the design constraints that come with the more-efficient manylinux ODP that many Python packages rely on.

3 Likes
8

Upon extensive testing I am very +1 on supporting dynamic linking on Linux via binary targets using artifact bundles. This implementation is exactly what I am looking for.

I would definitely use them for public distribution. I assume in order for a third-party to correctly use one in their project the binary target would require the -enable-library-evolution flag for compilation (at least in my testing it should). Would this pitch support that? I don't see why this feature should only apply to a select few.

9

you can do Whatever You Want, but i think the difficulty you would encounter is communicating to your users how to best consume an application that uses RLPs. can you give an example of a Linux-targeting application that could be improved by using RLPs that are vended to the public at large?

10

After having slept on it, I do agree with you - it was just an idea about Traits - I think it is significantly more convenient to specify the environment in this context (especially when using interactive shells), and it is a widely spread practice for similar use cases like picking local dependencies over remote ones in e.g. SwiftNIO, SwiftPM (1, 2, 3, 4, 5, 6), swift-build, Foundation and many more. As you say, a typical deployment will just have 2-3 RLPs.

(in fact, as a reflection it would be nice to be able to specify arbitrary command line arguments as part of the environment when using interactive shells, but that is a different topic...).

1 Like
11

Moving your question from the swift-build thread here @dschaefer2

If there are practical, impactful changes that can be implemented within a reasonable timeframe, we can discuss them. However, this pitch is intentionally focused on a specific use case—deployment with shared libraries and evolution enabled on Linux in controlled environments. As @stemmetje noted, this has been a persistent pain point for at least six years, often derailed by debates over a "perfect solution."

Talking about pros and cons, I can only give my take on it trying to summarize;

Pros

  • Enables Swift server-side Linux deployments with shared libraries and evolution support for controlled environments (both enterprise/service based ones, as well as ISV-style ones with shipment of SDK:s)
  • Provides practical experience and a possible stepping stone for future possibly more generic solution
  • Removes a significant friction point for also internal workflows (e.g. CI)
  • Sponsored implementation from the community, minimising core SwiftPM engineering time
  • May increase incentive to follow up with a more generic solution if these deployments get traction
  • Should not conflict with potential future more generic solutions
  • Removes need to ship custom toolchains to customers with XCFramework support enabled on Linux (what we at Ordo One need to do today for full disclosure... really would rather not.)

Cons

  • Support of the feature over time (including possible need to do extra work in the context of swift-build)
  • Can be mystifying to non-expert users (although that would be true for most expert-level features...)
  • May reduce incentive to instead do a more generic solution (in the eye of the beholder...)
5 Likes
12

A website like the Arch Linux User Repository (AUR; or CocoaPods) but for dynamic Swift dependencies (artifact bundles). Mainly for developers, but it would enable rolling updates. It doesn’t even need to be a website if developers publish their artifact bundles.

Because they would be dynamically linked, it would greatly improve not only the binary size of all Swift programs using them, but the productivity of developing and maintaining of said programs.

13

It's great to see this initiative gaining traction. We've successfully deployed RLP for Linux alongside our custom toolchain across approximately 20 repositories, distributing four artifact bundle libraries. With multiple daily updates, this approach eliminates the need to rebuild and redistribute all repositories whenever new API interface is introduced or internals of an API are changed.

From a development and release pipeline perspective, this results in significant time savings. The process is streamlined: merge a PR to release the API, trigger an automatic build, deploy the updated libraries, and restart the service. The entire cycle takes under 10 minutes. Without RLP, a release would require rebuilding all dependent repositories, each taking 3-4+ minutes, multiplied across our growing set of ~20 repositories:

api-release
api-release1066Ă—232 39.4 KB

api-deploy
api-deploy668Ă—240 35.9 KB

It would great to preserve the behaviour of SPM when used with repositories that use dynamic linking and XCFrameworks. Specifically, the artifact bundle contents should be copied by SPM into .build/<platform-triple>/ to ensure tests work correctly and to execute the binary via swift run. Currently setting LD_LIBRARY_PATH to .build/artifacts/<package>/<binary-target-name>/

In our case, rpath linking is too restrictive since we version our API and can't predict which minor or patch version will be used at build time. Instead, we control this dynamically by injecting the appropriate LD_LIBRARY_PATH before service startup.

4 Likes
14

i have updated the implementation to copy the RLPs into the build directory.

3 Likes
15

Great, thank you!

16

Since no one else seems to have opened the bike shedding part of the discussion, is there any particular reason for choosing the name "plugins" here? I find it quite confusing especially considering there's also an established meaning of plugins in the context of SwiftPM that has no overlap with this.

3 Likes
17

Colors are important of course :slight_smile:

Maybe something along the lines of Distribution Specific Dynamic Libraries would be more on the nose and less confusing?

( Organization-Specific Distribution ?)

3 Likes
18

From the perspective of someone who doesn't work on standard Linux distributions on a day-to-day basis, the name you suggested is a great deal clearer about what's being proposed. To me, aside from the SwiftPM connotations, the term "plugin" echoes a dylib that's specifically being loaded at runtime to add functionality to some other program.

7 Likes
19

i think Distribution-specific by itself is simply incorrect — RLPs are not generally compatible across even a single Linux distribution (a trivial counterexample is the exact same OS with different Swift runtime versions), so Organization-Specific Distribution-Specific Dynamic Library (OSDSDL?) is the bare minimum accurate naming. that’s obviously quite a mouthful though.

i thought about Replaceable Library Dependencies (RLDs) as an alternative, but Dependency is really a word we associate with package managers (like SwiftPM), and the emphatic goal of RLPs is to not have to install or run a package manager on the deployment target.

20

That’s fair enough, how about a slightly more generic term then which also takes that into account:

Runtime-Specific Shared Libraries

or

Runtime-Constrained Shared Libraries

or perhaps even

Environment-Specific Shared Libraries

?

I think the key with naming is to lead the mental model right - either of these do that reasonably I think (Runtime-Constrained is perhaps a bit negative though, probably prefer one of the other two, although Environment-Specific Shared Libraries is probably the most correct one).

2 Likes