[Pitch] Allowing Multiple Major Versions of a Package

Evolution Pitches
1

This pitch is based on some investigative and prototype work that I've developed over the last few months. The goal with this feature is to extend the versatility of SwiftPM as a sophisticated package manager tool that grants users the flexibility to migrate alongside major version bumps to their dependencies without the strain of being limited to a single version per package. I've brought this topic up for discussion some months back (Multiple Major Versions of a Package in SwiftPM: Opening the Discussion), and with this post I'm looking to formalize the steps needed to successfully implement this -- with your feedback, of course :)

Introduction

Since its inception, SwiftPM has resolved exactly one version of each uniquely identified package in a dependency graph. In graphs with few dependencies, this rarely causes problems. As graphs grow, however, it becomes common for a package to be reached through multiple paths (better known as a diamond dependency). When those paths require different, non-overlapping major versions of the same package, resolution fails. This pitch proposes allowing multiple major versions of the same package to coexist in a single resolved graph, so that projects with such conflicts can build and run successfully.

Motivation

Conflicting major version requirements on the same package can happen well beyond what the root package controls, in transitive dependencies the user may not even be aware of.

Today's workarounds all shift the burden onto developers: clients restrict dependency versions in their own manifests and give up on newer releases; package authors avoid major version bumps, which holds back API evolution for everyone downstream; and others fork and rename a repository to represent a new major version, creating a separate package identity that must be maintained by hand.

Rather than requiring developer intervention around a tooling limitation, SwiftPM should offer a solution. When different parts of a graph legitimately need different major versions of the same package, SwiftPM should resolve and build them together instead of failing.

Example

For a concrete view into this problem, consider the following example:

As a user, I have a root package (say, FunPackage) that I develop for others seeking to use fun things. In this package, I need some networking capabilities that I can utilize via a dependency on FunNetworking:

let package = Package( 
	name: "FunPackage",
	// ...
	dependencies: [ 
		.package(url: "https://foo.bar/fun-org/fun-networking", from: "1.0.0")
	]
)

With this simple package graph, I can easily declare an import against any modules available in FunNetworking in my own source.

Consider now that I must extend the capabilities of my FunPackage so it reports usage events, and I'd like to make use of an existing package, FunAnalytics, to do that. FunAnalytics is maintained by a different organization. Its 3.0.0 release moved to the new async/await-based FunNetworking 2.0 API, so its manifest looks like this:

// FunAnalytics, version 3.0.0
let package = Package(
    name: "FunAnalytics",
    // ...
    products: [
        .library(name: "FunAnalytics", targets: ["FunAnalytics"])
    ],
    dependencies: [
        .package(url: "https://foo.bar/fun-org/fun-networking", from: "2.0.0")
    ],
    targets: [
        .target(
            name: "FunAnalytics",
            dependencies: [
                .product(name: "FunNetworking", package: "fun-networking")
            ]
        )
    ]
)

And now, my FunPackage adds the dependency to FunAnalytics:

let package = Package(
    name: "FunPackage",
    // ...
    dependencies: [
        .package(url: "https://foo.bar/fun-org/fun-networking", from: "1.0.0"),
        .package(url: "https://foo.bar/amazing-org/fun-analytics", from: "3.0.0")
    ]
)

The dependency graph for this would look like the following:

FunPackage
├── fun-networking   1.0.0..<2.0.0   (direct)
└── fun-analytics    3.0.0..<4.0.0   (direct)
    └── fun-networking   2.0.0..<3.0.0   (transitive)

Notice that there are two instances of fun-networking, both requiring not-overlapping versions. Today, SwiftPM would report an error:

error: Dependencies could not be resolved because `fun-package` depends on 'fun-networking' 1.0.0..<2.0.0 and `fun-analytics` depends on 'fun-networking' 2.0.0..<3.0.0.

While there is technically a workaround by widening the version constraint of FunPackage's dependency on FunNetworking,

	.package(url: "https://foo.bar/fun-org/fun-networking", "1.0.0"..<"3.0.0")

every workaround has the same consequence. SwiftPM resolves exactly one version of fun-networking for the entire package graph, so any fix means changing which version of that dependency everything builds against.

For example, widening the requirement to "1.0.0"..<"3.0.0" like the above resolves FunNetworking to 2.x, which could mean that any reliance on FunNetworking's 1.x APIs used in FunPackage will need to be updated all at once, and pinning FunAnalytics to an older major version forces the whole graph back onto FunNetworking @ 1.x, which surrenders the usage of any newer features from 2.x.

This conflict isn't limited to packages the user declares themselves. Notably, it can show up anywhere in the graph, including far from the root where it is much harder to diagnose and fix. Using the same example, consider the case where its dependency tree instead looks like the following:

FunPackage
├── fun-analytics   from 3.0.0
│   └── fun-transport   from 1.0.0
│       └── fun-networking   1.0.0..<2.0.0
└── fun-ui   from 2.0.0
    └── fun-image-loader   from 4.0.0
        └── fun-networking   2.0.0..<3.0.0

There is little a user can do through its root FunPackage to avoid a conflict in SwiftPM's ability to resolve this graph beyond constraining their own dependencies rigorously or requesting that package authors create a fork for every new major version.

Proposed solution

When a package graph requires disjoint major versions of the same package, SwiftPM would resolve one version per major range instead of failing. In the example above, FunPackage would resolve fun-networking 1.x for its own use and 2.x for fun-analytics, and should build both in parallel.

A prototype of this work can be found here. This prototype is not a full and complete implementation of all the stages of the solution proposed below (there is no implementation of PURLs, and the PubGrub modifications are incomplete), however it does act as a proof of concept that SwiftPM is capable of building a package graph with multiple major versions of one package, and can link a binary that runs successfully.

Similarly, a project board was created to track the issues related to the investigation and implementation of the multiple majors feature: [SwiftPM] Multiple Major Versions · GitHub

The implementation can be broken into five stages, each described below:

  1. Identify packages with a PURL
    • Introduce PURLs as a way to identify packages, carrying provenance and version data that will allow SwiftPM to easily distinguish between major versions. A PackageIdentity can still be derived by a PURL.
  2. Resolve using major version buckets
    • Augment the PubGrub algorithm so that disjoint version constraints on the same package no longer produce an error.
    • SwiftPM would track disjoint major version ranges and would represent these choices as higher-level bucket nodes.
  3. Rewire the package graph
    • Where the graph would previously point every edge at a single resolved version of a package, point each edge at the major version bucket that satisfies that edge's constraint.
  4. Alias the modules
    • Synthesize module aliases that incorporate the package identity and major version, so that same-named modules from different buckets don't collide at build time.
  5. Introduce an opt-in flag
    • Developers will have to explicitly opt-in to this behaviour; a flag will be introduced to run alongside any swift commands (swift package resolve, swift build, swift run, etc.)

To confirm this works end to end, the resulting object files should contain distinct, mangled symbols for each major version, and any final linked binaries should run successfully.

Note that the initial implementation will support Swift modules only. When a conflict involves a package containing C modules, SwiftPM keeps today's behaviour and reports the resolution error.

Detailed design

Replace PackageIdentity with PURL

The resolver and package graph need a stable way to name each major version bucket of a package, as well as the edge packages described above. PackageIdentity can't express this today. It identifies a package only by name, so fun-networking at 1.x and fun-networking at 2.x share one identity, which is exactly why the graph can hold only one of them.

A PURL (Package URL) is an identifier (and has already been adopted in SwiftPM as a part of SE-0509) consisting of a scheme (which is always pkg), a type (a registered identity in the specification that refers to a particular language or build system), an optional namespace, a name, an optional version, and optional qualifiers. For example, the different versions of FunNetworking used in the example above may be identified (using the String representation of PURL) as:

// PURL scheme: pkg:<type>/<namespace>/<name>@<version>?<qualifiers>
pkg:swift/fun-org/fun-networking@1.x
pkg:swift/fun-org/fun-networking@2.x

PURLs also address a related SwiftPM weakness: today, package Foo from GitHub at 1.2.3 and package Foo from a custom registry at 1.2.3 share a PackageIdentity of Foo, so SwiftPM relies on heuristics (such as DependencyMapper), to decide which one is being referred to. A PURL carries the origin as part of the identifier, so the two would become easily distinguishable.

// Example PURLs for a package, fun-stuff, in two separate locations.

// For a package awesome.fun-stuff @ v1.x located on registry foo.bar.com
"pkg:swift/awesome/fun-stuff@1.x?registry_url=https://foo.bar.com"

// For a package with the same name, located on GitHub:
"pkg:swift/awesome/fun-stuff@1.x?vcs_url=https://github.com"

A PURL, conceptually, builds on top of what is already available in a PackageIdentity, since the name field (and, where applicable, namespace) is effectively the same as today's PackageIdentity. Any additional information acts as metadata, further differentiating two similarly-declared packages, and can be extended to return richer types compatible with SwiftPM's internal models:

extension PURL { 
	/// Where the package originates (source control, registry, etc.)
	public var kind: PackageKind { 
		// ...
	}
	
	/// The base-name identity of the package.
	public var identity: PackageIdentity { 
		// ...
	}
	
	/// The version of the package.
	public var version: Version { 
		// ...
	}
	
	// ... etc.
}

With the introduction of PURLs as the standard in which SwiftPM can identify packages, it becomes rightly more specific to refer to any given package anywhere in SwiftPM.

Augmenting PubGrub to bucket major versions

To support multiple major versions, the resolver will treat each major version of a package as its own bucket, represented by a single constraint. Ranges of major versions will be bucketed by the dependency edge itself, with each version therein representing a major-version bucket.

A bucket is identified by its package (what would canonically be the PackageIdentity) and a major version, and would cover that major version's range. A PURL-bucket would represent a major version range for that dependency:

// The PURL String representation, when printed:
"pkg:swift/fun-networking@1"
// Applicable versions for this would be 1.0.0..<2.0.0

// A possible extension to PURL, for use in PubGrub:
extension PURL { 
	/// The major version range that this PURL bucket represents.
	public var majorVersionBucket: Int?
	
	/// The string representation of the PURL bucket.
	public var bucketDescription: String? { 
		guard let major = self.majorVersionBucket else { return nil }
		return "pkg:swift/\(self.name)@\(major)"
	}
	
	/// Detects which available versions for the base package is
	/// eligible for this bucket.
	public func availableVersions(_ versions: [Version]) -> [Version] { 
		if let major = self.majorVersionBucket { 
			return versions.filter({ $0.major == major })
		}
		
		return versions
	}
}

For FunAnalytics, say its dependency on FunNetworking has expanded its version range constraint to 2.0.0..<4.0.0. There would then be two possible major-version buckets that represent this range, and therefore this new dependency edge constraint would span several majors. Here, this would correspond to two buckets:

// ... for some `container` representing the package `fun-networking`

let allVersionsFunNetworking = container.versionsAscending()
print(allVersionsFunNetworking)
// E.g.: 
// 1.0.0, 1.0.1, 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.2.1, 3.0.0, 3.1.0, 3.1.1

// bucket: FunNetworking@v2
let funNetworkingV2 = PURL("pkg:swift/fun-networking@2")
let v2 = funNetworkingV2.availableVersions(allVersionsFunNetworking)
print(v2)
// 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.2.1

// bucket: FunNetworking@v3
let funNetworkingV3 = PURL("pkg:swift/fun-networking@3")
let v3 = funNetworkingV3.availableVersions(allVersionsFunNetworking)
print(v3)
// 3.0.0, 3.1.0, 3.1.1

FunAnalytics is requesting any one version of FunNetworking within the 2.0.0..<4.0.0 range. If SwiftPM were to add one bucket per major version as separate constraints, it would require resolving to both a 2.x version and a 3.x version, which is incorrect. Instead, each dependency edge becomes an intermediate representation whose synthesized "dependencies" are the buckets representing a major version range.

// synthesized package: FunAnalytics->FunNetworking
// available versions: 2.0.0, 3.0.0
//
// version: 2.0.0
// dependencies:
//   - FunNetworking@v2
//
// version: 3.0.0
// dependencies:
//   - FunNetworking@v3

Each version of this intermediate edge corresponds directly to its "dependencies", which would be the major version bucket. Since this would imply the addition of new incompatibilities that PubGrub would need to derive and inject into its algorithm, it stands to reason that there must be some extra maintenance within the state to fully capture and track each of these synthesized decision nodes.

// ...
public struct PubGrubDependencyResolver { 
	// ...
	// Store some state that remembers the decisions made
	// for dependency edges requesting a package that will
	// resolve to many major versions. Possibly:
	public var multipleMajorDependencyEdges: [PURL: PURL]
}

Inject major version-specific nodes into the package graph

Once resolution is complete, each dependency edge has selected one major version bucket, and each bucket resolves to a single version. The package graph is then built from this result. Where today a single node represents the one version a package resolved to, the graph would instead contain one node per bucket that was selected by at least one edge.

In the running example, the graph would change as follows:

Today (fails when the ranges are disjoint):

  FunPackage ──► fun-networking
  FunPackage ──► fun-analytics ──► fun-networking 

With major version nodes:

  FunPackage ──► fun-networking@1 (resolved to some v 1.x)
  FunPackage ──► fun-analytics ──► fun-networking@2 (resolved to some v 2.x)

Where each major version nodes PURLs may look like, respectively:

pkg:swift/fun-org/fun-networking@1.1.0
pkg:swift/fun-org/fun-networking@2.4.3

Each dependency edge points at the major version node it selected, identified by a PURL containing the exact version it resolved to. Edges that select the same major version share a node (much like today's SwiftPM sharing the node between dependents who require the same dependency), so if another package also depends on fun-networking@2, it points at the same node, and every dependent of that major version sees the same resolved 2.x.

Leverage module aliasing to resolve same-name collisions

The resulting package graph once resolution is complete will introduce two or more of the same package at different major versions, which would typically contain modules of the same name. Today, two packages that have modules with the same name would cause SwiftPM to fail and would suggest to use module aliasing to resolve this, so that the mangled symbols (which use the module name) aren't identical.

Reasonably, module aliasing is therefore needed at this stage. After the graph is built, SwiftPM would need to add a phase that identifies packages that have multiple major versions and should generate a unique alias for each of their modules, derived from the package identity and major version:

fun-networking@1.x:  FunNetworkingModule  ->  FunNetworkingModule_v1
fun-networking@2.x:  FunNetworkingModule  ->  FunNetworkingModule_v2

The ModuleAliasTracker will then need a supplementary method that serves to gather and and create the aliases for all of the major versions of one package.


func addMultipleMajorAliases(_ package: PURL, modules: [Module], products: [Product]) { 
	// ...
	for product in products { 
		// ...
		for module in modules { 
			derivedAliases[module.name] = "\(module.name)_v\(PURL.majorVersion)"
		}
		// ...
		// This would populate the tracker with new synthesized aliases,
		// and should in theory be handled as any other alias.
		try addAliases(
			derivedAliases,
			productID: product.identity,
			productName: product.name,
			originPackage: package,
			consumingPackage: package 
		)
	}
}

Today module aliasing is driven by the user, who declares aliases in their manifest. This phase would instead be synthesized by SwiftPM during resolved package graph construction and would run alongside the existing alias handling. An open question is how SwiftPM should reconcile user-declared aliases with the major version aliasing.

Note on C module compatibility

SwiftPM passes module aliases to the Swift compiler through the -module-alias flag. This mechanism applies to Swift modules. C modules are compiled with clang, and requires additional investigation in how those sources are built.

Opt-in flag

Allowing SwiftPM to resolve multiple major versions of a package will be strictly opt-in. The entrypoint to users will be through the use of a flag (for example, --allow-multiple-majors, although the final name is to be determined) on commands that resolve or build a package graph, such as swift package resolve, swift build, and swift run. Without specifying this flag, resolution would behave exactly as it does today.

If users ever encounter a scenario where multiple major versions of a package dependency causes a resolution failure, the error message would mention this flag for discoverability:

note: this conflict involves disjoint major versions of 'fun-networking'.
Pass '--allow-multiple-majors' to resolve each major version separately.

Security

Older major versions may be known to carry vulnerabilities, and users may not be aware if an older major is resolved transitively. Notably, the attack surface grows with each additional copy of a package. As a security measure, SBOMs (which uses PURLs) are used to audit these potential vulnerabilities -- therefore, it would be possible to easily identify such a package since the PURL will act as the identifier. Further discussion would be appreciated here.

Impact on Existing Packages

This feature will be explicitly opt-in, not automatic. Thus, this shouldn't change how existing packages are resolved and built without explicit user intervention.

The Package.resolved will now need to accommodate multiple entries per package. This should also not break previous toolchain versions, however there is an open question as to how older toolchains should handle this new format. Other commands, such as dump-package and show-dependencies are affected by this change and similarly must accommodate multiple entries per package.

Only packages flagged for multiple major versions will introduce new package graph nodes; however, if there are further conflicts in their dependencies these subtrees could also duplicate.

The resulting binaries could inflate in size. There could also be longer download times as the amount of packages to checkout could, in the worst case, double.

Alternatives considered

Users can opt to module alias conflicting packages themselves, however this task is incredibly manual and tedious and is not guaranteed to work, particularly for dependencies deeper in the graph.

Rather than relying on PURLs, the initial investigation began by simply modifying a PackageIdentity in-place during resolution to include the major version appended, while maintaining a mapping in the Pub Grub state that tracks each of the packages involved in multiple majors. This is effectively the behaviour we will be getting by introducing PURLs as a the mechanism for identifying packages.

Future directions

  • Per-package opt-in/out
  • C module support
    TBD
3 Likes