Last June, Apple and GitHub announced that the GitHub Package Registry will support Swift packages . A few months later, Bryan Clark started a thread to gather ideas about a standard package registry API that could be implemented by anyone, not only GitHub.
For much of this year, I’ve had the privilege of working with Bryan, as well as Whitney Imura and many other great folks at GitHub, to define a draft specification for a Swift package registry service. I posted our initial pitch on June 4th, which we then developed into a formal proposal that was reviewed from December 8 – 17.
As announced in this thread the Swift core team reviewed our updated proposal and returned it for further revision, seeking stronger consensus around the use of URLs as package identifiers.
I'd like to use this thread to discuss the merits and tradeoffs of this approach as well as any alternatives that we might consider.
Proposed Solution: URLs as package identifiers
We believe that using URLs as package identifiers is intuitive and familiar for developers, and will best solve the immediate and future needs of this project.
Here is some relevant discussion from our updated proposal:
Changes to Swift Package Manager
Currently, the identity of a package is computed from the last path component of its effective URL (which can be changed with dependency mirroring and forking). However, this approach can lead to conflation of distinct packages with similar names as well as duplication of the same package under different names.
We propose to instead use a normalized form of a package dependency's declared location as that package's canonical identity. Not only would this resolve the aforementioned naming ambiguities, but it would allow for package registries to be adopted by package consumers with minimal changes to their code.
For the purposes of package resolution, package identities are case-insensitive (for example,
MONA) and normalization-insensitive (for example,
ñ). In addition, the following operations are performed to mitigate insignificant variations in how a dependency may be declared in a package manifest:
The list of URI transformations
- Removing the scheme component, if present:
https://github.com/mona/LinkedList → github.com/mona/LinkedList
- Removing the userinfo component (preceded by
@), if present:
email@example.com/mona/LinkedList → github.com/mona/LinkedList
- Removing the port subcomponent, if present:
github.com:443/mona/LinkedList → github.com/mona/LinkedList
- Replacing the colon (
:) preceding the path component in "
firstname.lastname@example.org:mona/LinkedList.git → github.com/mona/LinkedList
- Expanding the tilde (
~) to the provided user, if applicable:
ssh://email@example.com/~/LinkedList.git → github.com/~mona/LinkedList
- Removing percent-encoding from the path component, if applicable:
github.com/mona/%F0%9F%94%97List → github.com/mona/🔗List
- Removing the
.gitfile extension from the path component, if present:
github.com/mona/LinkedList.git → github.com/mona/LinkedList
- Removing the trailing slash (
/) in the path component, if present:
github.com/mona/LinkedList/ → github.com/mona/LinkedList
- Removing the fragment component (preceded by
#), if present:
github.com/mona/LinkedList#installation → github.com/mona/LinkedList
- Removing the query component (preceded by
?), if present.
github.com/mona/LinkedList?utm_source=forums.swift.org → github.com/mona/LinkedList
- Adding a leading slash (
file://URLs and absolute file paths:
file:///Users/mona/LinkedList → /Users/mona/LinkedList
Module name collision resolution
Swift Package Manager cannot build a project
if any of the following are true:
- Two or more packages in the project are located by URLs with the same (case-insensitive) last path component (for example,
- Two or more packages in the project declare the same name in their package manifest (for example,
let package = Package(name: "LinkedList"))
- Two or more modules provided by packages in the project have the same name (
let package = Package(products: [.library(name: "LinkedList")]))
This proposal directly addresses point #1 and lays the foundation for resolving #2 and #3.
Consider the following package manifest, which Swift Package Manager currently fails to resolve for all three of the reasons listed above (assume both external dependencies are named "LinkedList" and contain a library product named "LinkedList").
// swift-tools-version:5.3 import PackageDescription let package = Package(name: "Greeter", dependencies: [ .package(name: "LinkedList", url: "https://github.com/mona/LinkedList", from: "1.1.0"), .package(name: "LinkedList", url: "https://github.com/OctoCorp/linkedlist", from: "0.1.0") ], targets: [ .target( name: "Greeter", dependencies: [ .product(name: "LinkedList", package: "LinkedList"), // github.com/mona/LinkedList .product(name: "LinkedList", package: "LinkedList") // github.com/OctoCorp/linkedlist ]) ])
By adopting URLs as unique package identifiers,the existing
nameparameter in dependency
.packagedeclarationsbecomes redundant,and could instead be used to label nodes in the dependency graph.
- .package(name: "LinkedList", + .package(name: "LinkedList-Mona", url: "https://github.com/mona/LinkedList", from: "1.1.0"), - .package(name: "LinkedList", + .package(name: "LinkedList-OctoCorp", url: "https://github.com/OctoCorp/linkedlist", from: "0.1.0")
These package names (or, alternatively, the package URI) could then be used in
.productdeclarations to reference a particular package's module.
To resolve name collisions in package modules, a dictionary literal might be used to assign aliases for that module within the declared target.
dependencies: [ - .product(name: "LinkedList", - package: "LinkedList"), - .product(name: "LinkedList", - package: "LinkedList") + "MonaLinkedList": .product(name: "LinkedList", + package: "LinkedList-Mona"), + "OctoCorpLinkedList": .product(name: "LinkedList", + package: "LinkedList-OctoCorp")
Following from this example, source files in the
Greetertarget would be able to import both modules according to their assigned aliases.
import MonaLinkedList import OctoCorpLinkedList
Importing external modules without a package manifest
Developers enjoy using Swift as a scripting language, but wish there were an easier way to import external dependencies in a standalone file, without the overhead of the package manifest and directory structure. Various solutions have been explored through community projects like swift-sh, Beak, and Marathon, and discussions on the Swift forums.
For example, running the command
swift path/to/file.swiftwith the following file would automatically resolve and download the
LinkedListdependency before building and running the script executable.
// Proposed syntax by Rahul Malik, Ankit Aggarwal, and David Hart // See: https://forums.swift.org/t/swiftpm-support-for-swift-scripts/33126 @package(url: "https://github.com/mona/LinkedList", from: "1.1.0") import LinkedList // Variation with hypothetical module aliasing @package(url: "https://github.com/OctoCorp/linkedlist", from: "0.1.0") import LinkedList as OctoCorpLinkedList print("Hello, world!")
While the functionality described by this proposal isn't required for such a feature, a registry provides the same benefits of speed, efficiency, and security as it would in the context of a package.
Intermediate registry proxies
By default, the identity of the package is the same as its location. Whether a package is declared with a URL of
firstname.lastname@example.org:mona/linkedlist.git, Swift Package Manager will — unless configured otherwise — attempt to fetch that dependency by consulting
github.com, which may respond with a Git repository or a source archive (or perhaps
404 Not Found).
A user can currently specify an alternate location for a package by setting a [dependency mirror][SE-0219] for that package's URL.
$ swift package config set-mirror \ --original-url https://github.com/mona/linkedlist \ --mirror-url https://github.com/octocorp/swiftlinkedlist
Dependency mirroring allows for package dependencies to be rerouted on an individual basis. However, this approach doesn't scale well for large numbers of dependencies.
Swift Package Manager could implement a complementary feature that allows users to specify one or more registry proxy URLs that would be consulted (in order) when resolving dependencies through the package registry interface.
For example, a build server that doesn't allow external network connections may specify an internal registry URL to manage all package dependency requests.
$ swift package config set-registry-proxy https://internal.example.com/
When one or more proxy URLs are configured in this way, resolving a package dependency with the URL
https://github.com/mona/linkedlistresults in a
A registry proxy decouples package identity from package location entirely, which could unlock a variety of compelling use cases:
- Geographic colocation : Developers working under adverse networking conditions can host a mirror of official package sources on a nearby network.
- Policy enforcement : A corporate network can enforce quality or licensing standards, so that only approved packages are available.
- Auditing : A registry may analyze or meter access to packages for the purposes of ranking popularity or charging licensing fees.
Finally, I'd like to call attention to two points from the registry specification:
- The registry service interface is simple — all of the endpoints (with the exception of the publish endpoint, which has since been removed) can be satisfied by a static file server. (See § 4. Endpoints )
- Maintainers can host a package at their own domain and delegate to an external registrar with a custom
Linkheader (see § 4.1.1. Content negotiation )
Alternative suggestion: reverse-DNS identifiers
The feedback shared by the Swift core team included the following example of an alternative reverse-DNS package identifier:
// Proposed by SE-0292 .package(url: "https://github.com/apple/swift-nio", from: "2.0.0") // --> git, upgradable to registry .package(path: "/code/swift-nio", from: "2.0.0") // --> local, maybe override (based on name) // Alternative reverse-DNS proposal .package(url: "https://github.com/apple/swift-nio", from: "2.0.0") // --> backwards compatibility, always git .package(path: "/code/swift-nio", from: "2.0.0") // --> backwards compatibility, local, maybe override (based on name) .package(identifier: "com.apple.swift-nio", from: "2.0.0") // --> always registry .package(identifier: "com.apple.swift-nio", url: "https://github.com/apple/swift-nio", from: "2.0.0") // --> always git .package(identifier: "com.apple.swift-nio", path: "/code/swift-nio", from: "2.0.0") // --> local, means override
In our proposal, we considered this and other schemes for identifying packages, but ultimately rejected them in favor of URLs.
Package.swift manifests have always declared external dependencies using URLs, so this step represents a formalization of long-standing tradition rather than a departure from existing behavior.
Truth be told, I'm unable to articulate a reasonable case for preferring reverse-DNS identifiers over URLs in Swift Package Manager.
The feedback I've received so far stated a preference for this naming scheme, but didn't include much information about how this would work or reasons why it would be a better solution. Rather than try to guess at the reasoning or try (and fail) to make the case myself, I think it'd help to start by articulating some of the reasons why we ultimately decided against them.
Technical arguments against reverse-DNS identifiers
Identifying a package by a URL is simple and intuitive. By default, Swift Package Manager tries to fetch the package at the provided URL, either by downloading through the registry interface or cloning its Git repository. If a mirror is configured, Swift Package Manager will go to the mirrored URL instead. The proposal also suggests a future enhancement, where all package requests are immediately routed through one or more registry endpoints.
Reverse-DNS identifiers, on the other hand, lack fundamental addressability. You can't, for example, navigate to the package identifier
com.github.mona.LinkedList from a browser or
curl. Instead, you need to first configure a registry. The same is true for Swift Package Manager itself: either the user needs to configure a registry, or Swift Package Manager needs to hardcode a predefined list. (If so, who makes the cut? And if it's a ranked list, how is that order determined?) Finally, relying on registries for resolving package identifiers creates a point of failure; if a registry goes down, there's no way to fallback to Git like there is when you use URLs.
Practical arguments against reverse-DNS package identifiers
URLs have the benefit of gradual adoption. The proposed package registry can be used existing packages without any code changes. (In fact, you can already try it out for yourself today!)
By contrast, a reverse-DNS identifier would require all packages in a project's dependency graph to update to a new syntax and new
swift-tools-version before seeing the benefit of the registry interface. Such a move would be disruptive, requiring the entire ecosystem to opt-in to this new system. Network effects being what they are, it would be difficult to convince package maintainers to migrate to a system without first being able to demonstrate the utility of such a move (which requires others to migrate first). So we should expect that this transition would take some time, on the scale of years. If the Swift 2 → 3 migration or any of the changes to
PackageDescription API changes are any indication, we should expect this transition to be painful and confusing, too.
Also, we shouldn't take as a given that the project would survive the transition — there are plenty of examples of languages and libraries losing developer market share after updating to a new major API version, and never getting it back. If the upgrade path is too painful, users may start looking elsewhere, whether that's CocoaPods or some other solution (perhaps even a fork of SPM).
Logistical arguments against reverse-DNS package identifiers
Adopting a reverse-DNS style identifier such as the one suggested here would fundamentally change the package ecosystem from a federated, decentralized model into one with a central naming authority and governing body. Such an organization would need to answer the following questions:
- How are unique identifiers assigned to packages?
- What's the application process? How do you determine who gets to register a namespace?
- What is the process for renaming packages or transferring ownership?
- Who is responsible for maintaining and funding the infrastructure for this organization?
- What packages can be registered? Are there any prohibitions on names or functionality?
- How are disputes between maintainers adjudicated? How about among registries?
- How would you respond to a DMCA takedown request or other government order? How about a cease and desist order or other private civil actions?
We're encouraged by the positive response from the Swift core team and the community at large, and look forward to addressing the outstanding concerns about package identity. I'm genuinely interested in identifying and building consensus around the best solution for managing package identity now and in the future.