Swift Package Registry Service

Hello!

Last June, Apple and GitHub announced that the GitHub Package Registry will support Swift packages. A few months ago, Bryan Clark started a thread to gather ideas about a standard package registry API that could be implemented by anyone, not only GitHub.

Over the past few weeks, I’ve had the privilege to work with Bryan, as well as Whitney Imura and other great folks at GitHub, to define a draft specification for a Swift package registry service. I’m thrilled to share it with you all today.

Before diving into the proposal itself, I wanted to make two points at the outset, to help frame our discussion:

  • This proposal defines an open standard for Swift package registry services. GitHub will implement these APIs as part of its GitHub Packages offering, and anyone else will be able to create their own registry as well by implementing these endpoints.

  • This proposal defines the package registry API only. Swift Package Manager will need to add support for package registries as a client of this specification. We intend to determine that integration in a separate proposal.


Introduction

Swift Package Manager downloads dependencies using Git. Our proposal defines a standard web service interface that it can use to also download dependencies from package registries.

Motivation

Today, a package dependency is specified by a URL for a Git repository. When a project is built for the first time, Swift Package Manager clones the repository for each dependency and attempts to resolve the version requirements with the available tags.

Although Git is a capable version-control system, it's not well-suited to this kind of workflow for the following reasons:

  • Speed: Cloning a Git repository is significantly slower than downloading and expanding a Zip archive with the same content.
  • Efficiency: Cloning a Git repository downloads the entire history of a project, when only a particular version is used.
  • Security: Git tags can be reassigned to another commit at any time.
  • Durability: Git repositories can be moved or deleted.

Proposed solution

Define a standard interface for package registry services that Swift Package Manager can access to download dependencies as an alternative to direct Git access.

Many language ecosystems have a package registry, including RubyGems for Ruby, PyPI for Python, npm for JavaScript, and crates.io for Rust. In fact, many Swift developers develop apps today using CocoaPods and its index of libraries.

Each of these systems follows what we describe as a "push" model. When a package owner releases a new version of their software, a client runs a command locally and pushes the results to a server.

However, the "push" model reflects a tradition of software deployment that predates modern source code management and build automation. As package maintainers can attest, this approach often involves a lot of manual effort and trial-and-error guesswork. It also lacks strong guarantees about reproducibility and software traceability.

We describe our approach as a "pull" model, which takes inspiration from current best-practices like continuous integration (CI) and continuous delivery (CD). When a package owner releases a new version of their software, their sole responsibility is to notify the package registry. The server does all the work of downloading the source code and packaging it up for use.

A "pull" model for package registries not only means less work for maintainers but better outcomes for consumers.

Note: This proposal covers only the package registry interface. Adding support for package registries in Swift Package Manager will be discussed in a separate proposal.


(Continued in next post)

81 Likes

Detailed design

Notations

The following terminology and conventions are used in this document.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

API endpoints that accept parameters in their path are expressed by Uniform Resource Identifier (URI) templates, as described in RFC 6570.

Definitions

The following terms, as used in this document, have the meanings indicated.

  • Package: A named collection of Swift source code that is organized into one or more modules according to a Package.swift manifest file.
  • Release: The state of a package after applying a particular set of changes that is uniquely identified by an assigned version number.
  • Version Number: An identifier for a package release in accordance with the Semantic Versioning Specification (SemVer).

Conventions

This document uses the following conventions in its description of client-server interactions.

Application layer protocols

A client and server MUST communicate over a secured connection using Transport Layer Security (TLS) with the https URI scheme.

The use of HTTP 1.1 in examples is non-normative. A client and server MAY communicate according to this specification using any version of the HTTP protocol.

Authentication

A server SHOULD require authentication for client requests to publish or unpublish a package release. A server MAY require authentication for client requests to access information about packages and package releases.

A server SHOULD respond with a status code of 401 (Unauthorized) if a client sends a request to an endpoint that requires authentication without providing credentials. A server MAY respond with a status code of 404 (Not Found) or 403 (Forbidden) when a client provides valid credentials but isn't authorized to access the requested resource.

A server MAY use any authentication model of its choosing. However, use of a scoped, revocable authorization framework like OAuth 2.0 is RECOMMENDED.

Error handling

A server SHOULD communicate any errors to the client using "problem details" objects, as described by RFC 7807. For example, a client sends a request to create a package release with an invalid tag parameter and receives the following response:

HTTP/1.1 404
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en

{
   "detail": "tag '2.0.0' not found"
}

Rate limiting

A server MAY limit the number of requests made by a client by responding with a status code of 429 (Too Many Requests).

HTTP/1.1 429
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en
Retry-After: 60

{
   "detail": "try again in 60 seconds"
}

A client SHOULD follow the guidance of any Retry-After header values provided in responses to prevent overwhelming a server with retry requests. It is RECOMMENDED for clients to introduce randomness in their retry logic to avoid a thundering herd effect.

API versioning

Package registry APIs are versioned.

API version numbers are designated by decimal integers. The accepted version of this proposal constitutes the initial version, 1. Subsequent revisions SHOULD be numbered sequentially (2, 3, and so on).

API version numbers SHOULD follow Semantic Versioning conventions for major releases. Non-breaking changes, such as adding new endpoints, adding new optional parameters to existing endpoints, or adding new information to existing endpoints in a backward-compatible way, SHOULD NOT require a new version. Breaking changes, such as removing or changing an existing endpoint in a backward-incompatible way, MUST correspond to a new version.

A client SHOULD set the Accept-Version header field to specify the API version of a request.

GET /mona/LinkedList/list HTTP/1.1
Accept: application/json
Accept-Version: 1

A server SHALL set the Content-Version header field with the API version number of the response.

HTTP/1.1 200 OK
Content-Type: application/json
Content-Version: 1

If a client sends a request without an Accept-Version header, a server MAY either return 400 Bad Request or process the request using an API version that it chooses, making sure to set the Content-Version header field accordingly.

If a client sends a request with an Accept-Version header that specifies an unknown or invalid API version, a server SHOULD respond with a status code of 400 (Bad Request).

HTTP/1.1 400 Bad Request
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en

{
   "detail": "invalid API version"
}

If a client sends a request with an Accept-Version header that specifies a valid but unsupported API version, a server SHOULD respond with a status code of 415 (Unsupported Media Type).

HTTP/1.1 415 Unsupported Media Type
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en

{
   "detail": "unsupported API version"
}

Namespace and package name resolution

A namespace consists of zero or more path components preceding a package name and is used to distinguish between packages with the same name.

For instance:

  • A registry that qualifies package names with the username of the person or organization responsible for the package (for example, mona/LinkedList)
  • A registry that qualifies package names with a username and domain of the source host (for example, github.com/mona/LinkedList)
  • A registry that doesn't qualify package names, creating an implicit, global namespace for the domain (for example, LinkedList)

Namespaces and package names are both case-insensitive (for example, mona ≍ MONA) and normalization-insensitive (for example, n + β—ŒΜƒ ≍ Γ±).

A server MAY normalize or disallow characters in a namespace or package name according to the security considerations of Internationalized Resource Identifiers (IRIs) discussed in RFC 3987 and Unicode Technical Report #36.

Endpoints

A server MUST respond to the following endpoints:

Link Method Path Description
[1] GET {/namespace*}/{package} List package releases
[2] GET {/namespace*}/{package}/{version} Fetch a package release
[3] PUT {/namespace*}/{package}/{version} Publish a package release
[4] DELETE {/namespace*}/{package}/{version} Unpublish a package release

List package releases

A client MAY send a GET request for a URI matching the expression {/namespace*}/{package} to retrieve a list of the available releases for a particular package. A client SHOULD set the Accept header with the application/json content type and MAY append the .json extension to the requested URI.

GET /mona/LinkedList HTTP/1.1
Accept-Version: 1
Accept: application/json

A server SHOULD respond with a JSON document containing all of the releases for the requested package.

HTTP/1.1 200 OK
Content-Type: application/json
Content-Version: 1
Link: <https://swift.pkg.github.com/mona/LinkedList/1.1.1>; rel="latest-version",
      <https://github.com/sponsors/mona>; rel="payment"

{
    "releases": {
        "1.1.1": {
            "url": "https://swift.pkg.github.com/mona/LinkedList/1.1.1"
        },
        "1.1.0": {
            "url": "https://swift.pkg.github.com/mona/LinkedList/1.1.0",
            "problem": {
                "status": 410,
                "title": "Gone",
                "detail": "this release was removed from the registry"
            }
        },
        "1.0.0": {
            "url": "https://swift.pkg.github.com/mona/LinkedList/1.0.0"
        }
    }
}

The response body SHALL contain a JSON object nested at a top-level releases key, whose keys are version numbers for releases and values are objects containing the following fields:

Key Type Description Requirement Level
url String A URI for the release REQUIRED
problem Object A problem details object. OPTIONAL

A server SHOULD communicate the unavailability of a package release using a "problem details" object. A client SHOULD consider any releases with an associated problem to be unavailable for the purposes of package resolution.

A server SHOULD list releases in order of precedence, starting with the latest version. However, a client SHOULD NOT assume any specific ordering of version numbers in the response.

A server SHOULD respond with a link to the latest published release of the package if one exists, using a Link header field with a latest-version relation. A server MAY respond with additional Link entries, such as one with a payment relation for sponsoring a package maintainer.

Fetch a package release

A client MAY send a GET request for a URI matching the expression {/namespace*}/{package}/{version} to fetch a package release.

Source archive

A client MAY send a GET request for a URI matching the expression {/namespace*}/{package}/{version} to retrieve a release's source archive. A client SHOULD set the Accept header and SHOULD append the corresponding extension to the requested URI to indicate the desired archive format. The following table lists the supported archive formats along with the requirement level for a server.

Media Type Extension Kind of Document Requirement Level
application/zip .zip ZIP archive REQUIRED
application/gzip .tar.gz GZip Compressed Tape Archive RECOMMENDED
GET /mona/LinkedList/1.1.1.zip HTTP/1.1
Accept: application/zip
Accept-Version: 1
Want-Digest: sha-256

If a release is found for the requested URI, a server SHOULD respond with a status code of 200 (OK) and a Content-Type header corresponding to the archive format used (for example, application/zip for a Zip archive). Otherwise, a server SHOULD respond with a status code of 404 (NOT FOUND).

HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: public, immutable
Content-Type: application/zip
Content-Disposition: attachment; filename="LinkedList-1.1.1.zip"
Content-Length: 2048
Content-Version: 1
Digest: sha-256=a2ac54cf25fbc1ad0028f03f0aa4b96833b83bb05a14e510892bb27dea4dc812
ETag: e61befdd5056d4b8bafa71c5bbb41d71
Link: <https://swift.pkg.github.com/mona/LinkedList/1.1.1.zip.asc>; rel=describedby; type="application/pgp-signature", <https://mirror-japanwest.github.com/mona-LinkedList-1.1.1.zip>; rel=duplicate; geo=jp; pri=10; type="application/zip"

A server SHOULD respond with a Content-Length header set to the size of the archive in bytes.

A server SHOULD respond with a Content-Disposition header set to attachment with a filename parameter equal to the name of the package followed by a hyphen (-), the version number, and file extension (for example, "LinkedList-1.1.1.zip").

A server SHOULD respond with a link to a digital signature for the archive using a Link header field entry with a describedby relation and an application/pgp-signature type.

It is RECOMMENDED for clients and servers to support range requests as described by RFC 7233, caching as described by RFC 7234, and instance digests as described by RFC 3230 and RFC 5843.

A server MAY specify mirrors or multiple download locations using Link header fields with a duplicate relation, as described by RFC 6249. A client MAY use this information to determine its preferred strategy for downloading.

A server that indexes but doesn't host packages SHOULD respond with a status code of 303 (See Other) and redirect to a hosted package archive if one is available. Otherwise, a server MAY respond with a status code of 501 (Not Implemented).

HTTP/1.1 303 See Other
Content-Version: 1
Location: https://swift.pkg.github.com/mona/LinkedList/1.1.1.zip
Digital signature of source archive

A client MAY send a GET request for a URI matching the expression {/namespace*}/{package}/{version} to retrieve a detached digital signature that can be used to verify the release's source archive according to RFC 4880. A client SHOULD set the Accept header to application/pgp-signature and SHOULD append .asc to the originally requested URI (for example .zip.asc for the signature of the Zip archive).

GET /mona/LinkedList/1.1.1.zip.asc HTTP/1.1
Accept: application/pgp-signature
Accept-Version: 1

If a release is found for the requested URI, a server SHOULD respond with a status code of 200 (OK) and a Content-Type header set to application/pgp-signature. Otherwise, a server SHOULD respond with a status code of 404 (NOT FOUND).

HTTP/1.1 200 OK
Cache-Control: public
Content-Type: application/pgp-signature
Content-Disposition: attachment; filename="LinkedList-1.1.1.zip.asc"
Content-Length: 833
Content-Version: 1

A client SHOULD verify the contents of a downloaded source archive using the provided signature (for example, using the command gpg --verify LinkedList-1.1.1.zip.asc LinkedList-1.1.1.zip).

A server that indexes but doesn't host packages SHOULD respond with a status code of 303 (See Other) and redirect to the digital signature for the archive if one is available. Otherwise, a server MAY respond with a status code of 501 (Not Implemented).

HTTP/1.1 303 See Other
Content-Version: 1
Location: https://swift.pkg.github.com/mona/LinkedList/1.1.1.zip.asc
Metadata

A client MAY send a GET request for a URI matching the expression {/namespace*}/{package}/{version} to retrieve metadata about a release. A client SHOULD set the Accept header with an application/json content type, and MAY append the .json extension to the requested URI.

GET /mona/LinkedList/1.1.1 HTTP/1.1
Accept: application/json
Accept-Version: 1

If a release is found for the requested URI, a server SHOULD respond with a status code of 200 (OK). Otherwise, a server SHOULD respond with a status code of 404 (NOT FOUND).

HTTP/1.1 200 OK
Content-Type: application/json
Content-Version: 1
Link: <https://example.com/mona/LinkedList/1.1.1>; rel="latest-version",
      <https://example.com/mona/LinkedList/1.0.0>; rel="predecessor-version"

A server SHOULD respond with a Link header containing the following entries:

Relation Description
latest-version The latest published release of the package
successor-version The next published release of the package, if one exists
predecessor-version The previous published release of the package, if one exists

A link with the latest-version relation MAY correspond to the requested release.

A server MAY include additional fields in its response. It is RECOMMENDED that package metadata be represented in JSON-LD according to a structured data standard. For example, this response using the Schema.org SoftwareSourceCode vocabulary:

{
  "@context": ["http://schema.org"],
  "@type": "SoftwareSourceCode",
  "identifier": "@mona/LinkedList",
  "name": "LinkedList",
  "description": "One thing links to another.",
  "keywords": ["data-structure", "collection"],
  "version": "1.1.1",
  "codeRepository": "https://github.com/mona/LinkedList",
  "license": "https://www.apache.org/licenses/LICENSE-2.0",
  "readme": "https://github.com/mona/LinkedList/blob/master/README.md",
  "issueTracker": "https://github.com/mona/LinkedList/issues",
  "programmingLanguage": {
    "@type": "ComputerLanguage",
    "name": "Swift",
    "url": "https://swift.org"
  },
  "author": {
      "@type": "Person",
      "@id": "https://github.com/mona",
      "givenName": "Mona",
      "middleName": "Lisa",
      "familyName": "Octocat",
  }
}

Publish a new package release

A client MAY send a PUT request for a URI matching the expression {/namespace*}/{package}/{version}{?commit,branch,tag,path,url} to publish a release of a package.

PUT /mona/LinkedList/1.1.1 HTTP/1.1
Accept-Version: 1

A client MAY publish releases in any order. For example, if a package has existing 1.0.0 and 2.0.0 releases, a client MAY publish a new 1.0.1 or 1.1.0 release.

A client MAY include additional information in its request body encoded as a JSON document with the Content-Type header set to application/json. It is RECOMMENDED that package metadata be represented in JSON-LD according to a structured data standard, as discussed in "Fetch a package release". A client MAY send subsequent requests to update metadata for existing releases. A server MAY refuse to update the metadata for a package release by responding with a status code of 403 (Forbidden).

If no release currently exists for the requested version, a server SHOULD respond quickly with a status code of 202 (Accepted) to acknowledge that the request is being processed. This response SHOULD also contain a Location header with a URL that the client can poll for progress updates and a Retry-After header with an estimate of when processing is expected to finish. A server MAY locate the status resource endpoint at a URI of its choosing. However, the use of a non-sequential, randomly-generated identifier is RECOMMENDED.

HTTP/1.1 202 Accepted
Content-Version: 1
Location: https://swift.pkg.github.com/submissions/90D8CC77-A576-47AE-A531-D6402C4E33BC
Retry-After: 120

A server SHOULD attempt to access the source of the package release before sending a response, so that it may communicate any errors to the client immediately. A server MAY set a timeout to guarantee a timely response to each request.

If a request to publish a new package release were to fail, a server MUST communicate that failure in the same way if sending an immediate response as it would if responding to a client polling for status.

Source repository location

A server SHOULD determine the location of a package's source repository from the requested URI, if possible. A client MAY specify an explicit location with a url query item parameter.

PUT /mona/LinkedList/1.1.1?url=https://github.com/mona/LinkedList.git HTTP/1.1
Accept-Version: 1

A server MAY refuse the specified source repository location by responding with a status code of 403 (Forbidden).

HTTP/1.1 403 Forbidden
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en

{
     "detail": "invalid parameter, 'url'"
}

If a server is unable to access the repository, it SHOULD respond with 502 (Bad Gateway) with additional details about the failure in the response body.

HTTP/1.1 502 Bad Gateway
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en

{
   "detail": "unable to connect to repository"
}
Source revision

The version path parameter in a client's request MUST be a valid version number for the release and SHOULD correspond to a tag in the package's source repository. A client MAY specify a different commit reference with any combination of commit, tag, and branch query item parameters.

PUT /mona/LinkedList/1.1.1?tag=v1.1.1 HTTP/1.1
Accept-Version: 1

A server SHOULD respond with a status code of 404 (Not Found) if it's unable to resolve the requested version in the repository with the provided commit, tag, or branch parameters.

HTTP/1.1 404 Not Found
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en

{
   "detail": "tag '2.0.0' not found"
}

A server SHOULD respond with a status code of 409 (Conflict) if any provided commit, tag, or branch parameters cannot be resolved to a single revision.

HTTP/1.1 409 Conflict
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en

{
   "detail": "tag '2.0.0' doesn't refer to commit '1F34E'"
}

A server MAY refuse to publish a package if the specified tag or commit isn't signed by responding with a status code of 403 (Forbidden).

HTTP/1.1 403 Forbidden
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en

{
   "detail": "tag '2.0.0' must be signed"
}

A server SHOULD support Git repositories. A server MAY support other source control systems, such as Mercurial or Subversion.

Package location

A client MAY use path and url parameters together to publish a single package located in a subdirectory of a repository or to publish multiple packages from different paths in a single repository.

If a client specifies a path parameter, a server SHOULD look for a Package.swift file at that location. Otherwise, it SHOULD look at the root directory. If a Package.swift file doesn't exist at that location, a server SHOULD respond with a status code of 404 (Not Found).

Unpublish a package release

A client MAY send a DELETE request for a URI matching the expression {/namespace*}/{package}/{version} to remove an existing release from a package registry.

DELETE /mona/LinkedList/1.1.1 HTTP/1.1
Content-Version: 1

Support for this endpoint is OPTIONAL. A server that chooses not to process requests for unpublishing a package release SHALL respond with a status code of 405 (Method Not Allowed).

HTTP/1.1 405 Method Not Allowed
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en

{
   "detail": "packages cannot be unpublished through the API"
}

If a server accepts a request for removal, it SHOULD respond quickly with a status code of 202 (Accepted) to acknowledge that the request is being processed. This response SHOULD also contain a Location header with a URL that the client can poll for progress updates and a Retry-After header with an estimate of when processing is expected to finish. A server MAY locate the status resource endpoint at a URI of its choosing. However, the use of a non-sequential, randomly-generated identifier is RECOMMENDED.

HTTP/1.1 202 Accepted
Content-Version: 1
Location: https://swift.pkg.github.com/submissions/0CF1FC65-D928-41E5-A825-3BEB9D69C004
Retry-After: 120

Once a package release is removed, a server SHOULD NOT publish any new releases with the same combination of namespace, package name, and version number.

If a client attempts to unpublish a package that has already been removed from the registry, a server SHOULD return 204 (No Content).

HTTP/1.1 204 No Content
Content-Version: 1

If a client fetches a package that has been unpublished from the registry, a server SHOULD respond with a status code of 410 (Gone), or MAY instead redirect to an alternate release by responding with a status code of 301 (Moved Permanently).

HTTP/1.1 410 Gone
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en

{
    "detail": "this release was removed from the registry"
}

(Continued in next post)

24 Likes

Normative References

  • RFC 2119: Key words for use in RFCs to Indicate Requirement Levels
  • RFC 3230: Instance Digests in HTTP
  • RFC 3986: Uniform Resource Identifier (URI): Generic Syntax
  • RFC 4880: OpenPGP Message Format
  • RFC 5843: Additional Hash Algorithms for HTTP Instance Digests
  • RFC 6249: Metalink/HTTP: Mirrors and Hashes
  • RFC 6570: URI Template
  • RFC 7230: Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing
  • RFC 7231: Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content
  • RFC 7233: Hypertext Transfer Protocol (HTTP/1.1): Range Requests
  • RFC 7234: Hypertext Transfer Protocol (HTTP/1.1): Caching
  • RFC 7807: Problem Details for HTTP APIs
  • RFC 8288: Web Linking
  • SemVer: Semantic Versioning

Informative References

  • BCP 13 Media Type Specifications and Registration Procedures
  • RFC 6749: The OAuth 2.0 Authorization Framework
  • RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3
  • TR36: Unicode Technical Report #36: Unicode Security Considerations
  • JSON-LD: A JSON-based Serialization for Linked Data
  • Schema.org A shared vocabulary for structured data.

Security

A package registry can offer stronger guarantees for safety and security compared to downloading dependencies using Git. Providing a checksum for the source archive of a package release allows a client to validate the integrity of the downloaded content. Providing a digital signature allows a server to certify that content to a client in a verifiable way.

We acknowledge an inherent privacy concern with repository hosts and package registries logging the IP addresses for each package requested. The use of proxies and other techniques can help mitigate this risk, but we believe the best solution is for services to operate according to a stated privacy policy.

Impact on existing packages

Current packages won't be affected by this change, as they'll continue to be able to download dependencies directly through Git.

Swift developers can opt-in to package registries on a per-dependency basis by changing its URL from one that points to a repository to one that resolves to a package endpoint.

  let package = Package(
      name: "MyPackage",
      dependencies: [
-        .package(url: "https://github.com/apple/swift-argument-parser.git", from: "0.0.1"),
+        .package(url: "https://swift.pkg.github.com/apple/swift-argument-parser", from: "0.0.1"),
      ]

Alternatives considered

We first considered the more traditional "push"-based package registry design. However, this model creates a separate release process that runs parallel to version control, which not only duplicates effort but has the potential to introduce discrepancies between tags and releases.

Future directions

Defining a standard interface for package registries lays the groundwork for several useful features.

Swift Package Manager integration

A package registry can be useful in its own right, but enjoying its full benefit requires additional functionality in Swift Package Manager. We intend to submit a separate proposal for doing this.

Offline cache

Swift Package Manager could implement an offline cache that allows it to work without network access. While this is technically possible today, a package registry allows for a simpler and more secure implementation than would otherwise be possible with Git repositories alone.

Security auditing

The response for listing package releases could be updated to include information about security advisories.

/* straw-man proposal */
{
    "releases": { /* ... */ },
    "advisories": [{
        "cve": "CVE-20XX-12345",
        "cwe": "CWE-400",
        "package_name": "mona/LinkedList",
        "vulnerable_versions": "<=1.0.0",
        "patched_versions": ">1.0.0",
        "severity": "moderate",
        "recommendation": "Update to version 1.0.1 or later.",
        /* additional fields */
    }]
}

Swift Package Manager could communicate this information to users when installing or updating dependencies or as part of a new swift package audit subcommand.

$ swift package audit
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ High          β”‚ Regular Expression Denial of Service                         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Package       β”‚ RegEx                                                        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Dependency of β”‚ PatternMatcher                                               β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Path          β”‚ SomePackage > PatternMatcher > RegEx                         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ More info     β”‚ https://example.com/advisories/526                           β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Found 3 vulnerability (1 low, 1 moderate, 1 high) in 12 scanned packages.
  Run `swift package audit fix` to fix 3 of them.

Package removal reason

The request for unpublishing a package could be amended to accept additional parameters in the request body that explain why a package is being removed. For example, a maintainer could unpublish a package release in response to a security vulnerability or because they didn't intend to publish it in the first place. This endpoint could also be amended to distinguish between removal and deprecation of package releases.

Binary framework distribution

The package registry specification could be amended to add support for distributing packages as XCFramework bundles.

GET /mona/LinkedList/1.1.1.xcframework HTTP/1.1
Accept: application/vnd.apple.xcframework
Accept-Version: 1

Swift Package Manager could then use XCFramework archives as binary dependencies or as part of a future binary artifact distribution mechanism.

let package = Package(
    name: "SomePackage",
    /* ... */
    targets: [
        .binaryTarget(
            name: "LinkedList",
            url: "https://swift.pkg.github.com/mona/LinkedList/1.1.1.xcframework",
            checksum: "ed04a550c2c7537f2a02ab44dd329f9e74f9f4d3e773eb883132e0aa51438b37"
        ),
    ]
)
21 Likes

This will wreak havoc on graph resolution. How is SwiftPM supposed to know both URLs point at the same thing, so that you don’t end up with both in your graph at once?

Is there no way to stick with the existing URL and have an API that basically asks β€œDo you support the registry API?” Then SwiftPM could opt to interact with the registry if it got an intelligible response, and fall back to the old Git method if not?

4 Likes

I'm not sure I'm understanding the problem here, did you notice that it is a diff-style example where the github url was removed and replaced with the swift.pkg.github url? I'm just asking because you said "both URLs" but there would only be one.

I believe what @SDGGiesbrecht is concerned about is that you could end up with two dependencies each depending on the same package, one specifying β€œgithub.com” and the other specifying β€œswift.pkg.github.com”.

2 Likes

To be clear, that code listing under "Impact on existing packages" is presented as an example of how Swift Package Manager integration could be implemented, not a specific proposal for how it should be done.

If this approach were to be adopted, I can think of a few different ways to resolve transient dependencies with different URLs:

  • Add a Link header entry to the package registry response for a package release that points to the canonical source repository url
  • Update .package in the next swift-tools-version to include a name or some other canonical mapping
  • Define a mechanism for git URLs to automatically redirect to package registry API endpoints when accessed by Swift Package Manager, such as a User-Agent or other header

I'm not sure how Swift Package Manager currently deals with case differences in shared dependencies (e.g. github.com/alamofire/alamofire.git vs. github.com/Alamofire/Alamofire.git), but I think it'd be instructive to consider these problems together.

Anyway, this is absolutely something that we should figure out in the follow-up proposal for Swift Package Manager integration. Thanks for calling this out!

3 Likes

This looks great! Thanks @mattt + :100:

Couldn't you see that both have the same Package.name?

3 Likes

There are plenty of bugs in this area. SwiftPM invalidates all pins, retries resolution, and refuses to pin the offending package. Super slow, but it does succeed in the end. Xcode just crashes.

But I agree it doesn’t necessarily have to be solved here, only that it be on our minds so we don’t exacerbate it.

As I mentioned above, the whole thing is currently a mess. The declared name would be suboptimal as a unique ID, because if it clashes between two legitimately different packages, SwiftPM would think they were the same and never fetch the second. The diagnostics would then be completely tangled, sending you on a wild goose chase.

However, maybe SwiftPM could fetch both and then treat them the same if their entire contents match, thereafter discarding the extraneous copy?

But I’ll leave it at that, because we’re drifting off topic.

2 Likes

+1

I think we really need to take a look at package identity/name clashes in SwiftPM but that deserves its own separate discussion.

7 Likes

I'm very excited to see some movement in this space. A small comment below:

I just read through RFC 7807, and I'm not entirely sure I got this right, but shouldn't the type member be mandatory? If that's the case, shouldn't the current specification define some standard type URIs?

1 Like

To be honest, I'm not entirely confident in my own reading of RFC 7807. I would have liked for there to be a few more SHALL / SHOULD / MAY for each field. Looking at section 3.1 (emphasis added):

A problem details object can have the following members:

  • "type" (string) - A URI reference [RFC3986] that identifies the problem type. This specification encourages that, when dereferenced, it provide human-readable documentation for the problem type (e.g., using HTML [W3C.REC-html5-20141028]). When this member is not present, its value is assumed to be "about:blank".

As far as I can tell, an omitted type value shouldn't be a problem. I think it's acceptable to delegate this decision to each server implementation, as they may have their own preferred semantics / existing support ontology.

Perhaps having this example in the proposal is more of a distraction than it is worth considering there are many issues regarding the final SwiftPM integration that would have to be worked out in the future proposal.

Maybe just stating that dependencies from a registry will be specified in a manner that is distinct from and complementary to the current git-based dependencies is sufficient for this proposal.

1 Like

One use case made possible by a registry is having a mirror that tracks and curates (for example based on approved licenses) releases as they occur. Is there mechanisms for such mirrors to discover activity from its root repository to achieve such use cases?

At the very least seems like this would require a GET endpoint to list packages/sub-namespaces at each level. Feels like this endpoint would also be useful for discoverability/visibility.

Two questions for the current release list API-

  • will it be paginated or return the complete list of all releases for the package
  • is there a defined order that the releases are required be returned in
3 Likes

I agree. I'll update that for the final proposal.

There are various strategies for coordinating new releases between a mirror and its source, but those are outside the scope of this specification. For example, a package registry could provide users a web hook that can be pinged to initiate a refresh. Or it could simply poll at a fixed interval. I think a likely option will be for registries to provide a CI integration, such as a GitHub Action, to automate the PUT request to publish a new version whenever a tag is pushed.

The thing to keep in mind is that this specification defines a minimal set of common APIs. A registry can implement any additional functionality on top of this internally or though other API endpoints.

The current design doesn't paginate. My rationale is that the overwhelming majority of packages have fewer than 100 releases. Yet, as I type this, I now recall Apple's practice of tags for nightly releases of their open source libraries, and how it overwhelm's GitHub's web UI.

So I take your point. I'll add an affordance for pagination to the specification (probably something lightweight using Link relations). Thanks!

According to the spec, the precedence ordering is SHOULD but not SHALL. That might have to change if we add pagination.

I wonder how SPM handles differences between https urls and ssh urls.

2 Likes

I really like this proposal. Whenever a Swift package registry is brought up, I get nervous that we will end up with a centralized approach, or something that encourages name squatting.

I do wonder how SPM would differentiate http git urls from package urls. Github has long supported dropping .git from urls so that wouldn't be a reliable way to distinguish the 2. Perhaps a custom url scheme (that is actually just https) would work? Something like swift://swift.pkg.github.com/apple/swift-argument-parser? Or we could break the urls up and have a sources search list and a package from a registry would just be the name like .package(name: "apple/swift-argument-parser", from: "0.0.1").

3 Likes

I think its a great proposal in general and am happy to see traction in this area.

On the scope, I feel like package discovery is something that could definitely be improved upon but doesn't appear to be addressed here, would it be up to package registries to add search as part of their platforms if appropriate? Would we not want a standard way to search for packages to support a search command (eg. swift package search)?

3 Likes

Package discovery is something that I'm excited for registries to help solve. That said, I think search is a complex problem that could be better-served by existing standards like OpenSearch, or left as an implementation detail for each individual registry.

Without a centralized search index, I'm not sure how this would work. Certainly any individual registry could provide their own search tool, but it may not make as much sense in a subcommand of swift package.

1 Like

Not saying that this is the correct solution, but GitHub could respond differently based on a different media type query (Accept: vnd.org.swift+json) or user agent header (User-Agent: swift package manager 5.2.4).

I would discourage the use of custom URI schemes. Apple used this in early versions of iOS to support app linking, but has since β€” for good reason β€” moved towards universal links instead.

This is the approach I'm most familiar with by way of RubyGems. I'd have to ask around to see how Ruby developers feel about these in practice β€” especially with the advent of GitHub Packages.