Hi everyone,
Early in the process of pitching the package registry service, we decided to reduce scope to only the endpoints necessary for resolving package dependencies through a registry. Now that SE-0292 is accepted, I'm excited to revisit this functionality.
Below is our pitch for adding a new publishing endpoint to the package registry specification, which would provide a standard way for new package releases to be added to a registry. The first half of the post describes the feature in the structure of a Swift Evolution proposal. The second half contains an update to the registry specification, with more precise details about what this means for implementors.
Thanks to @yim_lee and @mmarston for providing some excellent feedback on an earlier draft of this proposal. I look forward to getting even more feedback from the community ahead of our formal proposal.
Package Registry Service - Publish Endpoint
Introduction
The package registry service defines endpoints for fetching packages.
This proposal extends the existing package registry specification with endpoints for publishing a package release.
Motivation
A package registry is responsible for determining which package releases are made available to a consumer.
Currently, the availability of a package release is determined by an out-of-band process. For example, a registry may consult an index of public Swift packages and make releases available for each tag with a valid version number.
Having a standard endpoint for publishing a new release to a package registry would empower maintainers to distribute their software and promote interoperability across service providers.
Proposed solution
We propose to add the following endpoint to the registry specification:
| Method | Path | Description |
|---|---|---|
POST |
/{scope}/{name}/{version} |
Create a package release |
The goal of this proposal is to provide enough definition to ensure a secure, robust mechanism for distributing software while allowing individual registries enough flexibility in their governance and operation. For instance, support for this endpoint would be optional, so package registries may elect not to allow packages to be published. And because there's an expectation of durability — that is, package releases aren't removed after they're published — registries make the ultimate determination of what is made available.
Detailed design
This proposal amends the registry specification with a new, optional endpoint that a registry may implement to support the publication of packages through the web service interface. To understand what the feature does and how it works, consider the following use case:
A maintainer of an open-source Swift package ( mona.LinkedList ) creates a new release (version 1.1.1 ), and wants to submit it to a registry ( packages.example.com ) for distribution.
First, they run the swift package archive-source subcommand to generate a Zip file ( LinkedList-1.1.1.zip ) of their package.
$ swift package archive-source
Next, they upload their release to a package registry by making the following request:
$ curl -X POST --netrc \
-H "Accept: application/vnd.swift.registry.v1+json" \
-F source-archive="@LinkedList-1.1.1.zip" \
"https://registry.example.com/mona/LinkedList?version=1.1.1"
The registry can respond to this request synchronously or asynchronously. This allows the server an opportunity to perform any necessary analysis and processing to ensure software quality and update its data stores.
After receiving and processing this request, the registry can make mona.LinkedList at version 1.1.1 available by including it in the response to GET /mona/LinkedList .
$ curl -X GET -H "Accept: application/vnd.swift.registry.v1+json" \
"https://registry.example.com/mona/LinkedList" \
| jq ".[] | keys"
[ "1.0.0", "1.1.0", "1.1.1", ]
The next time a developer with a package that depends on mona.LinkedList resolves the dependencies of that package, Swift Package Manager would see 1.1.1 , and may attempt to update to this new version. If the version is selected, the client would download the source archive for this release by sending the request GET /mona/LinkedList/1.1.1.zip .
Security
Although this proposal has no direct impact on Swift Package Manager, it's important to consider the security implications of introducing a publishing endpoint to the Swift package ecosystem. To do this, we employ the STRIDE mnemonic below:
Spoofing
An attacker could attempt to impersonate a package maintainer in order to publish a new release containing malicious code.
Because the likelihood and potential impact of such an attack is high, registry service providers should take all necessary precautions. The registry specification recommends the use of multi-factor authentication for all requests to publish a package release.
Additional countermeasures like rate-limiting suspicious requests and analyzing uploaded source archives can also help mitigate the risk of this kind of attack.
An attacker could also attempt to trick users into downloading malicious code by publishing a package with an identifier similar to a legitimate one (for example, 4pple.swift-nio , which looks like apple/swift-nio ). A registry can mitigate typosquatting attacks like this by comparing the similarity of a new submission to existing package names with a string metric like Damerau–Levenshtein distance.
Tampering
An attacker could maliciously tamper with a generated source archive in an attempt to exploit a known vulnerability like Zip Slip, or a common software weakness like susceptibility to a Zip bomb.
Registry services should take care to identify and protect against these kinds of attacks in its implementation of source archive decompression.
To further improve the security of package submissions, a registry could restrict publishing to trusted clients, for which a chain of custody can be established. (This is effectively the "pull" model described above).
Repudiation
A dispute could arise between a package maintainer and a registry about the content or existence of a package release.
This proposal doesn't specifically provide a mechanism for resolving such a dispute. However, the design supports a variety of possible solutions. For example, a software bill of materials and the use of digital signatures can both provide non-repudiation guarantees about the provenance of package release artifacts.
Information disclosure
A user could inadvertently expose credentials when uploading a source archive for a package release.
This threat isn't substantially different from that of leaking credentials in source code with version management software, so similar strategies can be employed here. For example, registry services can help minimize this risk by rejecting any submissions that contain sensitive information.
Denial of service
An attacker could upload large payloads in an attempt to reduce the availability of a registry.
This kind of attack is typical for any web service with an endpoint for uploading resources. A registry can mitigate this threat using defensive coding practices like performing authentication checks before processing request bodies, limiting the maximum allowed size of a message payload, and routing requests through a reverse proxy or load balancer.
Escalation of privilege
It's desirable for a registry to have information about the content of a release submitted for publishing, such as the package's supported platforms, products, and dependencies. Swift package manifest files are executable code and must be evaluated by the Swift toolchain to determine this information. An attacker could construct a malicious Package.swift file containing system calls in an attempt to perform remote code execution.
Registry services should take care to evaluate package manifest files in an unprivileged container to mitigate the risk of evaluating untrusted code.
Impact on existing packages
This feature provides a mechanism for package maintainers and registries to migrate existing packages from the current URL-based system to the new registry scheme.
The specific strategy for rolling out this functionality is something to be determined by each registry operator in advance of this feature.
Alternatives considered
Endpoint for scope registration
This proposal sets no policies for how package scopes are registered or verified.
Endpoint for publishing with "pull" model
Many package managers and artifact repository services follow what we describe as a "push" model of publication: When a maintainer wants to releases a new version of their software, they produce a build locally and push the resulting artifact to a server.
For example, a developer can distribute their Ruby library by building a .gem archive and pushing it to a server like RubyGems.org.
$ gem build octokit.gemspec
$ gem push octokit-4.20.0.gem
This model has the benefit of operational simplicity and flexibility. For example, maintainers have an opportunity to digitally sign artifacts before uploading them to the server.
Alternatively, a system might incorporate build automation techniques like continuous integration (CI) and continuous delivery (CD) into what we describe as a "pull" model: When a maintainer wants to release a new version of their software, their sole responsibility is to notify the registry; the server does all the work of downloading the source code and packaging it up for distribution.
For example, in addition to supporting the "push" model, Docker Hub can automatically build images from source code push the built image to a repository.
This model can provide strong guarantees about reproducibility, quality assurance, and software traceability.
Initial drafts for this proposal included separate endpoints for publishing with the "pull" and "push" models, with a preference for the former and its stronger guarantees of traceability. However, we determined that while these models provide a useful framework for understanding software distribution models, they are both accommodated by a single endpoint; a "pull" is equivalent a "push" where the client and server are a single entity.
Future directions
Swift Package Manager subcommand for publishing
Swift Package Manager could be updated to add a new swift package publish subcommand, that provides a more convenient interface for publishing packages to a registry. For example, it could automatically read the configuration in .swiftpm/config/registries.json to determine the correct registry endpoint, or read the user's .netrc file to authenticate the request.
The command could also subsume swift package archive-source and perform additional tasks before uploading, such as generating a software bill of materials or signing the source archive.
This feature wasn't included in the proposal because it's unnecessary for the core publishing functionality. We are also concerned that this command could bloat to the command-line interface and undermine the benefits of publishing within a CI/CD system. However, if the community finds this to be a useful feature, we'd be happy to include it in an amendment to our proposal.
Mechanism for syndicating publishing activity
A registry could syndicate new releases through an activity stream or RSS feed. This functionality could be used as an information source by package indexes or to provide federation across different registries.
Transparency logs
Similar to a syndication feed, each new package release could be added to an append-only log like Trillian or sigstore.
Package Registry Specification (Additions)
4.6. Create a package release
A client MAY send a POST request for a URI matching the expression /{scope}/{name}/{version} to publish a release of a package. A client MUST provide a body encoded as multipart form data with the following sections:
| Key | Content-Type | Description | Requirement Level |
|---|---|---|---|
source-archive |
application/zip |
The source archive of the package. | REQUIRED |
metadata |
application/json |
Additional information about the release. | OPTIONAL |
A client MUST set a Content-Type header with the value multipart/form-data , and a Content-Length header with the total size of the body in bytes. A client SHOULD set the Accept header with the value application/vnd.swift.registry.v1+json .
POST /mona/LinkedList?version=1.1.1 HTTP/1.1
Host: packages.example.com
Accept: application/vnd.swift.registry.v1+json
Content-Type: multipart/form-data;boundary="boundary"
Content-Length: 336
Expect: 100-continue
--boundary
Content-Disposition: form-data; name="source-archive"
Content-Type: application/zip
Content-Length: 32
Content-Transfer-Encoding: base64
gHUFBgAAAAAAAAAAAAAAAAAAAAAAAA==
--boundary
Content-Disposition: form-data; name="metadata"
Content-Type: application/json
Content-Transfer-Encoding: quoted-printable
Content-Length: 3
{ }
A server SHOULD require a client to perform authentication for any requests to create a package release. Use of multi-factor authentication is RECOMMENDED.
A client MAY publish releases in any order. For example, if a package has existing 1.0.0 and 2.0.0 releases, a client MAY publish a new 1.0.1 or 1.1.0 release.
Once a release has been published, any resources associated with that release, including its source archive, MUST NOT change.
If a release already exists for a package at the specified version, the server SHOULD respond with a status code of 409 (Conflict).
HTTP/1.1 409 Conflict
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en
{
"detail": "a release with version 1.0.0 already exists"
}
It is RECOMMENDED that a server institute policies for publishing new releases of a package after a scope is transferred to a new owner. For example, the next release of an existing package is published with a new major version, or only after a period of 45 days after transfer.
If the client provides an Expect header, a server SHOULD check that the request can succeed before responding with a status code of 100 (Continue) . A server that doesn't support expectations SHOULD respond with a status code of 417 (Expectation Failed) . In response, a client MAY remove the Expect header and retry the request.
HTTP/1.1 417 (Expectation Failed)
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en
{
"detail": "expectations aren't supported"
}
Support for this endpoint is OPTIONAL. A server SHOULD indicate that publishing isn't supported by responding with a status code of 405 (Method Not Allowed).
HTTP/1.1 405 (Method Not Allowed)
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en
{
"detail": "publishing isn't supported"
}
A server MAY respond either synchronously or asynchronously. For more information, see 4.6.4.
4.6.1 Source archive
A client MUST include a multipart section named source-archive containing the source archive for a release. A client SHOULD set a Content-Type header with the value application/zip and a Content-Length header with the size of the Zip archive in bytes.
--boundary
Content-Disposition: form-data; name="source-archive"
Content-Type: application/zip
Content-Length: 32
Content-Transfer-Encoding: base64
gHUFBgAAAAAAAAAAAAAAAAAAAAAAAA==
A client SHOULD use the swift package archive-source tool to create a source archive for the release.
A server MAY analyze a package to assess its viability, perform security testing, or otherwise evaluate software quality. A server MAY refuse to publish a package release for any reason by responding with a status code of 422 (Unprocessable Entity).
HTTP/1.1 422 Unprocessable Entity
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en
{
"detail": "package doesn't contain a valid manifest (Package.swift) file"
}
A server SHOULD use the swift package compute-checksum tool to compute the checksum that's provided in response to a client's subsequent request to download the source archive for the release.
4.6.2. Package release metadata
A client MAY include a multipart section named metadata containing additional information about the release. A client SHOULD set a Content-Type header with the value application/json and a Content-Length header with the size of the JSON document in bytes. It is RECOMMENDED that package release metadata be represented in JSON-LD according to a structured data standard, as discussed in 4.2.1.
--boundary
Content-Disposition: form-data; name="metadata"
Content-Type: application/json
Content-Length: 620
Content-Transfer-Encoding: quoted-printable
{
"@context": ["http://schema.org/"],
"@type": "SoftwareSourceCode",
"name": "LinkedList",
"description": "One thing links to another.",
"keywords": ["data-structure", "collection"],
"version": "1.1.1",
"codeRepository": "https://github.com/mona/LinkedList",
"license": "https://www.apache.org/licenses/LICENSE-2.0",
"programmingLanguage": {
"@type": "ComputerLanguage",
"name": "Swift",
"url": "https://swift.org"
},
"author": {
"@type": "Person",
"@id": "https://github.com/mona",
"givenName": "Mona",
"middleName": "Lisa",
"familyName": "Octocat"
}
}
If a client doesn't provide a metadata section, a server MAY populate the metadata for a release. A client MAY request that a server not populate metadata automatically by sending an empty JSON object ( {} ) as its request body.
If a client provides an invalid JSON document, the server SHOULD respond with a status code of 422 (Unprocessable Entity) or 413 (Payload Too Large) and MAY communicate validation error details in the response body.
HTTP/1.1 422 Unprocessable Entity
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en
{
"detail": "invalid JSON provided for release metadata"
}
4.6.3 Synchronous and asynchronous publication
A server MAY respond to a request to publish a new package release either synchronously or asynchronously.
A client MAY indicate their preference for asynchronous processing with a Prefer header field containing the token respond-async and an optional wait preference, as described by RFC 7240.
POST /mona/LinkedList/1.1.1 HTTP/1.1
Host: packages.example.com
Accept: application/vnd.swift.registry.v1
Prefer: respond-async, wait=300
4.6.3.1 Synchronous publication
If processing is done synchronously, the server MUST respond with a status code of 201 (Created) to indicate that the package release was published. This response SHOULD also contain a Location header with a URL to the new release.
HTTP/1.1 201 Created
Content-Version: 1
Location: https://packages.example.com/github.com/mona/LinkedList/1.1.1
A client MAY set a timeout to guarantee a timely response to each request.
4.6.3.2 Asynchronous publication
If processing is done asynchronously, the server MUST respond with a status code of 202 (Accepted) to acknowledge that the request is being processed. This response MUST contain a Location header with a URL that the client can poll for progress updates and SHOULD contain a Retry-After header with an estimate of when processing is expected to finish. A server MAY locate the status resource endpoint at a URI of its choosing. However, the use of a non-sequential, randomly-generated identifier is RECOMMENDED.
HTTP/1.1 202 Accepted
Content-Version: 1
Location: https://packages.example.com/submissions/90D8CC77-A576-47AE-A531-D6402C4E33BC
Retry-After: 120
A client MAY send a GET request to the location provided by the server in response to a publish request to see the current status of that process.
GET /submissions/90D8CC77-A576-47AE-A531-D6402C4E33BC HTTP/1.1
Host: packages.example.com
Accept: application/vnd.swift.registry.v1
If the asynchronous publish request is still processing, the server SHOULD respond with a status code of 204 (No Content) and a Retry-After header with an estimate of when processing is expected to finish.
HTTP/1.1 204 No Content
Content-Version: 1
Retry-After: 120
If the asynchronous publish request is finished processing successfully, the server SHOULD respond with a status code of 301 (Moved Permanently) and a Location header with a URL to the package release.
HTTP/1.1 301 Moved Permanently
Content-Version: 1
Location: https://packages.example.com/mona/LinkedList/1.1.1
If the asynchronous publish request failed, the server SHOULD respond with a status code of 205 (Reset Content).
HTTP/1.1 205 Reset Content
Content-Version: 1
A client MAY send a DELETE request to the location provided by the server in response to a publish request to cancel that process.
If a request to publish a new package release were to fail, a server MUST communicate that failure in the same way if sending an immediate response as it would if responding to a client polling for status.
If a client makes a request to publish a package release to a server that is asynchronously processing a request to publish that release, the server MUST respond with a status code of 409 (Conflict)
HTTP/1.1 409 Conflict
Content-Version: 1
Content-Type: application/problem+json
Content-Language: en
Location: https://packages.example.com/submissions/90D8CC77-A576-47AE-A531-D6402C4E33BC
{
"detail": "already processing a request to publish this package version"
}
If a client makes a request to publish a package release to a server that finished processing a failed request to publish that release, the server SHOULD try publishing that release again. A server MAY refuse to fulfill a subsequent request to publish a package release by responding with a status code of 409 (Conflict).
Even if projects had pinned the specific version they knew was license wise okey, the'd suddenly be in trouble (ignore the legal capability if such post-factum license change of an already released under one specific license artifact is okey or not -- for a new release to have a different license is fine, but such pulling the rug from under projects is pretty nasty and should not be supported).