SE-0292: Package Registry Service

[mattt]

@StanTwinB Thank you for sharing your feedback! I'll do my best to address each of your points below.

But before we dive in, I wanted to make sure that y'all saw the package registry specification itself (from this PR), which goes into much greater detail about the specifics of how everything works.

You can also refer to previous discussion threads [1] [2] [3] for additional context for some of the decisions we made regarding identity, security, and other concerns.

@StanTwinB

Package Identity

We recommend adding an additional identifier alongside the package name, to disambiguate when two vendors wish to vend a package with the same name.

From the specification:

3.6. Package name resolution

Each external package is uniquely identified by the canonical URL of its source code.

I agree that packages need to be properly namespaced in order to disambiguate between packages with the same name. Our approach uses URLs, which uniquely identifies packages and confers a host of additional benefits, including global addressability (you can copy-paste a URL into your browser), security (you can authenticate via existing TLS certificate infrastructure), and naming authority (we can delegate any ownership claims to ICANN and domain registrars).

@StanTwinB

Package signing

As noted earlier, signing is an important part of vending packages that consumers can verify and trust.

As discussed in this forum post, we believe that a checksum database / transparent log — in concert with the aforementioned TLS certificate infrastructure — can provide better security guarantees than digital signatures.

@StanTwinB

Package publishing

While pull-based publishing may work reasonably well for publicly-accessible, unauthenticated source code repositories where both the package registry and the end users can consult the original source code if necessary, it does not work well when the source code repository is private.

I don't necessarily agree with this assessment on a few points:

@StanTwinB

• Users cannot easily determine if the service imported the files correctly (e.g. via checksum validation), especially if the service is packaging them into a single archive as part of the process.

Any user with source access can verify the build product of the registry by running swift package archive on their project at the corresponding commit reference and comparing the resulting Zip archive.

@StanTwinB

• Users have no opportunity to sign the package prior to publication.

As discussed above, our security model doesn't require digital signatures. Though, for what it's worth, users can GPG sign the Git commit used to generate the source archive, and that commit ID will be included as a comment in the generated Zip file:

$ git rev-parse HEAD
b7c37c81f164e5dce0f64e3d75c79a48fb1fe00b3

$ swift package archive -o LinkedList-1.2.3.zip
Generated LinkedList-1.2.3.zip

$ zipnote LinkedList-1.2.3.zip | grep "@ (zip file comment below this line)" -A 1 | tail -n 1
b7c37c81f164e5dce0f64e3d75c79a48fb1fe00b3 

$ git verify-commit b7c37c81f164e5dce0f64e3d75c79a48fb1fe00b3
gpg: Signature made Tue Dec 16 00:00:00 2020 PST
gpg:                using RSA key BFAA7114B920808AA4365C203C5C1CF
gpg: Good signature from "Mona Lisa Octocat <mona@noreply.github.com>" [ultimate]

@StanTwinB

• Users will not always be willing or able to allow network access to their source repository from the service.
• Users will not always be willing or able to grant authorization to the service to read from the source repository.

Under these conditions, a registry may not be the most appropriate mechanism for publishing packages. Instead, a better option might be to distribute binary frameworks.

I would also note that the publishing model described by this specification doesn't preclude a registry from publishing packages out-of-band through another process. For example, if GitHub wanted to backfill support to existing packages, they might offer a tool that automatically creates package releases from existing semantic version tags.

@StanTwinB

Some additional supporting information in favor of using a HTTP publishing based registry rather than pull based git solution. A few other package registries (Ruby, Rust) that began with a git based registries have experienced challenges that have led them to pursue HTTP instead.

Thanks for sharing that. However, I'm not sure how this relates to your earlier point or this proposal specifically.

My understanding of the linked Rust RFC is that it's related to the problem of the crates.io registry using Git as a blockchain (which is similar to the situation CocoaPods faced with its Specs repository until it provided a trunk).

To be clear: this proposal indeed follows the HTTP-based publishing model advocated by this RFC. A client doesn't need to interact with Git in order to publish a new package. Rather, the registry pulls the source code from the repository specified by the client (which may be Git or another VCS) and publishes the source archive.

Are you concerned about the use of Git as a blockchain for a transparency log by the registry? Or are you advocating for the ability for clients to push Zip archives to registries as an alternative to the registry creating them from source?

@StanTwinB

Dependency resolution

When package registries are enabled, Swift Package Manager will first attempt to use package registry API calls to resolve qualifying dependencies, falling back to Git operations if those API calls fail.

Transparently falling back to a git endpoint when the package registry fails seems like unexpected behavior which could result in inconsistent dependency resolution across builds. As a developer I would prefer to have more specific control over where my dependencies come from. (At a minimum, users should be able to disable this fallback behavior.)

Thanks for this feedback. I agree that it would be desirable to disable this fallback behavior.

@StanTwinB

API Versioning

We strongly recommend the URL path include an API versioning component as a prefix for the paths listed in the proposal.

For this proposal, we considered four possible versioning strategies:

• Path : http://api.example.com/v1
• Subdomain : http://api.v1.example.com
• Custom MIME type in Accept header : Accept: application/vnd.example.v1+json
• Custom request header : Accept-version: v1

Each of these options has strengths and weaknesses, but we ultimately decided that the custom MIME type in Accept header offered the best combination of flexibility and ergonomics. It also has worked well in combination with the content negotiation used to upgrade existing package manifests from repository-based dependencies to registry-based dependencies.

@StanTwinB

Are there formal definitions for the metadata endpoints yet, or will the shape of metadata be at the discretion of the implementer? Is there any way we can include source license information in said endpoint definition?

Yes, indeed! Please see this section of the specification:

A server MAY include metadata fields in its package release response.
It is RECOMMENDED that package metadata be represented in JSON-LD
according to a structured data standard. For example, this response using the Schema.org SoftwareSourceCode vocabulary:

{
  "@context": ["http://schema.org/"],
  "@type": "SoftwareSourceCode",
  "name": "LinkedList",
  "description": "One thing links to another.",
  "keywords": ["data-structure", "collection"],
  "version": "1.1.1",
  "codeRepository": "https://github.com/mona/LinkedList",
  "license": "https://www.apache.org/licenses/LICENSE-2.0",
  "programmingLanguage": {
    "@type": "ComputerLanguage",
    "name": "Swift",
    "url": "https://swift.org"
  },
  "author": {
      "@type": "Person",
      "@id": "https://example.com/mona",
      "givenName": "Mona",
      "middleName": "Lisa",
      "familyName": "Octocat"
  }
}