[Accepted with Modifications] SE-0292: Package Registry Service

Hi folks,

The 3rd review of SE-0292 has concluded. The review has been generally quiet with feedback focused on making the OpenAPI spec more robust to explicitly highlight the support for redirecting proxies which have been one of the focus points of the 2nd review. As such, the proposal has been accepted with a few minor revisions:

  1. More clearly state in the proposal motivation or future directions sections that package registry is designed to address the immutability and durability concerns in light of cases like How an irate developer briefly broke JavaScript - The Verge
  2. Refine "A client SHOULD verify the integrity of a downloaded source archive" to "A client MUST verify the integrity of a downloaded source archive"
  3. Include the checksum of the source archive in the Package release metadata, and/or refine OpenAPI spec to make it clear it should include Digest and Content-Length headers.
  4. Make the support for redirecting proxies more explicit by explicitly listing 30x response code in the OpenAPI spec.

The proposal and its approval mark an important step forward in the evolution of the Swift package ecosystem. The proposal went through a long review cycle and reflects important feedback from many members of the community. I would like to thank everyone who participated in the review and the proposal authors for their patience and hard work getting this proposal over the finish line.

Tom Doron,
Review Manager

19 Likes

Just noticed that the entry for this on the Swift Evolution web site is still listed as being under active review.

1 Like

thanks, addressed

Did any work on binary packages end up being done as part of this proposal? I saw there was a note under "Future directions" so I'm assuming not.

I'm doing research right now on archiving binary libraries and doing security related dependency audits using tools like Artifactory. Artifactory doesn't have any support for SPM today but I'm just gathering the current state for stakeholders.

We have some sensitivities around source distribution. So while the work around Github is interesting, we're looking at the future directions of this proposal. Has any work been started or are there any new proposals being worked on?

Has anyone ever implemented a package registry service? I remember GitHub announcing it few years ago, but it seems like the project went silent.

Does Xcode support this?

1 Like

No, there is no support, from Xcode or anyone else. This standard seems dead.

That’s a bummer, as the standard seems really well defined.

I’m dealing with super modularized projects that have 30-50 Swift packages, so resolving the package graph with so many repos takes a while even on M1, not to mention the infamous SPMRepositoryError error 5 on every second attempt.

1 Like

Yeah, it's pretty bad. Hopefully Xcode 14's SPM integration is more reliable, but SPM's performance problems are foundational, so I don't think we'll see any improvements there.

1 Like

I think we can expect it in swift 5.7: make registry dependencies available in 5.7 by tomerd · Pull Request #4244 · apple/swift-package-manager · GitHub.

No idea if Github will implement the proposal as it seem to me that Mattt withdraw at the very end (it took ~2 years to get it in SPM).

Chiming in representing the SSWG and stuff we're aware of here:

@Jon_Shier @Srdan_Rasic this is far from dead actually! :slight_smile: It just takes time to adopt and implement such standards across the board.

Today (!) during swampUP 2022 @tomerd and the Artifactory folks have just announced that Artifactory implemented the registry APIs :partying_face: I could not find a concrete documentation page about it yet, but I'm sure all that will follow very soon.

And SwiftPM has just finished implementing the client-side features necessary for this in preparation for Swift 5.7 (make registry dependencies available in 5.7 by tomerd · Pull Request #4244 · apple/swift-package-manager · GitHub).

Such "ecosystem wide" features take some time to kick off, so don't claim them as as dead while the teams are still very actively working on making them happen and adopted throughout :smile:

13 Likes

That's good to hear. Hopefully we hear something from Apple in two weeks as well.

@daveverwer @finestructure What's the status on Swift Package Index supporting the registry service? I know y'all were a big voice in the proposal's review

1 Like

Adopting a registry API is certainly something we're considering for the Swift Package Index. However, it also comes with quite a few extra responsibilities and so we want to be careful when and how we're going to commit to it.

1 Like

@mattt @tomerd Has the implementation of this proposal actually been shipped? It's listed in the SPM changelog for Swift 5.7, but the package-registry subcommand doesn't seem to be available, even using the latest Swift toolchain. JFrog has recently launched support for SPM within Artifactory, explicitly stating that it's powered by the "addition of registry-based dependency management in Swift 5.7", yet it seems impossible to set it up as a registry with Swift 5.7. Perhaps I'm missing something obvious, but hopefully you can shed some light. Thanks!

2 Likes

thanks for pointing out the tool to edit the registries configuration file is not yet exposed. the tool is helper to manage the configuration file, but you can always manually edit the ~/.swiftpm/configuration/registries.json file, e.g.

❯ cat ~/.swiftpm/configuration/registries.json
{
  "registries" : {
    "[default]" : {
      "url" : "http://localhost:8082/artifactory/api/swift/swift-local-repo"
    }
  },
  "version" : 1
}
3 Likes

Ah that makes sense :slight_smile: Artifactory doesn't seem to be fully compliant with the specs yet (or at least they reject the Accept: application/vnd.swift.registry.v1+json header as far as I can tell), but that's a different problem. Thanks for your help!

I'm looking at Artifactory's SPM product today, and I noticed they're saying that their support can handle Swift Binary Packages:

JFrog now offers the first and only Swift binary package repository, enabling developers to use JFrog Artifactory for resolving Swift dependencies instead of enterprise source control (Git) systems.

However - I'm uploading binary packages to Artifactory and it's not offering them back out through the Swift package endpoint API. I haven't seen a spec for binary packages as part of this review - only a potential spec. Is there any sort of documentation on how a binary dependency should be supplied to a registry? The endpoint API mentions a Package.swift endpoint - but binary archives would not have a Package.swift file?

Hi @Colin_Cornaby this may be a terminology mismatch, JFrog uses the term "binary packages" broadly for archives that contain either source or pre-compiled artifacts, while SwiftPM uses the term "binary packages" only for the latter.

SE-0292 does not cover pre-compiled artifacts (SwiftPM's idea of binary artifact), only source based ones, and as such JFrog does not support them via that API either.

That means that the latest version of Xcode14 is still not supported. The current dependency difficulty is. Using Xcode to add dependencies will get stuck if the proxy is not started, which is very troublesome for users in China. We want Xcode to be the same mechanism as the Swift Package Manager App, with. swiftpm/configuration/registers.json, such as iOS App

@tomerd I'm not sure if this is the right place to have these discussions so apologies in advance, but I figured you might have some insights on the current state of things.

I was able to get this feature to work (both with a local registry server and Artifactory) with pure Swift packages (:tada:), but unfortunately Xcode still seems to lag behind; using 14.2, I'm unable to fetch SPM dependencies from a registry for a PBX project (specifically an iOS app), and I instead get this error message: invalid package type registry <package-id>. I'm guessing the PBX file has its own internal representation of SPM dependencies, and the format doesn't yet support package registries; is that the case? Suggestions or workarounds are very much appreciated :pray: