GitHub / Swift Package Management Service

Development Package Manager
1

Hello!

I head up the Package Registry project at GitHub and we're putting together a pitch for how the package registry service will work. I'd love to gather your ideas and concerns so I can incorporate those areas into the coming pitch.

Our goal is to present a package management service specification which could be implemented by anyone, not only GitHub. This spec will include necessary API endpoints, package naming (URL spec), versioning system, and other elements required or in addition to the basics. We plan to open source our implementation (written in Go) such that others can contribute back to it and progress can be made in the open.

We've learned from Rust, Go, and many others when thinking about package management services; and we intend to take the best of those worlds into this process.

What are your thoughts here?

Thanks!

33 Likes
2

I’m not sure I understand your question. (Though the likes you’ve received from the lead package manager developers indicate they understood the question. So maybe I’m just too tired to think straight.)

I don’t follow who you mean you are pitching to. Are you are pitching a GitHub idea to Swift? Are you pitching a Swift idea to GitHub? Or do you mean something entirely different?

The first quote sounds like you want to make an indexed database of packages. The second quote sounds more like you want to reinvent the Swift Package Manager.

If you meant the first, then don’t you just have to parse any GitHub repository that has a Package.swift in it and add that information to your database? But that would seem so straightforward, I’m not seeing why you need our thoughts or where the new specification you mention comes in.

If you meant the second, then... why exactly do you want to do that? As the Swift Package Manager catches up, many developers have been looking forward to retiring the Objective C holdovers of Carthage and CocoaPods, because they are tired of maintaining three separate package descriptions. You won’t see much excitement about a fourth redundant description being added to the mix.

So what are you actually hoping to do? And what are you actually trying to ask this community?

3

Can you elaborate a bit more about what you have in mind?

Since SwiftPM doesn’t yet support something like an index or centralized service, it is not very clear to me what is being proposed. Are proposing to implement a generic index service and then (for example) contribute SwiftPM support to use it? Or are you proposing to define SwiftPM specific APIs for use by the latter (which would then also need an implemention contributed)?

  • Daniel
4

It seems pretty clear to me that he’s asking for Swift community input regarding GitHub’s efforts to standardize a package registry service design and API, one of which will be the actual GitHub Package Registry.

Sounds great to me, since it certainly seems like Apple’s never going to do anything about this need, and there have often been claims that, should a registry design spontaneously arise, it might be considered.

9 Likes
5

What are your thoughts here?

I think this sounds fantastic and is a big unmet need in the Swift ecosystem. Only yesterday we were discussing this at https://www.serversideswift.info

Do you have any initial design/specification proposals that could be used as a basis for discussion? Or is the effort in the very early stages?

2 Likes
6

Thanks for stepping forward, Bryan: having GitHub involved in a package registry seems like a really fantastic addition to the Swift community!

In particular, I’m really pleased to see that GitHub is invested in the idea of registries being abstracted away by a web API. This is a direction many other language communities have taken and it allows for a lot of flexibility. For example, it lets enterprises run “internal” package registries that let their developers use packages that haven’t been open sourced. Or, my personal favourite, allows developers to run a daemon on their machine that can act as a caching proxy for the registry, allowing a full SwiftPM workflow even in a situation where they have no network access.

Below are some of my initial thoughts in response to your question. I’m looking forward to helping this process move forward!

My understanding of the shape of your pitch is that you consider a package registry to be a combination of the following things:

  1. A package versioning, dependency, and metadata specification that allows packages to express these properties about themselves.
  2. An API specification that allows a client to query a service about packages, in order to locate their source code or other artefacts.
  3. An implementation of 2, where one example will be provided by GitHub.

My view here is that the Swift package manager provides part (1), so my recommendation would be that the GitHub registry should follow the lead of many other package registries in extracting this information from the language-specific package manager manifest. In the case of Python, for example, pip and PyPI are built on top of the metadata specified by setuptools.

In cases where GitHub or the community identify a need for extra metadata, that metadata could either be specified in the Package.swift object itself, or in a static file alongside the Package.swift. In general I’d prefer to keep this stuff together, but I understand the desire to be able to evolve the metadata without evolving the build tool, so in the cases where this metadata does not affect the build process (e.g. license specifications) it seems reasonable to have a declarative file alongside to carry this metadata.

For (2), I think it would be most productive for you to prepare a sacrificial first draft of what you think this API should look like. This would let the community take advantage of your experience running other registries to get an idea of what a general shape for an ergonomic API would be, and then to provide concrete feedback to evolve that proposal. Otherwise the design space for possible APIs is huge and we’ll be at risk of discussing at cross-purposes forever.

A sad note on part (3) is that it’s unfortunate that the initial implementation will use Go, and not Swift. Of course I understand that you have your own operational realities to deal with, but still, a Swift implementation would be nice, and I expect the Swift server community will rapidly spin up several, if only to satisfy the needs of local resolver caching and other things.

12 Likes
7

Exactly! A good service should get all the information from the clients either at publish time or in the source control files. There is a lot to the Swift package manager already that handles versioning and that specification would be reflected in the service.

I'll have the spec soon. I wanted to give a heads up that its coming and gather any initial ideas so we can cover ground quickly.

First, I've learned already that we need to clarify the split of responsibilities between the service and the client. The Swift package manager already does the versioning and handles metadata, what GitHub is interested in is extending the publish mechanism into a new service with new APIs. Right now publishing and fetching happens through Git which has many drawbacks, I'll go into these here and in the specs.

The service API will look something like this:

HTTP PUT package@version (publish)
HTTP GET package/ (list versions)
HTTP GET package@version (download)
HTTP DELETE package@version (un-publish; when possible)

The API is designed to be simple; for implementation and usage. Using web APIs (compared to Git access) means we can put this data behind CDNs which will allow for much faster downloads from anywhere in the world. This will decrease overall build times and improve cache rates.

We'll also be working on a set of GitHub Actions (CI/CD and workflows) that help to automate publishing. Actions allow you to easily trigger a publish from a git tag or other events in the source code. All publish workflows within Actions would continue to take place through the Swift package manager but it simplifies the tedious processes.

And yes, apologies about the choice of language. We have our own requirements for running this system at scale. However with a simple API to follow it'll be easy to write the service in any language.

1 Like
8

To clarify a bit, the goal here isn't to create a centralized service but a spec for a decentralized service. Ours will be the initial implementation of this spec. We've taken a page from Go and Rust in that package identifiers should use URLs such that multiple services could be hosting Swift packages and authors know where their packages are coming from.

github.com/clarkbw/swifty
swift.pkg.github.com/clarkbw/swifty

The GitHub search APIs will be available for Swift packages and we are working to figure out how we improve the search for a better user experience on the website and locally.

The overall goal is to offer a registry service that has greater value than using git directly. This means a search index, greater security, simple APIs, faster build/download speeds, and better caching.

9

Thanks for taking this on, @clarkbw! I look forward to your spec pitch.

A simple API like this sounds great. I just have two pieces of early feedback which we can discuss when you post the actual pitch for the spec.

  • SwiftPM (and perhaps most package managers) requires information from the Package.swift manifest during dependency resolution. This is more than just the list of dependencies since SwiftPM's dependency resolution also looks at the tools version and targets (with SE-0226) in the manifest. I was wondering if the list versions API should include a hash of the package manifest at that version. We can then have an endpoint that returns the package manifest at a given hash. This will allow SwiftPM to separately download the package manifest during dependency resolution without having to download all of the sources. The hash will also allow SwiftPM to easily cache the package manifest since they usually don't change that often between the versions.

  • Some of the Apple folks recently discussed potentials ways of eliminating the various types of naming conflicts that we have in the Swift ecosystem (@johannesweiss have you posted a document somewhere already?). One of those conflict is the package name. SwiftPM currently doesn't allow having two packages with the same name because we need some way of uniquely identifying a package. The package URL isn't a good option because there are various ways to spell the URLs (https/ssh/scp) and it also breaks down when you're mirroring packages. We've been thinking of solving this by asking the package to specify a tuple of (package-name, reverse-domain identifier) to uniquely identify it. None of this is actually implemented or discussed in detail yet but I wanted to mention it so we keep some ability to enhance the spec to accommodate something like this.

3 Likes
10

I'm guessing this will be part of https://github.com/features/package-registry and is what was sort-of-announced at WWDC?

First I just want to say yes please! I've been hopeful about how this will improve the Swift ecosystem and I'm happy to hear the project's moving forward!

A couple questions

  1. Glancing through some of the documentation it appears the packages are made available directly to the language's package manager. Is the intention to do something similar with SwiftPM? e.g. support for .package(url: "https://swiftpm.pkg.github.com/OWNER/PACKAGE") as a Package.swift dependency.
  2. What information will be needed by a registry that is not currently supported in the SwiftPM Package.Swift manifest? IMO it would be a shame to already standardize requiring additional configuration files. Ideally the needed specification could be officially supported by the manifest, or if that's unrealistic support generic metadata (assuming that's not already possible?) that package repositories can use so that all package configuration is in one place.
1 Like
11

That's correct. We also handle multiple packages within a repo namespace. When the repo and package are the same name you can do https://swiftpm.pkg.github.com/OWNER/PACKAGE for short which is the same as https://swiftpm.pkg.github.com/OWNER/PACKAGE/PACKAGE and https://swiftpm.pkg.github.com/OWNER/PACKAGE/OTHER_PACKAGE for accessing any other package published within that repo.

If there are additional fields required, and I don't believe there are any right now, I'd like them to be incorporated into the manifest file. I believe its best to have it all within that one file. Our service will parse that file and make it available in the package information and search / navigation.

1 Like
12

Now that it is clearer what you are talking about doing, it sounds like a good thing.

Originally I had volunteered for the limited Vita when I saw the blog announcement about the coming support for Swift packages. But Swift packages didn’t turn out to be included yet. And as soon as I was in it gave me instructions about commands to set up and publish a package. That made no sense to me, since the package was already configured, published and fully functional simply by nature of being a public repository. I couldn’t really fathom what this new packages feature was for, so I just shrugged my shoulders and left it there to see if it might one day parse the Package.swift and start using the metadata to derive information under Insights or something like that.

But now that I understand the aim is to optimized the interaction between SwiftPM and GitHub by providing a faster alternative to the git protocol for use during fetching and resolution, I am excited about it again.

There are a lot of pieces of the package manager that are heavily integrated with Git: branch and revision dependencies, edit mode, submodules, etc. It would be a shame if “publishing” a package were to interfere with any of these capabilities. It should still be possible to put a dependency into edit mode and then switch it to another branch. I would hope that integration between SwiftPM and GitHub could be done in a way that the user doesn’t have to know the difference. When using functionality that has GitHub optimizations, it comes for free without the user needing to opt in; and when using edge cases or new features not handled by the GitHub API, it would fall seamlessly back to Git. Some examples of what I mean include:

  • If a “published” package has a particular version, then its repository also has the same version tagged in Git. If a version is tagged in Git, it is automatically available through the as “published” version as well (though maybe only after a repository‐wide opt‐in that allows GitHub to enforce the no deletion rule). Cloning and forking, pushing and pulling tags and so on would then also preserve or modify the version list, just as it does with Git.
  • The same URL (or identifier) in a package manifest is able to reference arbitrary Git revisions and “published” versions, and to seamlessly go back and forth.

By this I hope you mean other kinds of packages besides for SwiftPM. Otherwise, at least until SwiftPM thinks that way natively, having more than one SwiftPM package in a repository would cause mayhem in cases where SwiftPM tried to fall back to Git.

1 Like
13

Yes, thanks for this important point! I agree this should be supported and I think we can do this. We'll be capturing commit information for packages published which should mean we deliver the correct branch to a client for working on the code that built the package. Perhaps I'm being hand wavy here with the client responsibilities but the goal is to allow for this kind of behaviour.

Good point! This is mostly for other types such that you can have a Docker image and other packages also associated.

2 Likes
14

I agree it'd be a shame to introduce a separate manifest file just to carry extra meta data about a package. I think the Package structure can be changed to accomodate additional arbitrary data. It's actually already possible today:

// swift-tools-version:5.1
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    ...
)

extension Package {
  var authors: [String] { ["Johnny Appleseed <jappleseed@some-corp.com>"] }
  var description: String { "Lorem ipsum..." }
}
2 Likes
15

That has no more effect than a comment though. The extension cannot change Package’s conformance to Codable. (And if it could, you’d break its compatibility with SwiftPM.)

16

True. It wouldn't make its way to the output of swift package dump-package as is, but I suppose this Package.swift could be interpreted similarly to how libSwiftPM does it and have it spit out these additional pieces of meta data as well.

Obviously this is rather complex, and it'd be best to have PackageDescription support passing in an arbitrary Codable in support of package publication.