Without identity tied to the URL of the package, there would be no way to tell two packages at different locations with the same name are actually the same package. We could assume that to be the case, but that sounds like a problem waiting to happen. Instead, with the registry, the publisher would simply redirect from one location to the other so all consumers of the package end up in the same ultimate location, no matter the original URL used. That way this deduplication is under publisher control and not the whim of SPM.
And in the case of opaque identifiers, how does SPM decide which package is canonical without some sort of related record? The user shouldn't have to randomly choose one and SPM certainly shouldn't assume.
You need to check out the dependencies at some point, so this resolution should happen then. Even if you're shipping your codebase as tarballs, the resolution of dependencies that happens before then should have already solved any ambiguity.
For this case I think it's best to assume that Soto is now a new package (ie. not aws-sdk-swift). Ie. they'd have to manually change their module names to Soto (which fixes Swift name mangling) and if they have C modules, they'd manually need to change them from old_name_c_function to soto_c_function.
The idea is that if the opaque ID is equal, the packages are considered equal. Much like in almost all languages whose package managers support a registry that I'm aware of (Go being the exception with URIs).
That's what we're doing today. The package name (in Package.swift) and the last component of the URL need to be unique across the whole graph. And today there is no registry which enforces uniqueness.
This always requires an internet connection to all the present and old homes. Also it requires the owner still having an account with all source code hosts, and the source code host must implement some redirecting behaviour.
If the opaque ID is the same, the package is assumed to be the same. SwiftPM will just do the constraint resolution and download the right version from the registry. The registry makes sure there are no duplicates. Very much like it works in Cocoapods, Rust's crates.io, NPM, Haskell's Hackage, Python's PyPI, ...
With opaque IDs, the user who moves a package tells the registry that the package moved. Mirroring a registry merely means to download everything from a registry. Very straightforward
With URIs on the other hand, either the registries themselves or SwiftPM need to constantly fetch all the URIs and check if any of them redirect to another place that we already know (finding dupes). This is quite troublesome because these things can change all the time without the registry being able to police that. Also what do you want to do if a source host is currently unavailable? Or what if somebody maliciously configured a package cycle or changes the redirects all the time?
Also: If you let the registries to the deduping, then you'll trust the registries that they have given you the correct "identity database" (otherwise the registry could just say "https://github.com/apple/swift-nio is now https://malicio.us/foo/swift-nio). If you let SwiftPM do it, then everything will be very slow because it has to follow a lot links to find potential redirects. And you need internet access for that.
Do the other language support random registries, like Swift plans to? I'm not sure I see how you can combine opaque identifiers deduped by registries with multiple, non-federated registries and expect dependency integrity. This only works for say, CocoaPods, because there's only a single, central registry which serves as the source of truth for all published packages. If that was the plan for Swift, opaque identifiers would be easy, but it's not.
I thought we were discussing this subject in order to improve the status quo. Unless you're arguing that's good to be deduplicating this way (it seems like a problem waiting to happen to me), I'm not sure what your point is here.
No it doesn't. It requires the same internet connection used to check out dependencies directly to instead go through the registry.
Wait, I thought you didn't want the registry to serve that purpose? And doesn't this require an internet connection?
I don't think this is correct. Just like in your opaque identifier case, the author of the package published in the registry would tell the registry that things have moved. You seem to be under the impression that using URLs as identifiers somehow expects or enables all of this forwarding behavior without a registry. I don't think it does.
I'm pretty sure we've all be saying the registries will do the deduplicating this whole time, so I'm not sure what you're talking about here. But you do seem to be forgetting one big point: Swift intends to support an arbitrary number of registries, none of which have any claim to being an authoritative source of truth as to the identity of packages. Instead, that capability must be delegated to the package authors. However, if we use opaque identifiers for packages, it becomes trivial for different registries to have different sources for the same package. URLs, on the other hand, already have a notion of truth built in, as domains and paths are uniquely claimed. That is, if Registry1 and Registry2 both claim to have a package from https://github.com/apple/swift-nio, we know that must be the same package since it's not possible for two people to claim github.com and have code at that path. Thus the authoritative source of the package is established. With opaque identifiers this becomes impossible, and would force package authors to seek out and publish on every registry they can find for fear of name squatting or intentional collision.
To my mind, many of these issues would be solved by a single, centralized registry where publishers can establish identity and control their identifiers. However, that doesn't seem to be something Apple's willing to do. Nor does Apple want to establish even some federated authority. Instead, we have registries which anyone build and host as long as they conform to the spec but which have no claim to identity. So the standard must provide some other way to establish identity that protects against squatting and collision. So far, URLs seem to be the only option which give the community any way to do that.
Of course. All of the JVM ecosystem uses reverse-dns identifiers (e.g. com.apple.awesome + awesome-package), this includes all languages which run on the JVM which have adopted a similar style of registries and dependency management.
Repositories are free to be hosted by anyone, and organizations frequently do so. If you want to you can run your own and shadow everything else the public internet has to offer by "my copy of it." You can also chose to ignore the default repositories, and only use packages you have vetted and trust, no public internet package can sneak in then. In fact, many large companies do just that -- have one internal mirror of everything (e.g. using artifactory or sonatype nexus).
Order of preference is simple, whichever "first" repository has the artifact you're looking for "wins". This is a feature it allows companies to not fetch from external repos, but use internal ones which can be way faster.
// edit: removed NuGet note since I'm not 100% sure on the federation model.
As I understand it, there are no standard registries for many of those languages. Rather, software like Artifactory serves as artifact storage which can be fetched through various dependency managers like Gradle and don't care to establish the identity of the packages they fetch any further than required for dependency resolution. Now, if that's all the identity we're talking about here that's a different conversation, but I thought we were solving more than that.
Additionally, these services aren't really federated, as there's no single authority at the top of the stack. You can connect an Artifactory instance to any number of other services and it can act as a proxy, and many of those services are an authoritative source like CocoaPods or RubyGems, but that's a feature of Artifactory, not the dependency systems these languages use.
In any case, it still seems like those systems would suffer the same issues with opaque identifiers to some degree. If we don't care to solve those issues, that's fine, but it seems like we have an opportunity to do better.
What is those languages in this sentence? JVM languages? E.g. java, scala, kotlin, clojure etc?
There very much is a blessed default "standard" repository that pretty much all of them use: maven central. Even package managers which are not maven (gradle, ivy, sbt, lein, ...), still default to using maven central and resolve it because it is simply the good default to go to. There's alternatives, such as jcentral but usually to use those you'd add a single line to your package definition to use it in addition to or instred of maven central.
The same would be true here I imagine, we'd have some default related to swift.org I guess, and others can be configured.
I've not tracked this thread in depth, but wanted to chime in to clear up this confusion about such registries in the real world -- yes they exist and work just like that.
This really doesn't seem to be likely, given everything that's been discussed around the registry proposal and the fact that Mattt's been working with GitHub to define it. At best it seems that Swift can expect GitHub to host an integrated version of a Swift registry (if it's accepted), but I'm not sure that would play the same role as Maven Central. If Apple is planning on an official Swift registry they should announce that fact now, as it effects many of the decisions we're talking about. With a centralized registry at the "top" of the stack, opaque identifiers become much more trustable. Without one, which seems likely, Swift is in a different situation than the other languages.
To be clear, I don't think it matters if there would be a swift one or not, nor do I know if there will be.
What matters is that it is likely that one or some repositories are likely to be "default", be it the github one, a swift one or some other community maintained one (e.g. "Sundel's favorite packages" ).
I'm just drawing the parallel/potential to maven central at some default existing, while the entire system is completely distributed. If you don't like maven central, you can not use it at all, as is true with repositories here as well.
Pretty much, since it could be that "good default".
There is no "top" in the model at all. Everything is completely and fully distributed and repositories are "equal", other than how the user decides to configure them (e.g. order in configs might matter). There are "good defaults" and it does not mean that it is in any way "the one and only authoritative source of identifiers".
It seems odd to me to describe a system with a de facto authority as “fully distributed”, but whatever. I don’t think allowing an authority to organically arrive at some point in the future is a good idea. The registry proposal doesn’t envision such a future, GitHub doesn’t seem to want such a future, and the community won’t be best served by such a future.
Sure it is - a shared default does not mean a system isn't distributed/decentralized, because you can always choose to not use the default and use whatever you want instead.
This is in the same way git is distributed: even though most people use github as their upstream, nothing mandates doing so. And at any point, without contacting github, you may decide to move upstream somewhere else and just start doing that.
Anyway, we derailed from the core discussion so let's get back to that
@johannesweiss@tachyonics I think you both identify some important points. However, we appear to be reaching different conclusions, and I'm not sure if it's due to disagreement or misunderstanding on one or more points.
To help identify where we disagree, here's my understanding of your concerns and my responses to them. Please tell me if I've mischaracterized your positions or if you have any objections / questions to my points:
Swift Package Manager lets you override the location of packages with mirrors (SE-0219). This allows developers to point external dependencies to forks across the dependency graph. As you've identified, the trade-off is that it introduces context for understanding where a dependency is located.
Our proposal associates SHA256 checksums to versioned source archives in Package.resolved. If you do a fresh build of your project (such as in a CI) and a host tries to send you something else, Swift Package Manager would raise an error. This is an improvement over current behavior, which can't protect against external dependency code changing over time (whether because it's hosted elsewhere, or the version was re-tagged in the original repository).
Package.resolved provides a Trust on first use (TOFU) security model that can offer strong guarantees about the integrity of dependencies over time. A registry can further improve on this model by implementing a transparent log or some comparable, tamper-proof system for associating artifacts with valid checksums.
As I said in a previous reply, tooling — whether that's a new subcommand or making the output of build and resolve more verbose — could go a long way to helping us understand what's being downloaded from where.
"What if I'm on a corporate network and can't access the Internet?"
We've pitched an enhancement that allows users to configure an intermediate registry proxy. Whereas a mirror lets you redirect a single dependency, an intermediate proxy would route all packages directly. A corporate CI server, for example, could configure an intermediate proxy to an internal package registry to serve cached versions of external packages.
If you're concerned about downloading code at all as part of the build process, the alternative is to vendor all of your external dependencies. This is currently a manual process, but we could add functionality to do this automatically (for example, a swift package vendor subcommand that resolves and downloads dependencies to Vendor/Sources/ and rewrites Package.swift to declare dependencies using local paths).
"Why github.com/mona/LinkedList and not 'mona/Linked, hosted on GitHub?'"
It's tempting to treat GitHub and GitLab as interchangeable, but they're not. To use an analogy: we can't assume mona/LinkedList is the same on GitHub and GitLab, much the same as we can't assume that @mona on Twitter is the same person on Instagram. (To extend the analogy: our proposal is more like Mastodon than Twitter)
Packaging systems in other languages demonstrate that there's more than one way to solve the problem of external dependency resolution. A lot of things would be easier if all identifiers went through a central registry, but that's not how things work currently, and I don't think that's how things should work going forward. The decisions we made in our proposal were informed by facts specific to Swift and Swift Package Manager; given an alternative set of requirements, we might make different decisions.
I am arguing that this is the best option proposed so far, and also good enough. If we all came to the conclusion that this is indeed the case and we proceeded with opaque ID, then there's a separate discussion to have: How are we assigning the opaque IDs. Can anybody register any string? Do they need to start with a reverse DNS that you need to prove you have when registering? Something else? But that's entirely a second step in my mind. The first discussion to have is whether the location of the code and the package ID are linked. In my mind they should not be linked.
AFAIU, the code for all the dependencies can get downloaded from the registry (so we don't need to git clone anymore). And you can mirror that registry into your corp network.
No, at build time, SwiftPM knows (without internet) that opaque ID 1 and opaque ID 2 are the same package when their byte representation is equal. The registry would just make sure that there is of course only one package registered with a given name (sure, two registries could in theory have the same package ID). Again, must like Rust's crates.io (package name is tokio and Crates uses https://crates.io/crates/tokio if you want more information about it), Haskell's stackage (package name for example is bytestring and Hackage uses https://hackage.haskell.org/package/bytestring if you want more info).
You would have your internal mirrors and internal registry that always gives you all the code from inside of your corp network.
Isn't the main claimed advantage of the URL the "trust model" it gives you (I don't buy there's actually any more trust but we discussed that before). I'll quote @mattt :
And the URI-based proposal is not supposed to make "moving packages around" easy:
What you @Jon_Shier seem to be pitching is essentially opaque IDs where the opaque ID happens to be in the form of a URL. With one addition: That registries can also hold aliases. Is that correct?
Personally, at this stage I don't think it's helpful to discuss the shape of the opaque package ID (I think we should first decide that we want opaque package IDs). What I mean with that is that I personally don't mind if the SwiftNIO is called:
swift-nio, as in Rust (Crates), Haskell (Hackage/Stackage), JavaScript NPM, ...
com.apple.swift-nio, as I think in the JVM ecosystem (with initial proof that you own apple.com)
afb757d66ed2e63824e8aff4984d6f97c43d85ea (just a hash)
https://github.com/apple/swift-nio
What I do care about is that the package name is comparable by a simple string comparison. If we agreed on that, then I'd be very happy. Of course, I also don't think that URIs make good opaque IDs because location may change (and the ID now can't) and people may think it's a URI and not an opaque ID (so they may assume that https://github.com/apple/swift-nio == https://github.com/apple/swift-nio/ or so).
Isn't this a counter example? Assume your package graph contains both https://gitHub.com/apple/swift-nio and https://gitLab.com/apple/swift-nio. I think we agree that SwiftPM needs to de-dupe and find the "correct" SwiftNIO. However,
Registry 1 claims: I have a package, canonically hosted at https://gitHub.com/apple/swift-nio with an old alias https://gitLab.com/apple/swift-nio.
Registry 2 claims: I have a package, canonically hosted at https://gitLab.com/apple/swift-nio with an old alias https://gitHub.com/apple/swift-nio.
Registry 3 claims: I have a package, canonically hosted at https://gitHub.com/apple/swift-nio, no aliases
Registry 4 claims: I have a package, canonically hosted at https://gitLab.com/apple/swift-nio
So two registries agree that SwiftNIO has 2 package IDs but they disagree which one is the canonical one. The other two registries see SwiftNIO at only one place. Similar situations can even happen with non-malicious registries if for example one registry can't reach (or doesn't properly implement) the "forwarding". It's a distributed system of mutually distrusting parties, we can't easily reach one definitely correct view of the world (there's a blockchain joke somewhere in here).
What I'm saying is that the trust model is more or less "I trust the registry" (if I have multiple, I probably need an order how much I trust them) in both the URI-based and the opaque ID system. The opaque ID system is however much simpler and solves all the common cases.
I don't think they do. The one thing that URLs can give you is that when you first register, you prove that your code is hosted at a certain URL. Opaque IDs can do a very similar thing if we (at a later stage) decide that to register an opaque ID, you need to prove ownership of a certain domain/website/... . That's I believe what they do in the JVM ecosystem, I think you can only register com.corp.package if you provably own package.corp.comat the time of registration of the package. Conveniently, Github would then allow everybody (through Github pages) to claim io.github.user-name without having to pay money for a domain etc. The crucial thing is that there is only ever one ID that is immutable and doesn't change.
That's great. But it's not related to URI vs. opaque IDs though, right? And once you swift pacakge update your packages, there's again no more trust than what "opaque IDs+trusted registries" gives you too.
The same would apply if we use opaque IDs, right? In both, the URI and the opaque ID case, we usually download the code from the registry. In both cases can also get information about the source host and download it manually directly from there.
Right, which adds extra complexity and I fail to see the benefit. We also need to trust these intermediate registry proxies.
@johannesweiss I really appreciate your feedback, but I'm having trouble understanding some of your concerns, which don't seem to relate to what we're proposing. I think there's a misunderstanding that's causing us to talk past one another...
Based on your reply to @Jon_Shier, I think one point of misunderstanding may be that our proposal doesn't involve a list of registries like in Maven or other systems.
Where are registries #3 and #4 coming from in your hypothetical?
Here's how I understand this scenario to play out according to our proposal:
A package graph contains both github.com/apple/swift-nio () and gitlab.com/apple/swift-nio ().
Swift Package Manager sends a HEAD request to github.com/apple/swift-nio (), and the server responds with a redirect to the registry interface at swift.pkg.github.com/github.com/apple/swift-nio ()
Swift Package Manager sends a GET request to swift.pkg.github.com/github.com/apple/swift-nio (), which returns a package registry response that says gitlab.com/apple/swift-nio () is the canonical URL for the package.
Swift Package Manager now knows that it can relabel any dependency nodes in its resolution graph with github.com/apple/swift-nio () to instead be gitlab.com/apple/swift-nio ().
So long as there isn't a circular reference, where gitlab.com/apple/swift-nio () points to github.com/apple/swift-nio () as the canonical version, then there's no problem. There could be any number of registry interfaces out there (corporate networks, personal domains, mirrors, malicious websites), but those don't matter — we don't consult them.
Would it help to describe this proposal as an "alternative download interface" instead of using the word "registry"?
How would swift.pkg.github.com know about the new package URL at gitlab.com though? And what if gitlab.com has their own registry (e.g., swift.pkg.gitlab.com)?
I do agree with this; in the absence of any other information github.com/mona/LinkedList and giblab.com/mona/LinkedList should be treated as unrelated packages (and indeed an internal corp registry should happily be able to mirror both and resolve package graphs accordingly). But we were discussing the capability for a registry to be able to explicitly declare they are equivalent packages and dupe them in a package graph.
Essentially this does feel like what is being proposed where the identifier of a package is a canonical URI of the form 'https://home-registry/registry-specific-package-identifier'.
I'd prefer the package identifier not to be a URI for these points however ultimately I think the proposal is workable. I was primarily proposing using something like github.com/apple/swift-nio or apple/swift-nio, homed on github.com as the package identifier rather than https://github.com/apple/swift-nio (so specifically not a location/URI but carrying the same information) as opposed to huge fundamental differences to what has been proposed.
Yes, I totally did. I do apologise. I did now re-read the proposal and I can confirm that it doesn't mention that you can configure additional registries (which I was just assuming to be true, apologies). However, it also doesn't mention that you cannot add registries. Which especially given that it mentions search (at the very bottom) I find surprising.
I think that would help. Also would it help to explicitly spell out "No, you can't configure your own registries."
Knowing what I know now, I think I have much graver concerns.
Assumptions:
As a user, I cannot manually configure registries.
To benefit from the package "registry" feature, the selected git host for a certain package will ALSO need to (correctly) implement the SwiftPM registry protocol.
If the above is correct, then I have a bunch of questions:
How does the bootstrapping work? I understand that Github was involved and are willing to implement this. What are the timelines? What do we do before Github has implemented this?
How does one enable registry support in Github? Is it on or off by default?
Once Github has implemented the feature, how do we get bugfixes if they mis-implemented something? Will the code be open-source at least?
What about the people who don't have (or can't have) a Github account?
Are we expecting people that use say Gitlab to also create a repository on Github just so their package benefits from the registry feature?
If it is the case that to start with everybody should have a Github repository to benefit from this feature, do the package authors now also need to convince all of their users to enter https://github.com/my-org/my-package-just-here-for-the-registry despite the fact that they're actually hosted at https://my-skunk-works-git-host.com/swifty-awesome/swifty-awesome?
What if Github or another git host decommissions its SwiftPM package service for some reason? Are we then expecting everybody to move elsewhere? If yes, how does the forwarding now work given that the registry service was just shut down?
Your first assumption is incorrect, and that's where the intermediate registry comes in.
By default, Swift Package Manager will try to connect to package source hosts using the registry interface, falling back to Git if that's unavailable (and the user hasn't disabled this fallback behavior). If a mirror is configured for that package, SPM does the same thing but with the mirrored URL.
If a user specifies an intermediate registry (swift package config set-registry-proxy https://internal.example.com/), Swift Package Manager will only consult that configured URL through the registry interface.
If you have a moment, please take a look at this test harness I set up, which lets you try out the registry interface yourself today. It uses mirrors instead of a intermediate registry, but the effect is the same.
I like an escalator, because it can never break. It can only becomes stairs.
If the registry interface is available, Swift Package Manager will use that. If not, it can fall back on how things work right now. Our proposal is designed so that users can benefit from stronger security guarantees and better performance with little to no changes in their package manifest.
Let me know if you think the questions you raised here are still valid given this change in assumptions, and I'll do my best to address them.
Opaque package identifiers are preferred...Using opaque identifiers implies location is separate from package identity.
As someone who works on artifact registry service provider (AWS CodeArtifact), I agree that it is important to separate package location and package identity.
A section that details how the registries configuration works.
Yes, if SwiftPM extracts the package registry host based on the package URL then there needs to be an alternative configuration mechanism to use a registry at a different location, such as an Artifactory or CodeArtifact registry that isn’t at the same host name as the git repository. Specifically, our customers would request a way to configure SwiftPM to fetch all packages from the configured registries.
).
) and 
).
)