this problem isn’t new to me, but i’ve recently become more conscious of it now that i’ve started doing work in places that have “normal” internet connections, like my neighborhood Starbucks.
fetching dependencies with SPM takes a really long time, and it seems to disproportionately affect Apple packages like swift-collections.
some of these packages, like swift-syntax are quite large and i am not surprised that they take a very long time to download. but others like swift-collections are tiny (in terms of source code footprint) and still take 21s to download. the only explanation i can think of is that SPM is fetching the entire repo history, because it should not take 21s to download the source code of a package like swift-collections.
Yeah, this was quite a problem for us in the Lottie repo, which is very large (300+ MB).
We worked around this for now by publishing a second lottie-spm repo that is very small since it only contains a URL reference to a prebuilt XCFramework. This adds some overhead to our release process, though, and also means that consumers using this repo no longer have the source code of our package in their workspace (seems inconvenient).
I'm hoping that the upcoming Swift Package Registry lets us avoid this sort of issue, but haven't heard any updates on that in a while.
i always thought it was odd that package registries were under the purview of evolution in the first place, creating such a thing involves setting up and running a service, and that is as much an entrepreneurial pursuit as it is an engineering pursuit.
i am sure the swift team can do anything they set their hearts to, but founding and running a business seems quite incongruous with the typical work of designing a programming language. wouldn’t enabling shallow clones in SPM be a lot easier?
Registries are under evolution insofar as a standard was needed and changes were required for SPM to adopt that standard, which then required public changes to the package format. It seems clear Apple has no desire to build a registry themselves, instead hoping the community would build them instead. Obviously that hasn't happened, aside from support in newer versions of Artifactory (IIRC) for some reason. Whether that's due to community interest or the fact that Xcode still doesn't support registries at all I don't know. Since you're on Linux you could use them directly through SPM if there as a registry anywhere.
This is purely speculation, but the fact that Apple has recently become a sponsor of the Swift Package Index might mean that there's an ongoing partnership with the expectation of SPI eventually becoming a semi-official registry. Of course this is not confirmed by neither party, but I don't think it's too far fetched, especially considering how much of an anomaly it is for Apple to sponsor third party projects.
I think I’m just missing something, but why would a registry help? Because there’s enough information available up front that shallow clones become an option again?
Registries allow for querying for the exact or latest versions and include only the files necessary to build (or the whole repo at the moment, I don't recall) rather than a full git history. Even in the case where SPM can use a shallow clone a registry should still be faster since it doesn't have to call out to an external tool or do any git work.
The registry spec has defined specific endpoints for querying and fetching package manifests (which are needed during package resolution, so potentially for many versions), so SwiftPM won't need to download the actual sources for a version until it has determined the concrete one to use.
And with the package registry, consumers don’t have to interact with a git repo at all, right? So downloading the source for a version wouldn’t require downloading any git history (you just get a zip of the package source)?
as far as i an aware, there has been no progress made and no public indications from swift project leaders of having any plans to address this. i would love to be proven wrong though.
as i understand it, there are actionable things that can be changed in SwiftPM that would alleviate the pain for 99% of users interacting with popular open source libraries:
SwiftPM could special-case GitHub URLs to fetch individual package manifests across versions without cloning the full repo.
SwiftPM could perform a shallow clone of a dependency’s git repo, which would immediately lighten the burden for consumers of packages that whose maintainers have done the work to optimize the repo today but cannot do anything about the repo history, which must remain untouched for a number of reasons.
the first bullet point would only benefit GitHub packages, but these are the most popular packages and also the ones most likely to suffer from the “long and illustrious repo history” problem.
alternatively, Apple could make a serious financial investment in a centralized Swift package registry. as primary maintainer of swiftinit.org, i would be interested in collaborating with Apple on such a project, and perhaps others may be as well. but this requires a non-trivial subsidy to be economically viable, as a package registry is a money-losing endeavor.
They are actionable but not necessarily easy. I have a couple of people helping out and we're just getting up to speed, but we have seen the GitHub issues you are referring to. We just need to figure out the best approach and will be reaching out for guidance once we get there.
But it has been pretty consistent feedback that package resolution needs attention.
