Why is fetching dependencies with SwiftPM so slow?

this problem isn’t new to me, but i’ve recently become more conscious of it now that i’ve started doing work in places that have “normal” internet connections, like my neighborhood Starbucks.

fetching dependencies with SPM takes a really long time, and it seems to disproportionately affect Apple packages like swift-collections.

Fetching https://github.com/apple/swift-markdown
Fetching https://github.com/apple/swift-collections.git
Fetching https://github.com/apple/swift-cmark.git
Fetching https://github.com/apple/swift-atomics.git
Fetched https://github.com/apple/swift-markdown (21.79s)
Fetched https://github.com/apple/swift-cmark.git (21.79s)
Fetching https://github.com/apple/swift-system.git
Fetched https://github.com/apple/swift-collections.git (21.81s)
Fetching https://github.com/apple/swift-nio-http2
Fetched https://github.com/apple/swift-atomics.git (3.44s)
Fetching https://github.com/tayloraswift/swift-grammar
Fetched https://github.com/apple/swift-system.git (3.43s)
Fetching https://github.com/tayloraswift/swift-hash
Fetched https://github.com/tayloraswift/swift-hash (1.09s)
Fetching https://github.com/swift-server/swift-backtrace
Fetched https://github.com/tayloraswift/swift-grammar (1.18s)
Fetching https://github.com/apple/swift-nio.git
Fetched https://github.com/swift-server/swift-backtrace (1.56s)
Fetching https://github.com/apple/swift-syntax
Fetched https://github.com/apple/swift-nio-http2 (30.78s)
Fetching https://github.com/apple/swift-nio-ssl.git
Fetched https://github.com/apple/swift-nio-ssl.git (44.06s)
Fetching https://github.com/tayloraswift/swift-mongodb
Fetched https://github.com/tayloraswift/swift-mongodb (7.99s)
Fetched https://github.com/apple/swift-syntax (103.10s)
Fetched https://github.com/apple/swift-nio.git (155.72s)

some of these packages, like swift-syntax are quite large and i am not surprised that they take a very long time to download. but others like swift-collections are tiny (in terms of source code footprint) and still take 21s to download. the only explanation i can think of is that SPM is fetching the entire repo history, because it should not take 21s to download the source code of a package like swift-collections.

is there anything that can be done about this?

4 Likes
2 Likes

See here:

1 Like

Yeah, this was quite a problem for us in the Lottie repo, which is very large (300+ MB).

We worked around this for now by publishing a second lottie-spm repo that is very small since it only contains a URL reference to a prebuilt XCFramework. This adds some overhead to our release process, though, and also means that consumers using this repo no longer have the source code of our package in their workspace (seems inconvenient).

I'm hoping that the upcoming Swift Package Registry lets us avoid this sort of issue, but haven't heard any updates on that in a while.

6 Likes

i always thought it was odd that package registries were under the purview of evolution in the first place, creating such a thing involves setting up and running a service, and that is as much an entrepreneurial pursuit as it is an engineering pursuit.

i am sure the swift team can do anything they set their hearts to, but founding and running a business seems quite incongruous with the typical work of designing a programming language. wouldn’t enabling shallow clones in SPM be a lot easier?

Registries are under evolution insofar as a standard was needed and changes were required for SPM to adopt that standard, which then required public changes to the package format. It seems clear Apple has no desire to build a registry themselves, instead hoping the community would build them instead. Obviously that hasn't happened, aside from support in newer versions of Artifactory (IIRC) for some reason. Whether that's due to community interest or the fact that Xcode still doesn't support registries at all I don't know. Since you're on Linux you could use them directly through SPM if there as a registry anywhere.

This is purely speculation, but the fact that Apple has recently become a sponsor of the Swift Package Index might mean that there's an ongoing partnership with the expectation of SPI eventually becoming a semi-official registry. Of course this is not confirmed by neither party, but I don't think it's too far fetched, especially considering how much of an anomaly it is for Apple to sponsor third party projects.

2 Likes

I think I’m just missing something, but why would a registry help? Because there’s enough information available up front that shallow clones become an option again?

Registries allow for querying for the exact or latest versions and include only the files necessary to build (or the whole repo at the moment, I don't recall) rather than a full git history. Even in the case where SPM can use a shallow clone a registry should still be faster since it doesn't have to call out to an external tool or do any git work.

2 Likes

The registry spec has defined specific endpoints for querying and fetching package manifests (which are needed during package resolution, so potentially for many versions), so SwiftPM won't need to download the actual sources for a version until it has determined the concrete one to use.

8 Likes

And with the package registry, consumers don’t have to interact with a git repo at all, right? So downloading the source for a version wouldn’t require downloading any git history (you just get a zip of the package source)?

4 Likes

Correct.

2 Likes

Any updates on this issue?

as far as i an aware, there has been no progress made and no public indications from swift project leaders of having any plans to address this. i would love to be proven wrong though.

5 Likes