What is the best practice for ensuring that I have permanent access to a dependency that was used in a build?
Assuming that a direct dependency is hosted in GitHub, if the creator of that package decides to delete the repository then my project will no longer be buildable.
One possibility is to make a fork of every direct dependency and then use that fork. This introduces a lot of manual maintenance, and doesn't help with indirect dependencies.
Is there a tool or technique that given a Package.swift fill will at least download all direct and indirect dependencies so that I can ensure that someone deleting a package will not break my builds?
This is the thing I miss from Cocoapods. Being able to pull dependencies on the project folder and commit them removed any uncertainty around this. Now is not rare that CI builds fail because a som dependency fails to be pulled. I would be happy with a setting to define a folder to checkout dependencies that can be committed, nothing fancier.
This is a common issue and I would also like to know how other people handle it.
The Package.resolved file written by the Swift package Manager has a list of what versions of what packages was used while building, and of course for the builds that are important to you you would like to guarantee access to them. So how would you do that?
You could mirror the according Git repositories, but note that history in Git repositories is not fixed (and of course for the same URL there could be a completely new repository), so just keeping some mirrors up to date is still no guarantee. Obviously you need to backup the checked-out package versions, maybe then keeping track of them in your own repositories, or just keep all checked-out sources together with important versions your software?
Note that once you have your own alternative repositories with the according version tags you can redirect the according URLs via Git configuration.
Anyone doing some efforts to have a such guarantee for dependencies, and how do you do it?
The .build/checkouts folder contains the checked-out versions (not when building with Xcode), you would need to copy the contents (without .git, and better without .gitignore files) to your own repositories where you commit and push the new state and set the according version tag (but you will have a problem with repositories referenced by commit numbers in Package.swift).
You will have a "garantee" when you actually use builds which use your own repositories (not to mention that you need to backup your repositories).
With Xcode the checkouts are somewhere else, but when using a CI pipeline your program would be built outside Xcode and .build/checkouts is then the right place to look for them. This is just a detail. And I think such a backup process should not be part of the usual build process, as doing it for every build (if activated) might be too much. It could be a nice third party tool to be called manually or to be integrated in a well-setup CI process. It is even not too difficult.
But aside from these very concrete details, I would be interested to know how others deal with this problem.
If it's a hobby project then I wouldn't worry about long term access.
If it's your full time job project, you should maintain a fork of everything modified to point to eachother and optionally also switch to binary distribution to speed up your builds. "Watch" new releases on GitHub and review all the changes when updating to the newer version (and also review everything you add in the first place).
I don’t disagree with the suggestions but I’m also surprised this is what people recommends.
Isn’t it just simpler to let the tool that promises to manage your dependencies let you define the checkout directory? This has worked flawlessly for cocoapods for many years, solves a real problem and doesn’t require complexity on the part of the developer.
This is also something that a registry would help with. Once published, registries can keep the archived version of the library forever, and I don't believe unpublishing is even allowed, according to the spec. The compressed archive for each version is minimized to just what's required for the build, not the entire repo, and is extremely easy to store long term. It could even be offered as a separate download that could serve as a local cache or be part of a local backup automatically. Unfortunately it seems like Apple has little interest in actually hosting such a service, and without official integration with Xcode, third party efforts are unlikely to succeed.
costs could easily spiral depending how how fine-grained you want the versioning to be. a lot of ecosystem packages are not very good at tagging semver releases (or any releases at all), so it’s very common for projects to just depend on the default branch, which in practice is the same as depending on arbitrary commits.
By default, those would be incompatible with the registry standard (as I recall it) and would just fallback to the existing git integration. Releases through the registry require specific versioning, preparation, and publishing, which allows steps to optimized artifact size (by default it's just a zip archive, which for most source libraries, should be trivially small).
Edit: As an example, macOS' default zip compression turns Alamofire's 42 source files, with 2253 blank lines, 5223 comment lines, and 8949 source lines, into a 172 KB zip file.
i’m not sure if that is well-suited to solving the problem here, which is dependencies becoming inaccessible over time. a package that is diligently tagging releases is probably a “mature/established” project that is unlikely to blink out of existence. i feel like this is a problem that is far more likely to crop up when using someone’s code on GitHub that was never really intended for public consumption in the first place and could be taken down at any time.
of course, it is never a “good” idea to depend on code that is not intended for public consumption, but it is the reality of the package ecosystem that for certain algorithms or integrations, there are few alternatives to using somebody’s code dump on GitHub.
Right. With such a risky dependency the repo going away is probably the least of your concerns. However, once the dependency moves at all into proper versioning, a registry would solve a whole host of issues, not just persistence.
yes, i agree that a package registry is sorely needed, and i do wish Apple — if they are not interested in doing it themselves — would contract with a third party (like Swiftinit) to provide such a service, as it is pretty close to what we are already doing right now. i just wanted to point out that a lot of projects are not really able to stay within the “nice” semver universe for reasons out of their control.
Perhaps it's different for backend projects, but Apple platform projects don't really have that problem. There are popular backend libraries that don't version?
historically (over the past 3 years) swift-markdown users have had to depend on main as it has a very irregular release cadence; i’m sure everyone can think of at least one library that does not release as often as they would like.
but again, limiting your analysis to “popular libraries” sort of handwaves the bulk of the problem, which is those very niche, very specialized “unpopular” libraries that nevertheless we all end up depending on unless you have a habit of writing everything from scratch.
In that case, the persistence problem is solved simply by it being an Apple (swiftlang) library that is essentially guaranteed to never go away. Beyond that, the earlier suggestions would suffice. Unfortunately, no single solution can suffice for libraries which flout best practices.
I don't remember if there has been any conversation in the past that spoke against adding a parameter to SPM to change the local directory. I know Xcode is another story but hopefully having it in SPM means it makes its way into Xcode eventually.
IME failing to checkout dependencies is rare, but that depends on both your CI and package hosts. GitHub Actions accessing packages on GitHub is usually only down when GitHub itself is down, which is unlikely to build successfully anyway.
But registries do help a bit in this case, as archive downloads are far simpler to handle than the fill git checkouts that SPM currently requires. If you wanted you could even check the archives directly into your repo, as that would be a much easier integration to build (assuming SPM is never changed) instead of trying to move entire git repos.
