Hey everyone,
I have drafted up a pitch proposal for SwiftPM to overhaul the behaviour of various CLI commands to treat the Package.resolved as a proper lock file. This is still a very early draft but I wanted to share it already to get more feedback on the general idea. I am especially interested in experiences that people had with other dependency managers.
You can find the whole pitch here
Looking forward to feedback and questions!
Franz
Big +1 on this - the fact that the resolved file doesn't actually act as a lock file is extremely confusing for even experienced developers and a massive regression after removing Package.pins
The pitch suggests deprecating swift package resolve since it actually downloads as well as resolves, which may be confusing. However, the suggested replacement, swift package fetch, seems to be potentially confusing in the opposite way: it resolves as well as downloads. Can we simply use the common term for the combination of these actions, install? We could even define it as the combination of a fetch and a resolve. (I've never understood why SPM insisted on justifying its own vocabulary here.)
It's a great point you brought up here and I have been going back and forth on the naming. My though process was:
swift package resolve is very confusing naming wise since it also clones/downloads any dependency. It is also unclear if and when it does a dependency resolution.swift package install was my first name but what if we want to add a similar feature such as cargo install where one can install an executable product into a system path and make it available globally.swift package fetch is where I ended up with because the primary goal when invoking this command is to fetch all dependencies. A resolution only happens in the case where there is no Package.resolved file present otherwise it will just do clones/downloads of the dependencies.I am not too tied to fetch and open to suggestions but in my opinion not using install leaves room open for installing executables in the future.
FWIW, we already have a prototype for this available in 5.10 development snapshots called swift package experimental-install.
Personally I don't think swift package install should ever be used for system installs, especially if we're worried about commands being confusing. That seems more like it should be an option on build. swift build -c release --install <optional location> If we really wanted it on package install we could do it as an option, like NPM: swift package install --global. But like I said, I'm not a fan. (Personally I don't find swift package resolve confusing in the first place, how is it expected to resolve if it can't download?)
I think that’s an important point to focus on. The goal of the proposal is that it shouldn’t resolve in most cases but just download/clone. If I already have a Package.resolved file I just want to fetch everything and not run a resolution at all. That’s why I opted for deprecating the resolve name because it coneys the wrong semantics .
Resolution should only happen if there is no resolved file or users are actively opting into resolving by calling swift package update
That feels a bit pedantic. I'd expect resolution to download and then ensure my resolved versions, whether from the .resolved file or the latest, are available. This is how the other dependency managers I'm familiar with work (CocoaPods, Carthage).
@FranzBusch there is also the use case where users update package.swift with new dependencies, or modified constraints in which case a resolution is called for. iiuc the proposed model above will require them to call package resolve (or update but that has other side effects), or say “yes” to the prompt
That's a good point. In most managers you can "update" or "install" with a single package to isolate your changes, in addition to being able to call them generally.
If we are going to discuss naming, why does swift package {resolve/fetch/update} refer to dependencies?
swift package dependencies update/resolve
swift package dependencies checkout/install/download/sync/fetch
In the end it is all about bike shedding the name. FWIW those are names other package manager call this operation:
pod installcarthage bootstrapcargo fetchnpm installpipenv installbundle installFrom this list it looks like install is the command that most other package managers go with. I do find the choice of cargo interesting with the fact that we have already an swift package experimental-install
However, I don't want to spend too much time on bike shedding here. The part which I would like to get the most feedback on is the actual behaviour of the commands when a Package.resolved file is present or not.
Correct. There are basically two cases where fetch/resolve/install (whatever we name it) is going to end up running a resolution in the end:
Package.resolve file is presentPackage.swift and Package.resolved are out of sync and we need to run a re-resolution.Overall it reads well and I am happy that SPM is getting some love in this direction.
I stubbed my toes a few times now where I was not really certain what SPM would be doing - and the documentation of the various options was rather thin. So any step towards clarity is a good one in my book.
I 100% agree with this. Messing around in "global space" should be very explicit.
IMHO "install" should be the "make my local setup ready to go please" command - but I can definitely live with fetch.
And if you'll allow a little side-rant:
SPM feels sllooooooow.
I switch between pnpm and SPM on a daily basis and the difference is brutal. I love swift (and ...not love.... nodejs), but if you do a pnpm install it finishes in a few milliseconds and apologizes that it took so long (because it had to fix a million things in the lockfile or something.)
In contrast it feels SPM takes that same time to just figure out what it wants to do today before it leisurely starts doing its job.
I understand this is not very actionable, that there are reasons for how it is ATM, and that I could take a stab at it myself, and how this is not part of this pitch.
I mainly bring it up because if this pitch enabled new possibilities for performance improvements (like better caching, better parallelization, new shortcuts or fast-path optimizations) going after them would be effort well spent.
I didn't really mind/notice because of how absurdly slow npm is on Windows:
$ time npm --version
8.8.0
real 0m4.118s
user 0m0.121s
sys 0m0.416s
To steer the conversation in a productive/relevant direction though, I am massively in favor of the ability to use Package.resolved as a lockfile and I'm surprised it isn't already like that. I don't have any strong opinions on the naming of fetch/resolve/update/install as long as it's possible to update only one package at a time (which it seems like there will be a flag for that).
A big +1 on the general idea. I have been hurt as well by how dependency resolution behave.
I have a question. When Package.resolve is out of date, should swiftpm (1) avoid updates as much possible, or (2) update everything.
First, no update is needed when the changes to the manifest are not related to dependencies, and when the resolved versions are compatible with the new versions in the manifest. Then when dependencies are only removed, the only change needed is to remove them the Package.resolved. And when dependencies are only added, only the added dependencies might need resolution.
I think (1) is better.
I personally came to the conclusion that Swift PM should tell the user that the two files got out of sync and prompt the user to run a swift package install before proceeding with the command they issued before e.g. swift build. This is super important for the CLI invocations since those might run in CI environments where your really want to have the default of no updates at all.
For IDEs, the story is different. They can file watch the Package.swift and see if a user did a change to the manifest and run a swift package install automatically without necessarily requiring a prompt. However how IDEs integrate with Swift PM is outside of Swift evolution.
Hi All,
While this thread is a few years old, if we're looking at revisiting SwiftPM's package resolution I think it might be worth looking at how Go's resolution system works. It has some desirable attributes, such as being reproducible without conflicts between a manifest and a lock file. Please note that I'm using SwiftPM terminology where the Go terminology is a bit different below so that the parallels can be drawn. The links below will of course use the original Go terms.
How does it accomplish this? First, it uses a minimum, instead of a maximum version selection algorithm for resolution. Given a set of versions in a manifests in the transitive closure of packages the resolution will always be the same outside of some kind of tampering or rewriting of history. Minimum version selection also has a side effect of promoting dependency cooldowns, which can help to mitigate common types of supply chain attacks.
So, how does the Go package manager help to avoid tampering? There is both a public sum database, as well as a local sum file with checksums for the currently selected versions. If there's tampering then the package manager produces loud errors about it. This file can be committed, which is recommended, or left uncommitted and the package resolution remains repeatable without the extra level of checksums.
If SwiftPM were to adopt a similar mechanism, then it might gain the overall benefits of conflict freedom and reproducibility. There are additional capabilities in SwiftPM that would need to be considered in this mechanism. The first is the branch-based dependency.
Branch-based dependencies are inherently maximum version selecting from a package manifest perspective, which requires some additional piece of metadata to indicate what revision is resolved at a given time to prevent each build from attempting to fetch and checkout the latest from the package's branch. This would yield an unsatisfactory user experience, and so today it is the Package.resolved that indicates the last commit that was resolved, awaiting the user to do a swift package update to bring it to the latest commit at that point. Committing the resolved commit in the Package.resolved might help to make things more reproducible at the cost of eliminating the maximum version selection. Leaving the file uncommitted limits reproducibility for all packages permitting resolution to be closer to maximum version selection of the branch in CI, or a clean workspace.
Go can handle branch-based dependencies with a separate package replacement mechanism where a package can be given an alternate local path. Branch-based dependencies are often used in place of a git mono-repository, and so this replacement gives the opportunity for an external entity (or user) to manually align the revisions. If SwiftPM had a similar replacement mechanism then the package manifest could then use regular repo and version-based constraints, and the replacement list can be used to override it in the workspace.
In summary, we can have reproducible SwiftPM builds without a lock file that can potentially conflict with the package manifest by transforming the lock file into a simple sum database with the purpose of helping to detect tampering. This is accomplished using minimum version selection, which has another nice advantage relating to supply chain attack mitigation. The case of branch-based dependencies and their inherent maximal version selection can be handled in another way with a package replacement mechanism that allows external entities to align the versions independently.
Cheers,
Chris