`swift package install` Subcommand

Hello Swift Forums,

I'd like to know everyone's thoughts on a swift package install subcommand for SPM, which would clone & compile a remote target executable, then move the compiled binary to a specified output path, ie
swift package install https://github.com/SerenaKit/NVRAMKit --target nvramutil -o /some/directory

Which would clone the given URL, compile the executable target specified, and output the binary at the specified output path.

Please let me know your thoughts on this, I have a working implementation thought a bit janky :sweat_smile:

2 Likes
1 Like

:sweat_smile: Should’ve searched more earlier, sorry
Tried to delete but no permissions to do so..

Not at all! It's an interesting topic that comes up with some regularity, worth resurrecting the discussion if people are interested.

1 Like

Has there ever been an implementation made before? I've made kind of a working one however there are still topics I want to discuss which haven't been fully resolved by those convos, ie, where should the bins even be installed? /usr/local/bin seems like the best solution here, however it may interfere if an existing binary exists there with the same name as the one being compiled, and something like ~/.swiftpm/bin would require the user to add said directory to their $PATH

I think everyone could easily agree about something like the form in your original post, where the destination directory is specified manually. You currently have to manually filter the products directory to figure out what is actually part of the product (dynamic libraries, resources, etc.) and what is just intermediate junk (object files, source indices, synthesized headers, etc.) I am not sure install is the right word for it, since it is more a matter of bundling in order to vend, but it would be the first step toward a real installation command and still very useful on its own.

Deciding on a default install location for permanent installations will be more contentious. It must support all the things SwiftPM does (most notably dynamic libraries and resources), and that makes it a non‐trivial decision.

We cannot support install only as an one-off option. We need to have a full set of tool to install, update and remove them. Also, how should we deal with dynamic libraries? If we place them together, different versions of the same dependency will certainly collide.

One solution I’ve come up with is to use the pre-pitched SwiftPM scripting support, which means we can put such file:

#!/usr/bin/env swift script

@package(…)
import Executable

Executable.main()

into $PATH, and manage it with the script management toolset.

1 Like

I disagree, I very much think limiting it to just executable targets would be enough, that would be the use case of this command most of the time, besides, if we were to add library support, wouldn’t that come with a whole plethora of issues? Executable target support seems simpler and just enough, in my opinion

This I agree with, and would require storing the installed items in a file recording:

  • Path
  • Name
  • Version (somehow?)
    After implementing a system to record the items installed, removing & upgrading would be easy to implement (in theory…) but how would you record the version?

With custom SwiftPM plugin, a tool developer is already able to write such a plugin to copy or link its product into a designated directory (with sandbox disabled), which can provide much more functionalities than what you proposed. To officially support this feature, we still need to investigate deeper, and I personally don’t think it’s of very high priority — particularly given that we have a solution by SwiftPM plugin now.

I think it's a great idea & would love to see it become part of the package manager. I have written a program to accomplish the same task (mine's a bit janky as well). I'm using "deploy" as the verb. Naming the executable swift-deploy allows the Swift frontend to pick up it up from path through the swift command, so "swift deploy" is all I need to type to create a zip archive for deployment.

A plugin still requires cloning the package yourself, and also only works for packages with the plugin. I think this proposed feature is trying to replace what mint does, and personally I’m all for that. Mint is getting a bit old and isn’t really maintained anymore (many easy issues and prs are sitting unfixed and unmerged). But it provides very useful functionality for distributing executable tools written in swift.

One possible default would be copying the tool and any dynamic libraries and resources it requires into ~/.swiftpm/bin/tool-name_version/ and then symlinking the executable into /usr/local/bin. This avoids clashes in dynamic library version I think (the executable should find libraries within the same directory as it). And would make versioning pretty easy. A more portable solution may be symlinking to ~/.swiftpm/bin/tool-name instead and requiring the swiftpm bin die to be added to the path (swiftpm could offer to automatically add that to the path on most systems the first time install gets run.

1 Like

What I meant was that installing an executable must work no matter how many of SwiftPM’s features it and its dependency graph exercise. It is not reasonable to tell users they can no longer update an installed executable because one of its transitive dependencies started using this or that feature.

I am not suggesting that dynamic libraries or resources should be somehow installable as independent entities. My only concern is that if two installed executables depend on the same resources at different versions, they cannot stomp each other out, but must exist side‐by‐side. That rules out naïve dumping of products into /usr/local/bin.

The second is the strategy I use externally now (because of reduced odds of conflict with other installers), but the first seems reasonable to me as well, and I would be fine with either. I just suggest deferring it and focusing on arbitrary user‐controlled installation, because I expect it to not be long before someone comes around insisting that that is not where things belong on platform such‐and‐such. Discussions about this feature have shipwrecked on this very point several times in the past.

The first part sounds reasonable, but that means we must find a way to version an installed executable, a possible idea I had was recording the commit hash when installing a package, and when a user asks to upgrade the package, check the latest commit hash of the remote URL of the repo that the executable was compiled from

+1 on the general idea of spm install.

I would suggest looking at nix for examples of how multiple versions of executables & dylibs can be installed and used in a single environment.

4 Likes
./configure —prefix=/where/to/install && make install

Is available and used on all *nix systems for ages and is a foundation for all Linux package managers as well as for brew used on macOS.

It works nicely by installing what is required where specified. Versioning and upgrades are left to the next layer to handle. Specifically brew does that very nicely allowing multiple versions of the same tool to exist at the same time.

So from where I sit, simple spm plugin that will mimic the ages old make install into given prefix will provide a solid foundation to build upon without reinventing the wheel…

Update: I've made working, reliable implementations of 3 subcommands:

  • swift package install-package
  • swift package update-package
  • swift package remove-package

When using swift package install-package, the compiled binary is moved to ~/.swiftpm/bin, if the user doesn't have ~/.swiftpm/bin in their PATH, a warning is emitted.

Each installed package is recorded in a JSON file in ~/.swiftpm/ by the name of installed-packages.json, the following info is recorded down:

  • Name (of the binary)
  • Repo URL
  • Commit checksum built from (so that later, we can compare when upgrading)
  • Branch built from
  • Full path as to where the compiled binary is placed

Upgrading also works, and does so by comparing the checksum from which the executable was built from to the latest commit checksum of the repo, if it doesn't match, then it clones the repo, builds a new executable and replaces the old one.

Here is the implementation:

2 Likes

Probably a bit of a nitpick, but the trailing -package feels a bit clunky in these commands.


My main question is will these commands be able to install local packages? (an essential ability imo) Also, if given that the subcommands aren't necessarily operating on a package in a directory, it seems a little odd having them under the package subcommand because the description, and --package-path option, of the package subcommand contradict that usage:

OVERVIEW: Perform operations on Swift packages
...
  --package-path <package-path>
                          Specify the package path to operate on (default current directory). This changes the working directory before any other operation

Another consideration is the reliance on the package being a git repository, there might need to be some fallback behaviour for local packages that aren't git repositories (e.g. always assuming that the installed version is out of date when running update). Or maybe the update command doesn't really make sense for local packages. Maybe the update command shouldn't be included at all, because it could trip people up if they forgot which source they currently have a specific tool installed from (e.g. local vs github). As a precedent, Rust's cargo doesn't have an update command as far as I can tell, you just run install again (it's always clear where the package is going to get installed when done this way).

Rust's cargo tool might be a good place to look for inspiration. More specifically; cargo install, cargo uninstall and cargo pkgid (pkgid is used to identify packages in the uninstall subcommand).

1 Like

Swift package already have update and remove subcommands, so I couldn't just add new ones with the same name.

The implementation clones the repo URL given, adding support for local repos (ie swift package install .) could be done, i'll look into it today.

I believe we can do better, which is why I made the update command. If we went the rust way, we'd have to get people to manually edit the JSON, and I do not want to do that.

I'll be honest, I didn't want to go the route of relying on git. However, git makes it way easier to version each installed package / executable, believe me, it was either this or having to manually get the checksum of the binary produced, which would require us to recompile the repo every time the user requested to update.

It would be good to support selecting a package in all the ways SwiftPM does (source control, file system, package registry). You can likely route it through the same code that hunts down a dependency to save yourself some work. Presumably its means of comparing state can be repurposed as well.