Should SPM allow installation of package on the system

Evolution Discussion
,
1

Hi,

Do you think it will be useful if SPM could install package on the system(in /usr/local/bin for exemple)?
This project GitHub - swiftbrew/Swiftbrew: Homebrew for Swift packages actually make it possible.

Thank you,

3 Likes
2

It has been in the Evolution Ideas document in SwiftPM since the beginning.

There are several tools and scripts in the wild that attempt to do this. Mint is the most popular that I have seen.

But to actually do it right is far more complicated than most of those tools admit. Despite having tried many such tools, I still find myself having to clumsily use my own script, because each tool inevitably neglects some SwiftPM feature, such as dynamic libraries. The arrival of resources (SE‐0271), will make it even more complicated.

There are several major design hurdles:

  • You cannot just dump the executable into /usr/local/bin (which is roughly what Mint did last I checked), because you will loose any dependencies and resources. Neither can you dump all the products into a global location, because a dependency might overlap with a different package, but at a different version.

  • You cannot install a binary built elsewhere (which is what Swiftbrew appears to be doing), because the matching Swift toolchain isn’t guaranteed to be in the same location. macOS can safely assume consistency with some stable system libraries like Foundation and Dispatch. But XCTest and SwiftSyntax’ implicit toolchain dependencies are another story. Outside Darwin, not even Foundation is reliable unless you build from source on the destination machine.

Those are all things that “just work” with swift build and would need to also “just work” with a hypothetical swift install command.

6 Likes
3

I definitely think it'll be valuable to have a swift install command. @SDGGiesbrecht laid out many of the problems that we need to address for implementing such a command in a cross-platform manner. It would be great if someone wants to drive a pitch/proposal for adding the install command.

cc @Yonas_Kolb :slight_smile:

8 Likes
4

What’s the value of SPM doing this? I’d rather keep my system installs in one place, like brew.

4 Likes
5

A few reasons I can think of:

  • The workflow to publish packages for Homebrew is a little cumbersome. You need to make and maintain a formulae and most likely your own tap in a separate repo.
  • While Homebrew is very widely used on macOS, on Linux that's not necessarily true, so publishing for multiple platforms may require more than putting it on Homebrew.
  • Running multiple versions of a package is a little tricky in Homebrew. You have to explicitly call brew switch <formula> <version> to move between them, they can't be used side by side easily.
3 Likes
6

The flip side of this is that if we push the work onto SwiftPM that the complexity still exists, but it is now SwiftPM's problem. There's a lot of historical baggage here: for example, Python's pip tool has had to bend over backwards to behave well alongside tools like apt-get. There are many Linux package managers, each with a package format different from the others, and if Linux ever starts packaging Swift packages then SwiftPM and the system package manager will have to learn to get along.

On the other hand, if SwiftPM simply decides not to do this, then we can have a clear separation of worlds: SwiftPM is for development, not distribution.

8 Likes
7

An alternative might be to directly integrate with various official system package managers:

$ swift package export apt-get

> swift package export PackageManagment

...

But I for one don’t like managing multiple redundant manifests side by side, so I consider the status quo to be unsatisfactory.

1 Like
8

+1 for an install command that all it does is copy the statically linked binary to /usr/local/bin and any associated resources to /usr/local/share (or any prefix).

−1 for an install command that can update/upgrade/manage installations

−1 if the idea is to install dynamic libraries and manage the resulting dependency hell.

SwiftPM is good as a resolution and build tool, and should be careful (IMO) to stay focused on that. I +1 a limited install function because it will increase adoption of minor tools before they become popular enough to warrant being packaged by the big-kids (eg. brew/apt).

9 Likes
9

No one (that I know of) is suggesting installing libraries for shared use. Dynamic libraries only come up because an executable can have one in its dependency graph and it still needs to work. Putting all the products in an isolated directory somewhere and then only symlinking the executables in /usr/local/bin would be one way to make sure multiple executables can have incompatible dependencies and still live side‐by‐side without a problem.

1 Like
10

:+1: As for prior art, there's $ npm install --global package-name which installs a binary globally, and is a very useful and popular feature in the Node.js community. I have made lots of Node.js command-line tools that can be easily installed with a single command. I would love to be able to also distribute Swift command-line tools this easily.

11

No one (that I know of) is suggesting installing libraries for shared use. Dynamic libraries only come up because an executable can have one in its dependency graph and it still needs to work. Putting all the products in an isolated directory somewhere and then only symlinking the executables in /usr/local/bin would be one way to make sure multiple executables can have incompatible dependencies and still live side‐by‐side without a problem.

K cool, I mentioned it since that it was most of the “equivalents” do (pip, gem, npm (less so)), they have their equivalent of “dylibs” installed in the prefix and manage the dependency hell. I feel this is outside the scope of SwiftPM and it’s notable that system packagers usually end up installing the more popular tools themselves since managing large amounts of software (where that software is more than just a single binary) is best served by a central committee that knows all the caveats.

1 Like
12

This would be especially useful if we get package support in scripts. Otherwise, consider a simple Vapor microservice - it would take a good 5 minutes or more to start! It would have to download and build everything from scratch, including dependencies like SwiftNIO.

You could call it "installing", or you could call it a "shared cache of prebuilt products" - either way, it would be nice if all my locally-running packages had access to it.

13

A shared cache would be a separate feature, though it is also on SwiftPM’s general wish list. master already supports some aspects of cross‐module optimization, and as that grows, SwiftPM will want to take advantage of it. To that end, instead of only the final products, a shared cache will probably contain the results of each build phase—dependency clones, object files, etc. SwiftPM would then reach as far back as useful for optimization, but no further.

14

An install feature would be good, but I don’t want it installing in /usr/local/bin. Maybe ~/bin, or maybe ~/.spm/package.bundle.name/version/bin, with a path updated that adds all those folders — or something like that.

2 Likes
15

It’s pointless if it doesn’t install somewhere in PATH.

3 Likes
16

...

No package manager installs into the default path (on macOS at least). Is it even possible anymore? So since it needs to be added to the path anyway, what does it matter?

17

brew installs to /usr/local/bin, which is also in the default PATH on macOS. If you’re referring to SIP, then /usr/local is not SIP protected.

This is also the default path for gem, pip and node. The only PM that doesn’t install to /usr/local/bin that I know of is MacPorts—and I get some people consider it the only real package manager, but there’s no accounting for taste.

1 Like
18

Ah, right. I guess I was thinking of the other directories brew warns about. Is it a good idea to have multiple managers writing into the same directory? I could see /usr/local/bin/spm/, but I'm still not sure it's a good idea.

19

It’s not ideal, but it already happens. /usr/local is a free for all, you get no guarantees about it.

I maintain I don’t see the point of an install command if it doesn’t “just work”, nobody will be able to say just: [sudo] swift install https://githhub.com/foo/bar && bar --help in their project README.

Really this is all you need to do right now:

git clone foo
swift build -C foo
sudo cp foo/.build/foo /usr/local/bin
foo

What the theoretical install command saves is:

  1. temporary directory
  2. clean up
  3. remembering where the build output goes

If a README also has to instruct a user on how to edit their PATH then—believe me—you have created a useless command.

2 Likes
20

There seem to be several potentially related features for Swift PM that I could see affecting how a feature like this would be implemented.

  • The idea from this thread of installing a single executable globally
  • Shared cache as Karl brought up. I understand this could be a separate feature, and perhaps it is unrelated completely.

The last one I'm not sure has been explicitly brought up yet:

I'd personally love to be able for my project to have a devDependencies section or something like that that would specify the version of an executable that project uses, and be able to install and run those tools (different versions, in parallel on a machine) using something like swift run <my tool> <command>

A lot of our tools for iOS development, for example, are written in ruby or javascript (ex: Fastlane or Apollo CLI), both which let you run the command line utilities using bundle exec <the tool> or npx <the tool> so that you run the correct version of that tool for the project you are in.

I don't bring this up to really gauge interest in that feature, but I think it would be useful to understand what the overall vision of Swift PM is when we talk about it managing executables.