Swift Package Installation

Swift Package Installation

Introduction

It would be desirable to simplify the use of executables that have been compiled with swift build and installing the binary, payloads and other requirements into either a global location usable by all users on the system, or make it available only to the current user.

This would simplify the distribution and usage of Swift-authored tools, as instructions for installing the software would be as simple as directing the user to type swift tool install.

Other ecosystems like Node and .NET have made it easy for consumers of software to grab software and make it accessible with a single command. In the Node/NPM world this is achieved with the npm install tool and in .NET with the dotnet tool install command.

Motivation

Today when you build a Swift application using swift build, the resulting executable and satellite dependencies are deposited in a platform and configuration specific way. Today the result of swift build is deposited into the .build directory, in itself, not entirely obvious, and inside there is a platform specific directory, as well as a debug or release directories that contain the executable.

If would be ideal if the instructions shared with the world were as simple as instruction users to install a program with: swift build install --``global and this would take care of installing the executable in the right location, and any dynamic library dependencies that are part of the project.

This command line tool could be used to install a Swift-version of the cron command line tool into /usr/local/bin:

$ swift build --target swift-cron
$ swift build install --global
$ ls /usr/local/bin/swift-cron
-rwxr-xr-x  1 miguel  wheel  518886 Apr  6 16:39 /usr/local/bin/swift-cron

Linux distributions that compile binaries and distribute those additionally require the installation step to be done to a special installation root directory, and often configure the location for files. For example, configuration files go into $sysconfdir which is /etc and we would likely want our software to be installed into /usr, so an RPM build file could look like this:

swift build --target swift-cron
swift build install --prefix=/usr --prefix-sysconf=/etc

The above sets the prefix for all the assets to be /usr, so binaries would go into /usr/bin, and libraries into /usr/lib, but any configuration files would go into /etc via.

Proposed Solution

Given that swift build can build either products or individual targets that can include more than executables, the installation would handle installing executables, libraries and shared libraries.

Additionally, to make it so that Swift software that is installed in ways that conform to the macOS, Linux file system standards or BSD standards, we borrow the concept of prefix-based software installation where various installation subdirectories are derived from the prefix directory, but can also be overwritten on a case by case basis.

also for Linux distributions to package and , as well as to make it simpler to use Linux packaging technologies that both require custom directory installations ($sysconfdir, $libdir and so on), as well support for RPM-build

The following options would be added:

  • --install would instruct the build process to install the assets, this option by default would default to installing to the user home directory (for example on Linux, this would be in $HOME). The --install flag additionally would have:
    • --globalwould direct the installation to install these into the appropriate system-wide directory (for example, on Linux this would be /usr/local/bin for the executable, and /usr/local/lib for libraries)
    • --prefix and --prefix-xxxx flags would override the directory prefixes set either by the default, or the --global option, to ensure that different assets can be installed in the proper locations
  • --uninstall would instruct the build process to remove the files that would have previously been installed with the --install flag, and like --install, this would also take the --global flag and the various --prefix flags.

These are the various directories that could be configured for different assets, the following shows a some defaults that would make sense on Linux, but would

  • Executable target: goes into --prefix-bin, this is computed as: $prefix/bin
  • Library targets: goes into --prefix-lib , this is computed as $prefix/lib

Additionally, we could
And on the systems where this makes sense, we could additionally install a pkg-config file that would allow the Swift library to be easily referenced by other ecosystems.

Open Topics

Directory Configuration

While not necessary to support these, it would be nice if these capabilities were supported in Swift out of the box.

For system-wide tools, it might make sense to define a global variable that describes where to fetch assets from, in the example I showed above, the imaginary swift-cron command would likely expect a cron.config file on /etc.

Historically on Unix this has been handled by using compiler level defines, or hardcoding the contents on a config.h file. For Swift Build, these paths could be generated as a global class or module containing the paths, for example:

public struct SwiftPackageConfiguration {
     public let prefix = "/usr"
     public let sysconfdir = "/etc"
     public let bindir = "/bin"
}

Versioning Support

One thing that I did not considered in this proposal is a versioning story - the heart of this proposal is really the binary tool installation, and not the library assets.

I suspect that there is no need to version the binary for a tool (chances of me needing swift-cron-1, swift-cron-1.2, swift-cron-3.2 installed in parallel is not likely high).

Libraries are a different story, those could either be versioned as they tend to be in Unix (libSwifcron.so.1.0, libSwiftcron.so.1.2), or they could be versioned by using a different directory for each version (/usr/lib/libSwiftCron/1.2/libSwiftCron.so).

The former has the advantage that it would work with existing idioms and would be picked up by the system dynamic linker without additional system configuration.

30 Likes

Have you read this other thread yet? It outlines some of the hurdles the design will have to deal with.

3 Likes

Iā€™d rather we keep package installation within a real package manager rather than spreading them among every languageā€™s package manger. An enhancement to brew to let it install SPM build products, perhaps even directly from a repo with a package definition, would be great.

8 Likes

Brew doesn't work on Linux unfortunately though. Although I agree having it nested inside swift build feels awkward to me. Why not make use of SwiftPM and have the command be swift package install? That keeps the logic of installation and what dependencies are needed inside SwiftPM rather than leaking it into the build system itself.

Overall though, I'm all for this!

2 Likes

In general I like the concept in terms of improving the user experience of getting Swift package based tools installed.

However, I also things itā€™s going to take a lot of individual pieces to get us there (because of open questions and problems linked to in other thread on this topic).

What Iā€™d like to see us do is first focus on a proposal for a new SwiftPM product type representing an ā€œinstallable artifactā€ (tarball, cpio, etc)... this would let us make significant forward progress on designing how packages need to describe their installable assets, and cross platform/portability concerns, without immediately jumping headlong into the whole space.

Once that is done (or concurrently) Iā€™d suggest the next step is to focus on a ā€œsandboxed localā€ installation mode where those artifacts get dropped into -/.swiftpm/artifacts or something, with some helpers to get into PATH. What Iā€™d really like to see here is something analogous to Nix (https://nixos.org/).

9 Likes

To give a concrete idea of why I consider just the ā€œdeplorable artifactā€ problem hard, here is a concrete example of all the kind of hacks we regularly do to take a SwiftPM package and make it into something packed for macOS or (portable) Linux distribution: Deployable Artifacts thoughts Ā· GitHub

Iā€™m going to just not edit the deplorable typo out of this post. :joy:

17 Likes

You're so spoiled for choice on Linux you could probably extend one of the existing package managers to do this.

My logic just this: one installation location is better than numLanguages + numPackageManagers locations, especially when the capabilities of the language installations are so primitive compared to real package managers.

I would suggest SPM focus on being a good dependency manager first, then branch into whatever.

3 Likes

I completely agree with Daniel's point about separating the building of an installable set of artifacts from the subsequent actual installation of those artifacts.

In addition to the other advantages (such as easily being able to deploy to a different machine etc), it eliminates the need to use any elevated privileges during the build. Even if some of the installable artifacts require special file ownership or permissions at deployment time, those can be set in a produced CPIO (or other kind of archive) without any special privileges at the time of building (as opposed to using chown or chmod on an actual file in the file system, as is commonly seen with make install).

It also lets whoever is doing the deployment choose how to do that ā€” for example, on macOS it's common to use darwinup to install archives of artifacts, rather than just copying them into place.

2 Likes

It's worth pointing out that cargo, by default, installs to ~/.cargo/bin on UNIXy systems, and the installation of cargo itself has a step that adds that to the shell's PATH automatically.

7 Likes

Furthermore, cargo will only install binaries, not libraries or other products, which I think is very reasonable as well. Installing those requires some level of familiarity with where things are supposed to go on the respective platform, and is a job better done using the respective package manager. That's even leaving out version conflicts etc.

I think this makes sense as a utility for packagers, but would be wary of having this be something end-users are expected or even recommended to use for installing things system-wide.

While other tools like pip, nvm or rubygems have similar capabilities, the communities around those languages have all developed tools like nvm, virtualenv or rvm to avoid actually installing anything system-wide, so I think defaulting to installing somewhere in the home directory (and probably warning against using this for system-wide installs) would be a happy medium.

2 Likes

Brew doesnā€™t work on Linux unfortunately though.

Homebrew actually does have Linux support. (Disclaimer: I have never tried it so I donā€™t know how well it actually works.)

1 Like

Do we actually want to reinvent and reimplement a whole package manager (not dependencies only) inside SwiftPM? That sounds like a huge effort for something that would always be second-in-class to the platforms native package manager at best.

2 Likes

10/10, would agree with again.

2 Likes

Would it really be second-in-class, though? My experience with CentOS's native package manager is that it does at least as much harm as good when you're working on a project whose dependencies themselves depend on different versions of the same package. If we can combine the versioning info in package.swift files with Swift's resiliency features to solve at least a class of "dependency hell" problems, I'm very much in favor of it.

Applying resilient builds to frameworks which don't support it is dangerous, leading to build and linker errors, as well as runtime crashes. Applying it as a blanket policy would be harmful. It's also not good for the ecosystem due to the severe limitations it imposes on libraries. So I don't think it's a path to what you want here.

1 Like

Right, you'd have to take into account whether a library supports resiliency before including it in that "class" of problems which can be solved this way.

I dunno. My main point was to separate out these two discussions to unblock forward progress.

4 Likes

If we decide to go ahead with this pitch, I think we should not reinvent the wheel but stick to what the expectations are for the given platform. For example, this part of the pitch

this option by default would default to installing to the user home directory (for example on Linux, this would be in $HOME )

is not quite correct. For example, cargo does this thing where it creates its own little .cargo which is not appropriate; it should be following the XDG Base Directory specification. https://github.com/rust-lang/cargo/issues/1734

2 Likes

Which makes it a deployable typo?

5 Likes