Upcoming Enhancements to Swiftly

Server
,
1

Hello Everyone,

Swiftly is a command-line tool to help you to get started with the Swift toolchain that you need for your Swift development. It also helps you to switch easily between multiple versions of the Swift toolchain, including the latest and greatest snapshots, so that you can check that your code is ready for the next release.

We have started working with Swiftly and are proposing the following changes to move it towards being the default install experience for Swift:

macOS support
Swiftly currently only works with Linux. We would like to bring the experience to macOS too.

Command proxies
Currently, Swiftly can install multiple toolchains, but there's a single set of symbolic links that point to the toolchain that is in-use. The setting is global and you have to remember to use the correct toolchain for your project before you swift build or swift test your code. If you're working on multiple projects at the same time this can be error prone.

We plan on moving Swiftly to a proxy mechanism that allows you to specify the toolchain that you want to build, test, or compile like this:

swift build +latest # Installs the latest toolchain and uses that to build
swift test +5.10.1 # Uses the 5.10.1 toolchain to test (install if necessary)
clang +main-latest -o foo.o foo.c # Compile using the latest main snapshot toolchain

If you forget to specify the toolchain selector (with the '+') then the proxies will try to use the global default, much like how the symlinks work today. But, if you want to set a local default for a project just use it.

cd my/project
swiftly use 5.10.0 # Sets the default for the project to 5.10.0

The setting is stored in a .swift-version file in the project and you can share this with the rest of your team, updating it as you decide to adopt new toolchains.

Shell script migration to pure Swift
Currently, there is a shell script that performs a number of tasks, such as installing Swiftly, and installing system packages needed by the toolchains. You normally encounter this script when you follow the instructions using curl. We would like to move this logic into Swiftly itself, written in Swift.

Swiftly itself can run through its own installation process, such as detecting your Linux distribution, processor architecture, determining your shell (sh, bash, or fish), and updating your profile (PATH, environment variables). The same code runs independently of the delivery process, whether direct download, or a system package installer. Plus, it doesn't depend on having curl installed.

Since Swiftly is a user-level toolchain management tool we feel that it is out of scope for it to try to directly update your entire system, installing missing packages, and requiring your sudo password to do that. Instead, it will query for missing packages, if any, and give you the commands that can set up the necessary prerequisites. For most developers your system will already have the necessary packages and so things will just work. There's no need to prompt for a password to perform these checks.

Remove GitHub API dependencies
Swiftly uses the GitHub REST API at the moment to query for toolchain releases and snapshots. Also, this is how it detects self updates and downloads its latest version. However, there are limits to this API. For example, there are actual rate limits that can impact some users. It doesn't have the necessary metadata to determine if a particular release or snapshot is supported on your Linux distribution, or processor architecture leading to failed downloads.

Swiftly can switch to use swift.org itself to query for the necessary metadata. There's a much higher rate limit, which allows us to finally unveil the hidden list-available subcommand that allows you to perform queries for available toolchains that match your environment.

swiftly list-available 5.10 # Show me the 5.10.x releases for my system
swiftly list-available main-latest # Show me the latest snapshot releases for my system

Swiftly will also be published on swift.org. That's right, the same place where you get your toolchains is the place where you can also get your toolchain manager, Swiftly. This allows us to remove the dependency on the GitHub API completely, replacing it with queries to swift.org itself.

I hope that these new enhancements will help you with your Swift development and also make Swift even more accessible to newcomers. What do you think of them?

Cheers,
Chris

25 Likes
2

That's fantastic!

By the way, if Swiftly will be the standard installer for Swift, what do you think about swift sdk install and swift sdk remove?

I think that Swiftly can also be the best tool to handle Swift SDKs even if it becomes just a wrapper for swift sdk install and swift sdk remove because Swift SDKs, too, are published on swift.org.

2 Likes
3

Love to see Swiftly gaining macOS support and all the changes you proposed. Can't wait to use it to manage my toolchains!

Also big thanks to @patrick for creating this project and working on the initial implementation!

4 Likes
4

Overall I think these are positive steps and I'm pleased to see it making progress to becoming the default way to install Swift.

I have one thing I disagree with:

Swiftly's goal has always been make the installation of Swift seamless. Removing the step of installing dependencies would be a big step backwards for that. I strongly disagree with the statement that "for most developers your system will already have the necessary packages and so things will just work" - this does not align with the experience we've seen on the forums here, Stackoverflow and the Vapor Discord. Most people using Swift on Linux for the first time do not have the required dependencies installed and working them out and installing the right ones (even copy-pasting the output) is a significant barrier to entry. Additionally, swiftly is currently used to install Swift on Heroku and in CI systems where no dependencies are installed - that step would now break correct? So instead of having a single command to install and run everything we'd need to migrate to multiple commands, work out what OS we're using and then install the right dependencies, which is one of the specific problems Swiftly was designed to solve. It also doesn't line up with tools from other languages, like rustup.

12 Likes
5

Thank you for this suggestion. Can you raise this enhancement as an issue for further consideration on the GitHub project?

3 Likes
6

Thank you Tim for the feedback.

Now that I know that Swiftly is used in Heroku/CI systems that will be a use case that I'll be taking into consideration with the enhancements. I do expect that there will be some disruption though since the delivery mechanisms for Swiftly are about to change, such as the URL to curl to get the script using the current mechanism. That will be changing over to swift.org from a GitHub page. I think that the key will be to provide a soft landing with effective communication. Where do you think is a good place to provide guidance on migrating CI systems to the new Swiftly?

In terms of a seamless installation experience I expect that the most seamless will be through the system level package manager that's tightly integrated into the OS. If a user wants a Swift then being able to apt-get install swift or yum install swift will install the toolchain on the system so that all users get some stable release that works well in the context of the whole system and all users. The package's dependencies are encoded in the swift package and everything gets installed automatically and trusted as the system default.

But, the system level packages will sometimes lag behind the latest stable releases of the toolchains, and they often don't permit installing a variety of toolchains at the same time. These and other problems are covered in Swiftly readme in more detail.

Swiftly is providing a way to overcome these limitations at a user level like the other similar language tools like rustup, and nvm. These all have a slightly more involved guided installation process than system packagers. All three must update the user's profile (path, environment variables, etc.) and so they provide instructions to the user that they must either close/reopen their terminal or source the environment file manually. Swiftly also prompts the user right away to confirm that they want the standard or custom installation, overridable with a special '-y' flag. If someone tries to install swiftly on an unsupported distribution then the user is prompted to pick the closest one. I assume that CI type systems must have built into their automation scripts, adjusted from the instructions in the README.

Due to the way that the Swift toolchains are structured today on swift.org there is an additional requirement over rust/node/go that certain system-level dependencies be present on the system. Today it prompts for a sudo password to ensure that they are all installed, which some (myself included) find breaks the mental model of user-level tool. That breakage of the mental model can erode trust. What's this tool even doing? Why? Do I need to be concerned about the security here?

So, we feel that this is a bit like prompting for the install confirmation, or providing the guidance that the user should source the environment file, or re-open the terminal window. The system level package install commands can be provided as guidance, but only if they are applicable. So, if the packages are already there then don't even mention it. I understand now that our assumption that most systems already have the necessary packages is not accurate, especially for CI systems.

For CI systems, there was some process that discovered and added the "-y" flag to the automation scripts or else the installation might block forever on a prompt. Someone probably tried it manually first, and then adjusted the automation to suit the needs of the Swiftly installer. I expect that they do the same thing with the user profile path updates, and also the system package installs.

Something that I think we could do as a usability enhancement to this flow is to detect whether swiftly is running as root and then do the package installs behind a prompt, proceeding right away if the "-y" flag is given. I think that as long as it is transparent about what's happening and why, this could work within the user-level tool mental model. If there's no missing system dependencies then there is no prompt at all. What do you think about this as an option? Maybe it helps with the CI workflow at least?

7

I agree with the other SSWG folks -- we should make installing very simple and not regress this functionality. It's important for newcomers to Swift to not immediately be bombarded with Swift's build issues and system dependencies they really don't care about in their "I have 10 minutes to try swift hello world".

That flow is good, but I don't think it's necessary to "detect if running as root"?

Specifically, if we can detect dependencies (or their absence) without being root, why not provide the same type of "You seem to be missing dependencies, install them (show what it would install)? [yes]/no". Tools like apt info and apt list don't require root for the querying of information, so that seems possible. This experience isn't unlike homebrew.

And yes you can always just say no to such tool suggestion and satisfy dependencies yourself.

What do you think about such approach?

8 Likes
8

Forums and release notes are the best place - we already picked up the org move when CI started failing (unfortunately GH pages don't redirect) and that was an easy fix.

This would be a nice long term solution but is not something we currently have. I imagine it will require a fair bit of work from @mishal_shah and the Platform Steering Group to get those shipped. As you mentioned, we then also have the issue of the repositories generally being pretty behind in terms of updates. It's also notable that neither Go or Rust rely on system level package managers, because of the degraded experience when it comes to shipping version updates among other issues.

You can see an example of Swiftly being used in CI in this GH action - the big advantage of Swiftly is that we can run that on all different Linux platforms and not have to worry about which one we're running on, as Swiftly takes care of all of that.

I'd say this is more of a messaging issue. If I install a Swift toolchain on macOS I get prompted for my password to allow it, the installation on Linux isn't really any different. What we probably can do is be cleverer about detecting dependencies and the environment we're runniging in and only prompt the user for the password if needed

Ah exactly yes this is a good goal.

Overall, as Konrad mentions, Swiftly will become the de facto way for people to install Swift on their systems and we should do everything to make it as seamless as possible for those that want to try out Swift. If it's "run this one command, it will install everything for you, and you can start coding right away" that's a far better experience than "missing dependency, please install libpackage-dev". Some people are likely to give up pretty quickly in this case. After all if the tool knows what dependencies are missing, why can't it install them

7 Likes