RPM and Debs for Swift: Call for the Community

Currently Swift on Linux is distributed via tarball and Docker, and we would like to start supporting RPM and Debs officially on swift.org. There are multiple community members who have already created RPM and Debs packages, and we would like to combine the efforts.

We are looking for volunteers to help build RPM and Debs packages.

Next steps:

  • Identify active RPM/Debs packages for Swift
  • Design an approach for distributing RPM/Debs
25 Likes

This is great to finally see coming together! I'm definitely interested in this. I've previously setup a deb repository for Swift. It doesn't entirely conform to the Debian packaging requirements (namely, it uses the same path setup as macOS and Windows rather than placing the binaries in /usr/local or /opt). I think that can we can address that fairly easily. Getting the infrastructure setup to generate the package and push them to a location is the more involved part.

I think that longer term improvements will be enable integrating this process and unify the behaviour across the platforms. Until then though, we should be able to automate the steps as a post-build step.

2 Likes

+1, I have an RPM recipe around which installs to /opt/swift-$version.

one interesting aspect here is lack of ABI on Linux and how we want to address swift version updates in that context.

5 Likes

cc @server-workgroup @millenomi

1 Like

For RPMs, I have a little info I can share. :upside_down_face:

For folks unfamiliar with the Fedora/RHEL-side of things, I've cobbled together some background:
RPM is the package type for Red Hat-based distros which include

  • Fedora
  • CentOS
  • Oracle Unbreakable Linux
  • Red Hat Enterprise Linux
  • Other flavors that have sprung up due to CentOS being deprecated

Every distro that isn't Fedora can install Swift via the EPEL (Extra Packages for Enterprise Linux). EPEL is typically not enabled by default so the system administrator has to explicitly enable it.

Fedora has Swift available out-of-the-box via sudo dnf install -y swift-lang which will also install its dependencies. Within Fedora, there are always three, sometimes four active versions:

  • Rawhide, which is is always the latest, cutting edge version which will typically have the latest kernel soon after it's released. It is where packages are typically released and updated first. This is the permanent beta release of, well, everything.
  • Current, which is the, well, current version (as of right now it's Fedora 34). It tracks behind Rawhide but is still typically very up-to-date with the kernel, packages, etc.
  • Previous, always the Current -1 version which will get three months of updates, and then it will be cut off and no longer receive anything

A typical release of Swift to the Fedora ecosystem means providing it for all four environments, which can sometimes be tricky because Rawhide may introduce problems that have to be resolved (e.g., I'm currently working on putting a PR together to remove Cyclades from the LLVM-part of Swift because it was removed in the latest kernel). Paradoxically, while Rawhide is the most aggressive, EPEL is the most conservative and some packages lag behind Fedora significantly (e.g., CMake) which requires some creative workarounds (for example, I download and build my own version of CMake and set it at the front of the path before building Swift).

One thing that is non-negotiable in the Fedora/RHEL environment is that every package that is offered via the official repositories must be built from source using their build system; binary blobs are not allowed. That said, it's possible to download an RPM file from anywhere and install it manually the downside to this is that there is no automatic updating mechanism so the package will not be automatically updated (which has its pros and cons).

The TL;DR is that building a Swift RPM requires a fair amount of constant tweaking between versions of Swift and versions of Fedora (with Rawhide being the proverbial canary-in-a-coal mine; what happens there ultimately filters down to Linux distros everywhere).

if anyone is interested in the gory details, I maintain a GitHub repo of what I use to create the RPM file that is basically my current working state.

I'm happy to work with everyone to make Swift more accessible to Linux users.

Ron

9 Likes

Thank you for bringing this up Mishal! I think this is very important in terms of making Swift easier to use on Linux in the manner in which Linux users expect software to be distributed! +1 from me!

2 Likes

Me and @tomerd discussed this privately a little. My intuition here is that if we distribute rpms/debs as standalone installable packages, we are saying it is the developer’s responsibility to compile their application to use those versions of Swift, which is not too dissimilar from the approach we have with the Docker image; but if we provide a package source of some kind, or want these packages to be published in an existing package source, then they may run afoul of ABI issues introduced by compiler changes.

One thing we can do in that case is provide separate packages for each minor of Swift. For example, rather than swift-lang, we would have swift-lang5.4 and swift-lang5.5. @tomerd mentioned separate directories in /opt for each such version. Each of these packages could still be supported with monthly dot fixes since those only ever pick up corelibs patches — I would assume here it just becomes a matter of avoiding ABI-breaking changes, though we may have to turn on library evolution to make this a hard guarantee.

Otherwise, I’m open to alternatives.

5 Likes

I do not believe library evolution is semantically meaningful on Linux. To put it more concretely, it is my understanding that Swift on Linux has no stable ABI guarantees, and so while you can enable the flag it is not guaranteed to produce the outcome we want.

With that said, I agree that we are more likely to get an ABI-stable runtime if we go down that road.

At this moment I think we have an absence of good options. Ultimately we rely on the idea that the ABI of the corelibs will not change without any guarantee that that will happen (or, so far as I know, any tooling to prevent it). That's necessarily a dangerous place.

Probably the best option is to do two things in parallel. First, just do our best: enable library evolution for the corelibs and start shipping the packages, gated by Swift version and trust that patch number updates won't cause problems. Secondly, start pursuing the stabilising of the Swift ABI on Linux.

Until we do the latter we are simply on unsupported ground.

1 Like

While downloading a deb package from swift.org would be certainly progress you also might consider building the Debian way like llvm: https://tracker.debian.org/pkg/llvm-defaults.
Since a lot of distributions pull from Debian (Ubuntu, Mint, ...) you would reach a lot of people, and installing would be very easy (sudo apt install swiftlang).

I'm using those for some of my ARM docker images, @futurejones seems to maintain them well. Presumably they would just work for all archs, but no idea: swift-arm - Repositories · packagecloud

P.S.: It feels more like step 3, but once we got Swift on the systems, more work probably needs to be done on resources, to get Swift apps on the respective FHS required (I once did all that work for OpenGroupware, a heavily NSBundle based system, changes have been quite a lot). And as a step 9 maybe even provide the tooling to do a swift dpkg build etc, which I once started but never finished.

P.S.2: I'd be very happy if things are not only approached from a Linux server perspective (which people often have for understandable reasons). It would be very nice to use Swift for Linux desktop apps etc as well.

3 Likes

That I think is what @mishal_shah was talking about. Something like this: https://apt.llvm.org. But I could be incorrect.

My hope is that we make it frictionless as possible and make it easy to install swift using the natural package method for the platform. If someone can just add an apt source and then apt-get off of swift.org... it fits with the expectation and follows the well trodden path of LLVM.

This requires adding a repository for alien packages. Why not make it a native Debian package, built according to Debian's rules on a Debian server, with packages.debian.org hosting the binary? That's as frictionless as you can get. :slight_smile:

2 Likes

I can't speak for Debian, but for Fedora that requires an account, which I have and use to publish Swift. If there is to be a more "official" presence then presumably someone more official than some rando like me would be set as the project owner, who can then designate administration to other trusted folks to push out new versions, etc.

I would love it if we added official ARM64 builds on swift.org. @futurejones has done fantastic work maintaining the port, and according to his repository, the build machine (some beefy 80-core server) is provided by the Works on Arm project, so it would be great to have these integrated in to the official build/release process if that is at all possible.

One of the things I think Swift desperately needs to improve is how we communicate platform support. Just now, I went to rust-lang.org, clicked the enormous "GET STARTED" button, clicked "Other installation methods" and found a list of official toolchains and binaries to download. Checking out their platform support page, there is another clear and comprehensive list of ports arranged in tiers of support. Hey, look at that - there's a port to the Sony PSP from 2004; it's a tier-3 port and doesn't support the standard library, but that's cool to know! I think I still have one of those somewhere...

Try to do the same thing on Swift.org. You won't find any mention whatsoever about ARM linux support. Nothing about the Android port. Nothing about WebAssembly. Nothing about OpenBSD. You're left to search for random forum/blog posts or GitHub repositories on your own. Even the Windows port is presented as an individual hobby project more than an official Swift.org toolchain distribution (a fact which is emphasised by the compiler printing compnerd.org Swift version 5.5-dev (LLVM ..., Swift ...). I don't know what people new to the project are supposed to make of all of that, but to my eyes it seems disorganised, fragile, and frankly unprofessional.

We have the same issue with the VSCode plugin, which developers here have been complaining about literally for years. The Swift project seems to completely refuse to endorse its own projects, instead preferring that "random" developers distribute critical tools from their personal accounts. It doesn't inspire confidence, and it's just... baffling.

I can only assume that Apple is trying to avoid any implication of endorsement for legal reasons, but there's no reason "Swift.org" or the "Swift open-source project" couldn't have more of a presence. I think it hurts the project, especially when you compare it to Rust or any other language. You get the feeling that anything other than the Apple ports is an unstable hack.

I hope that this signals a move to rectify that situation.

13 Likes

@mishal_shah I can help with the .deb packaging for Debian/Ubuntu distributions. I have been building and distributing .deb install packages for Swift on Arm64 for a couple of years now.
These are available for manual install via swift-arm64 github repo or using the swift-arm package repository
Packages for the x86/amd64 versions can easily be generated from the existing swift.tar.gz files from swift.org. This can be done using a separate Jenkins pipeline on the Swift CI server or interagated into the existing swift build jobs.
I have just done a test build and upload for Ubuntu 20.04 as an example.
http://futurejones.xyz:8080/view/5.4%20Release/job/swift-apt-builder/job/ubuntu-20.04-swift-5.4-release-amd64/
https://packagecloud.io/swift-arm/release/packages/ubuntu/focal/swiftlang_5.4.1-01-ubuntu20.04_amd64.deb
Happy to help in any way I can.

4 Likes

I don't know the current rules, but I think for Debian you used to need a sponsor from the team to get a package in.
Missing ABI stability is almost certainly going to be a problem for Debian upstream support.

Yep, I needed to be sponsored for Fedora as well.

Though I'd recommend as @gonsolo is suggesting that it be available via the default repositories for each platform; being able to bring up a machine and install Swift via apt-get or dnf is so frictionless, and is pretty much how every language and tool I use is delivered. It also makes bringing up containers or VMs with ansible cleaner because there's no need to add extra steps for downloading the image to somewhere, then explicitly installing it.

1 Like

What I meant to conclude with is that having them in the official repos could be a goal, the road to get there is certainly long and there's a lot that needs to be worked out to make that happen.

In the meantime, having a common build script(s) between a downloadable RPM/DEB and the official ones would be a positive step; the user can choose to install via the official repositories provided by the likes of me for Fedora, or they can download the RPM directly from swift.org for some other RPM-based platform that I don't handle (e.g., SUSE).

Knowing nothing about what it takes to stabilize the Swift ABI for Linux. Is that even a thing the community could achieve?

I expect the only reason why there is a stable ABI for Darwin is because Apple paid skilled/knowledgeable developers to make it so. Are there developers with those equivalent skills in the community who are able to volunteer or be paid to achieve the same?

Do other similar languages, like Rust, make ABI guarantees?