RPM and Debs for Swift: Call for the Community

Development Compiler
21

AFAIK specifically Rust has no ABI, but I think Linux C++ does. Rust (and Go) are little less affected because they (I think) have no runtime component. There is a difference between hosting the compiler and hosting runtime/standard libraries (and packages).

Generally pretty much all the common languages have ABI stability in some form. You wouldn't want your Python server to stop working because you have Python 2.4 installed instead of 2.5 (which is what happens w/ Swift right now).

This also affects the package manager a lot. E.g. ideally you'd want packages to result in properly versioned Linux shared libraries.
This is part of my ask for not just considering server side Linux, which usually just links everything together in a big Docker blob or similar.

It generally "just" means not to break the ABI and commit to keeping it stable for a certain version. Since almost all work here is AFAIK done by Apple people, it would need a commitment from there.
It ends up being actual work because you might need to find workarounds for older versions (e.g. the current thing being an async/await runtime for older Swift versions).

22

C++ on Linux doesn't provide ABI stability. There is some attempt to preserve stability, but it does break periodically. The biggest one in recent history was C++11 which broke ABI on std::string and std::list. For most distributions that is not an issue because everything is built at a single point so it appears to allow things to work. The final result of this is a dual-ABI setup that is even more fragile :-(.

IIRC, binutils also introduced ABI breakages (via gold and bfd) in handling of unwind tables though it has been long enough that my memory is far too foggy for this to be anything more than anecdata.

Although, I do admit that the ABI handling is far better now than it was with pre-GCC-4.x. And they have also introduced __attribute__((__abi_tag__(...))) in GCC to support multi-ABI libraries.

1 Like
23

Yes, the community can achieve it. As with all things it requires getting clear commitments from the core team on what they need, and then knocking them off.

In this case my understanding is that the goals are not that high: that the Swift ABI is probably stable on Linux, but no-one is enforcing that today.

They don’t, but they don’t need to, because they don’t have things like swift-corelibs-foundation, which are Swift libraries distributed with Swift as binaries, not as source.

24

Perhaps I am misremembering, but I could have sworn when @Ron_Olson was making his Fedora package for the first time there being some gotcha about having the "system" LLVM/LLDB and Swift installed at the same time.

I tried looking at the swift-lang.spec to see if there was some change of path or mutual exclusion of the two packages but nothing is jumping out at me.

25

Yes, the solution was that I basically dump everything into /usr/libexec/swift with swift, swiftc, and sourcekit-lsp being symlinked into /usr/bin; made packaging it so much easier. :slight_smile:

With this configuration the system clang and lldb do not conflict with the Swift-specific ones; because Swift uses relative paths (i.e., swift looks for lldb in the same directory), it all "just works".

26

This may be a naive/impertinent/stupid question but would we ever be able to upstream the LLVM/LLDB changes so that it would more seamlessly integrate with the ecosystem? Or are we effectively at a point where there is just a permanent fork of those tools just for Swift?

1 Like
27

The fact that the ABI can change any time anything changes makes this one of the reasons why I think we should sit down and give explicit guarantees about what we do, since changing minors here is liable to build binaries built with the package as it updates (even binaries that are built from it that may want to depend on it.)

28

IIRC It is more complex than that since there are parts of the runtime that even on Linux have unneeded code to support objc Darwin things. This can be eliminated resulting in an improved runtime. I don't remember all the details though. @John_McCall @Joe_Groff @Mike_Ash @Arnold do you all remember?

29

That being said, long term we should want for Swift to be as "boring" and "normal" as possible. To me that means long term we are going to want to be able to have swift built upon llvm.org packages that way we don't have to do what you mentioned above.

That being said, I do believe that is some time away since LLDB's language plugin API in llvm.org will need to be significantly improved to unblock swift LLDB linking against llvm.org libraries. But my memory could be incorrect. @Dave_Lee knows more about this.

30

We've eliminated the most obvious overheads, like class metadata having a bunch of ObjC fields. The biggest remaining ObjC-forced complexity that I know of is that the value-witness table pointer field in type metadata precedes the address point, but that's not really a cost, it's mostly just build-time complexity.

1 Like
31

I don’t think that necessarily matters: we don’t have to remove that if we don’t want to, we could always stabilise an ABI that has unnecessary cruft in it. Not saying I want to, just noting that it is an option.

1 Like
32

Hey what’s with supporting Snapcraft?
Or any package manager so it’s easy for the developer to install!

33

@futurejones this is awesome!

2 Likes
34

@futurejones It would be great to figure out how we can setup something similar on swift.org. Are the sources for swiftlang.xyz available for the community view? If yes, where can we find them?

1 Like
35

@mishal_shah, setting up something similar to https://swiftlang.xyz on Swift.org would be fairly straight forward but it is not a simple process and there are number of separate items required that are then combined to create the completed repository.
Here is an overview of what is needed.

The Repository Server.

The repository needs a dedicated server, preferably bare metal, running Ubuntu 20.04.
The hardware does not need to be too large or powerful (>2Ghz quad core cpu, 8Gb ram) but it will need ample fast ssd storage and plenty of network bandwidth.
The following software is used to create, run and maintain the repository.

  • nginx
  • wget
  • curl
  • gpg
  • dpkg-sig
  • reprepro

Repository Domain Name

The repository requires a fully qualified domain name and ssl certificate. This is usually a sub-domain of the organization e.g. archive.swift.org
DNS will need changed to point the sub-domain to the repository server.
SSL certificates can be obtained and installed using certbot and letsencrypt

Creating Signing Keys

gpg is used to create a repository signing key. The public key then needs to exported and made available in the root of the repository for users to download and install.

Creating the Repository

reprepro is used to create the repository. This involves setting the required directory structure for each distribution e.g. ubuntu, debian etc. and adding a config file.

Debian Package Creation

This is a whole other area for discussion. Package creation should not be taking place on the repository server.
The main and key point that we need to know for the repository is the package name.
This needs to be swiftlang and not swift-lang.
There has been a registered debian package called swift for many years, long before Swift the programming language even existed. If we use swift-lang the hyphen indicates that it is part of the existing swift package family.
There are many swift- packages currently available for ubuntu/debian that have nothing to with Swift the programming language. -

  • swift
  • swift-bench
  • swift-doc
  • swift-object-expirer
  • swift-account
  • swift-container
  • swift-object
  • swift-proxy

I am using fpm for debian package creation from swift.tar.gz files. It works very well and can be added as build step on the swift ci.

Uploading Packages

First the deb packages need to be uploaded to temp directory on the repository server.
reprepro is then used to add the package to the repository. reprepro automatically takes care of all code signing and creates any extras directories may be needed.

Installing and Using the Repository

To use the repository you need to do 2 things -

  1. add the repository signing key.
  2. add the repository url to the apt sources list.

This can be done manually or using a simple bash install script

References

https://manpages.ubuntu.com/manpages/focal/man1/gpg.1.html
https://wiki.debian.org/DebianRepository
https://wiki.debian.org/DebianRepository/SetupWithReprepro
https://fpm.readthedocs.io/en/latest/

6 Likes
36

We have created a new repository for RPM/Debs work.

Repository: GitHub - apple/swift-installer-scripts

Let's start by creating pull request for the spec files on this new repository, and we can work together on getting CI setup on ci.swift.org.

11 Likes
37

Moving the RPM install path discussion here from Update the README with Linux package / installer info (RPM/DEB) by shahmishal ¡ Pull Request #37 ¡ apple/swift-installer-scripts ¡ GitHub

We need to decide between /usr/, /opt, and /usr/libexec/.

Install Location
/usr Packages included in swift package will conflict with other packages (clang, llvm, lld, lldb, and more)
/opt Use this location until we resolve the conflicts in /usr dir. However, for Fedora we will need to select different location
/usr/libexec Install the swift package here and symlink specific binaries into /usr/bin

In my opinion we don't need to have the same path for all distribution.

I propose Ubuntu 20.04, Ubuntu 18.04, CentOS 7, 8, and Amazon Linux 2 should install the package in /opt and Fedora in /usr/libexec until we resolve the conflicts.

3 Likes
38

I would suggest to keep all RPMs behave aligned — which means /opt for Ubuntu & Debian, /usr/libexec for CentOS, Fedora & Amazon Linux.

These two “general” platforms often behave differently from each other, but distributions within each of them have much smaller differences. Dividing the platforms into these two groups is most straightforward, and it also just won’t make Fedora looks like an “exception”.

1 Like
39

We are making progress on Swift RPM packages and we are looking for feedback.

swiftlang is installed at /usr/libexec/swift and swift , swiftc and sourcekit-lsp are symlinked into /usr/bin/

IMPORTANT INFO

  • Links below are for testing purposes only
  • Do not use these links in production
  • Links might stop working without any notice
  • Official rpm packages will be signed

CentOS 7

$ curl https://download.swift.org/test-rpm/repo/centos/releases/7/swiftlang.repo > /etc/yum.repos.d/swiftlang.repo 
$ yum install epel-release 
$ yum install swiftlang

CentOS 8

$ curl https://download.swift.org/test-rpm/repo/centos/releases/8/swiftlang.repo > /etc/yum.repos.d/swiftlang.repo 
$ yum install epel-release 
$ yum install --enablerepo=powertools swiftlang

Amazon Linux 2

$ curl https://download.swift.org/test-rpm/repo/amazonlinux/releases/2/swiftlang.repo > /etc/yum.repos.d/swiftlang.repo 
$ yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm 
$ yum install swiftlang
9 Likes
40

Check out Swift 5.6 Blog Post - Swift.org - Swift 5.6 Released!

Thank you to everyone who helped with Swift 5.6 experimental RPM packages!

We also provide RPMs for Amazon Linux 2 and CentOS 7 for experimental use only . Please provide your feedback.

Use the instructions below for RPM installation:

Amazon Linux 2

$ curl https://download.swift.org/experimental-use-only/repo/amazonlinux/releases/2/swiftlang.repo > /etc/yum.repos.d/swiftlang.repo
$ amazon-linux-extras install epel
$ yum install swiftlang

CentOS 7

$ curl https://download.swift.org/experimental-use-only/repo/centos/releases/7/swiftlang.repo > /etc/yum.repos.d/swiftlang.repo
$ yum install epel-release
$ yum install swiftlang
3 Likes