Which clang package should we install?


(Ian Partridge) #1

Currently our Swift 4.2 Dockerfiles install a non-default clang. The 16.04 Dockerfile installs Clang 3.8, and the 18.04 Dockerfile installs Clang 3.9.

I'm starting to look at what our 5.0 Dockerfiles should contain.

The install instructions on swift.org/download just say apt-get install clang - so, whatever the distribution default is.

On Ubuntu 16.04 that will install Clang 3.8 and on Ubuntu 18.04 it will install Clang 6.0.

It looks to me like Swift 5 is based on LLVM 7 so I wondered if that meant we should install Clang 7 in our Dockerfiles?

Clang 7 is available in the Ubuntu 18.04 updates repo, however it is not packaged for Ubuntu 16.04.

Question: does it matter which version of Clang we install? Is it essential we install Clang 7 for Swift 5? (if so, I don't think this is possible on Ubuntu 16.04).

Meta-question: Why do we need to install Clang at all - shouldn't the download from swift.org come with everything Clang-related it needs?

CC: @tanner0101 @tomerd @johannesweiss


(Ben Langmuir) #2

Today's toolchain development snapshot for master is the first one to ever contain a working clang installation. CC @Alex_L


(Alex L) #3

The toolchain snapshots for master contain clang for both macOS and Linux since

landed.

I'm trying to cherry-pick it into swift-5.0 as well, but the tests haven't passed yet so it's not yet merged:

So hopefully once that PR lands you won't need to install another clang with swift-5.0.


(Ian Partridge) #4

Oh wow, thank you both! I tried out the latest master snapshot and this is what I see. I'm trying to find out what the minimal set of Ubuntu packages required are:

$ docker run -it ubuntu:bionic /bin/bash
# apt-get update && apt-get install -y wget
# wget https://swift.org/builds/development/ubuntu1804/swift-DEVELOPMENT-SNAPSHOT-2019-02-14-a/swift-DEVELOPMENT-SNAPSHOT-2019-02-14-a-ubuntu18.04.tar.gz
# tar -zxf swift-DEVELOPMENT-SNAPSHOT-2019-02-14-a-ubuntu18.04.tar.gz --directory / --strip-components=1
# cd /root && swift --version
swift: error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory
# apt-get install -y libatomic1 && swift --version
swift: error while loading shared libraries: libedit.so.2: cannot open shared object file: No such file or directory
# apt-get install -y libedit2 && swift --version
Swift version 5.0-dev (LLVM 1fb4269ad2, Clang cba11ceb9f, Swift 6159ba810d)
Target: x86_64-unknown-linux-gnu
# swift package init --type executable
/usr/bin/swift-package: error while loading shared libraries: libsqlite3.so.0: cannot open shared object file: No such file or directory
# apt-get install -y libsqlite3-0 && swift package init --type executable
/usr/bin/swift-package: error while loading shared libraries: libcurl.so.4: cannot open shared object file: No such file or directory
# apt-get install -y libcurl4 && swift package init --type executable
/usr/bin/swift-package: error while loading shared libraries: libxml2.so.2: cannot open shared object file: No such file or directory
# apt-get install -y libxml2 && swift package init --type executable
/usr/bin/swift-package: error while loading shared libraries: libbsd.so.0: cannot open shared object file: No such file or directory
# apt-get install -y libbsd0 && swift package init --type executable
Creating executable package: root
Creating Package.swift
Creating README.md
Creating .gitignore
Creating Sources/
Creating Sources/root/main.swift
Creating Tests/
Creating Tests/LinuxMain.swift
Creating Tests/rootTests/
Creating Tests/rootTests/rootTests.swift
Creating Tests/rootTests/XCTestManifests.swift
# swift run
/root: error: manifest parse error(s):
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "CoreFoundation.h"
         ^
/usr/lib/swift/CoreFoundation/CoreFoundation.h:25:10: error: 'sys/types.h' file not found
#include <sys/types.h>
         ^
/usr/lib/swift/CoreFoundation/CFStream.h:20:10: note: while building module 'CDispatch' imported from /usr/lib/swift/CoreFoundation/CFStream.h:20:
#include <dispatch/dispatch.h>
         ^
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "dispatch.h"
         ^
/usr/lib/swift/dispatch/dispatch.h:32:10: note: in file included from /usr/lib/swift/dispatch/dispatch.h:32:
#include <os/generic_unix_base.h>
         ^
/usr/lib/swift/os/generic_unix_base.h:24:10: error: 'sys/param.h' file not found
#include <sys/param.h>
         ^
<unknown>:0: error: could not build C module 'CoreFoundation'

At this point I'm not sure what provides the headers that are missing. Any ideas?


(Ben Langmuir) #5

Those smell like something that would come from build-essential or one of its dependencies (maybe libc-dev ?). Not sure.


(Ian Partridge) #6

Thanks, it was libc6-dev. Here's where I am now:

$ docker run -it ubuntu:bionic /bin/bash
# apt-get update && apt-get install -y wget
# wget https://swift.org/builds/development/ubuntu1804/swift-DEVELOPMENT-SNAPSHOT-2019-02-14-a/swift-DEVELOPMENT-SNAPSHOT-2019-02-14-a-ubuntu18.04.tar.gz
# tar -zxf swift-DEVELOPMENT-SNAPSHOT-2019-02-14-a-ubuntu18.04.tar.gz --directory / --strip-components=1
# apt-get install -y libatomic1 libedit2 libsqlite3-0 libcurl4 libxml2 libbsd0 libc6-dev
# swift package init --type executable
Creating executable package: root
Creating Package.swift
Creating README.md
Creating .gitignore
Creating Sources/
Creating Sources/root/main.swift
Creating Tests/
Creating Tests/LinuxMain.swift
Creating Tests/rootTests/
Creating Tests/rootTests/rootTests.swift
Creating Tests/rootTests/XCTestManifests.swift
# swift run
clang-7: error: invalid linker name in argument '-fuse-ld=gold'
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)

Hmm so we need gold. Ok but shouldn't Swift come with the linker it needs...

# apt-get install -y binutils
# swift run
/usr/bin/ld.gold: error: cannot open crtbeginS.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtendS.o: No such file or directory
/usr/bin/ld.gold: error: cannot find -lstdc++
/usr/bin/ld.gold: error: cannot find -lgcc_s
/usr/bin/ld.gold: error: cannot find -lgcc
/usr/bin/ld.gold: error: cannot find -lgcc_s
/usr/bin/ld.gold: error: cannot find -lgcc
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)

Hmm now I need a load of GCC libraries? Am I on the right path here?


(Neil Jones) #7

Even though installing clang in technically not needed with this build when you install clang there are many dependencies that clang needs that are installed when you install clang.

The following NEW packages will be installed:
  binfmt-support binutils binutils-common binutils-x86-64-linux-gnu clang clang-6.0 gcc-7-base lib32gcc1 lib32stdc++6 libasan4 libatomic1 libbinutils libc-dev-bin libc6-dev libc6-i386
  libcilkrts5 libclang-common-6.0-dev libclang1-6.0 libffi-dev libgc1c2 libgcc-7-dev libgomp1 libitm1 libjsoncpp1 libllvm6.0 liblsan0 libmpx2 libobjc-7-dev libobjc4 libomp-dev libomp5
  libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libquadmath0 libstdc++-7-dev libtinfo-dev libtsan0 libubsan0 linux-libc-dev llvm-6.0 llvm-6.0-dev llvm-6.0-runtime manpages-dev
  python python-minimal python2.7 python2.7-minimal
The following packages will be upgraded:
  gcc-8-base libgcc1 libstdc++6

The "Swift Clang" still needs most of, if not all of, the same dependencies to be installed. I have found that the simplest way to achieve this is to just install clang as we have before.


(Ian Partridge) #8

Thanks Neil. For a Docker image we don't want any unnecessary packages though.

I was able to get a "Hello World" Swift program to build and run by adding libgcc-7-dev and libstdc++-7-dev to my previous list. So my complete list so far is:

libatomic1 libedit2 libsqlite3-0 libcurl4 libxml2 libbsd0 libc6-dev binutils libgcc-7-dev libstdc++-7-dev.

Plus libpython2.7 is required for the REPL.

By the way, the resulting "Hello World" binary links 63 shared libraries, which kind of blows my mind!


(Ian Partridge) #9

It's worth mentioning that my Docker image containing just the packages needed to build and run "Hello World" is 254MB. The same image with apt-get install clang added to it increases in size to 630MB. So I think it's definitely worth figuring out exactly which packages we need, and which we don't.


(Neil Jones) #10

Thats a big difference. Definitely worth removing all the unnecessary installs.
I am looking at creating a swift-essential package similar to build-essential so we can sudo apt-get install swift-essential.
Also a swift-essential-dev which will include extra dependancies needed for server-side Swift such as Kitura, Vapor, Perfect, Swift-Nio etc.


(Ian Partridge) #11

I think my preference would be for docker pull swift:latest to get an image which comes "batteries included", with everything you need to build and run Swift programs. It should be aimed at people discovering the language for the first time, who want to do things including run the REPL.

Different server-side frameworks may have different requirements, plus users of them may need different packages depending on which databases etc. they are using.

I would also like us to have a separate "swift runtime" image which contains only the packages needed to run an already built Swift binary. In other words, you would build your app with the build image, and run it with the (smaller) run image.


(Ian Partridge) #12

I'm pausing my work on this until https://github.com/apple/swift/pull/22273 has landed, because it will make a big difference to which packages we need to include in the Swift 5 Docker image.


(Alex L) #13

The PR just landed with the Linux support for clang & the libc++ headers alongside.


(Ian Partridge) #14

https://github.com/apple/swift/pull/22273 has landed - thanks to @Alex_L and @mishal_shah!

There is a test toolchain for Ubuntu 16.04 available at https://ci.swift.org/job/swift-PR-toolchain-Linux/167/artifact/branch-swift-5.0-branch/swift-PR-22273-167-ubuntu16.04.tar.gz

I downloaded this and took it for a spin:

$ docker run -it ubuntu:xenial /bin/bash
# cd /root && apt-get update && apt-get install -y libatomic1 libedit2 libsqlite3-0 libcurl3 libxml2 libbsd0 libc6-dev binutils libgcc-5-dev libstdc++-5-dev libpython2.7
# apt-get install -y wget && wget https://ci.swift.org/job/swift-PR-toolchain-Linux/167/artifact/branch-swift-5.0-branch/swift-PR-22273-167-ubuntu16.04.tar.gz
# tar -zxf swift-PR-22273-167-ubuntu16.04.tar.gz --directory / --strip-components=1
# swift package init --type executable
Creating executable package: root
Creating Package.swift
Creating README.md
Creating .gitignore
Creating Sources/
Creating Sources/root/main.swift
Creating Tests/
Creating Tests/LinuxMain.swift
Creating Tests/rootTests/
Creating Tests/rootTests/rootTests.swift
Creating Tests/rootTests/XCTestManifests.swift
# swift run
[2/2] Linking ./.build/x86_64-unknown-linux/debug/root
Hello, world!

So I think this looks good!

Once there are toolchains on swift.org for Ubuntu 18.04 I will check that too. Then we can start building the Dockerfiles for Swift 5.


(Argyrios Kyrtzidis) #15

I'm wondering if it is necessary to require libgcc-5-dev libstdc++-5-dev now that the toolchain contains clang & libc++ ? @Aciid are the flags coming from swiftpm ? Any thoughts ?


(Ian Partridge) #16

Compiled binaries also link against the system libc++:

# ldd ./helloworld | grep stdc
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1b2bc1c000)

So it's not just the toolchain that requires it.


(Ankit Aggarwal) #17

Is that a binary compiled without SwiftPM? SwiftPM adds -lstdc++ if there are cpp files in a target. If clang in the toolchain means we can use -lc++ on linux then we can make that change in SwiftPM (as long as there is no fallout for existing packages).


(Ian Partridge) #18

Ah yes, that was a swiftc helloworld.swift. It didn't occur to me that the linking might be different if SwiftPM did it.


(Ankit Aggarwal) #19

Looks like libswiftCore.so depends on stdc++:

$ ldd /lib/swift/linux/libswiftCore.so

libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fb17ab01000)


(Argyrios Kyrtzidis) #20

I see, so it's more about getting swift to build with the same clang that we bundle in the toolchain.