Dockerized Swift, Build Times and Other Notes Regarding CI


(Fastmail) #1

Hi all,

I’ve been working on running Swift applications and building the Swift toolchain in docker containers. I’d like to share some thoughts and notes I’ve made, since they may be of interest to others. Especially to those of you stubbornly clinging to 10.10 on 2013 era Macbook Airs which have all the horsepower of a donkey.

For those of us who are unfamiliar with docker, it’s essentially an alternative to virtual machines, where the host shares its kernel with the container. Consequently the VM overhead is removed entirely, but you have a controlled environment that is versionable, space efficient and full of awesome, basically.

Let’s start with the headline:

Renting resources from Amazon using docker-machine to build Swift.

The magic sauce is:

docker-machine create --driver amazonec2 --amazonec2-instance-type c4.8xlarge --amazonec2-request-spot-instance --amazonec2-spot-price 1 --amazonec2-root-size 120 --amazonec2-access-key BLANK --amazonec2-zone e --amazonec2-secret-key BLANK --amazonec2-vpc-id BLANK aws

Docker-machine manages the underlying connection between the `docker` command line tool and the daemon running on the host. In this case, we can provision a machine in the cloud with 36 cores, 10gbis fibre and an enormous SSD. It will cost no more than a dollar an hour (amazon spot pricing lets you bid on capacity with a max bid, here set to $1). All our commands will execute there, entirely removing bandwidth, capacity or power issues on our end. (Yes, it’s the 1960s again, thin clients etc).

If you’re willing to spend the aforementioned 50 cents or so, you can achieve the following result (clean build):

time (./swift/utils/build-script --assertions --no-swift-stdlib-assertions --llbuild --swiftpm --xctest --build-subdir=buildbot_linux --lldb --release --foundation -- --swift-enable-ast-verifier=0 --install-swift --install-lldb --install-llbuild --install-swiftpm --install-xctest --install-prefix=/usr '--swift-install-components=compiler;clang-builtin-headers;stdlib;sdk-overlay;dev' --build-swift-static-stdlib=1 --install-destdir=/ --install-foundation --reconfigure > /dev/null 2>&1 )

real 8m54.418s
user 156m25.255s
sys 9m34.455s

Which is, all things being equal, pretty nice. I’m curious what most others face. Nightly untested builds would be quite doable, and I think using Docker for the Linux builds and development would be of significant interest to many, including whoever at Apple is co-ordinating the CI efforts. Docker is very stable, simple and powerful in its ability to control for many factors when working on such large projects as Swift.

(For those curious, my use case here was that withVarArgs is not yet available in the snapshot from 10 Dec, but it is in master.)

Quickly trying out Swift

In the same manner as IBM’s Swift sandbox (https://developer.ibm.com/swift/2015/12/03/introducing-the-ibm-swift-sandbox/), it’s now trivial to provision something a bit smaller in the cloud or locally and type

`docker pull swiftdocker/swift`

With the image downloaded (warning, it’s a couple of gigs, so I’d recommend using a fat pipe or a cloud machine), you can proceed to

`docker run --privileged -it swiftdocker/swift swift`

and immediately get a Swift interpreter! The privileged flag is some kind of weird voodoo to get LLDB to work properly, since it needs to think it has access to the underlying hardware. Any suggestions on patches to fix this issue would be most welcome.

Hacking on Swift

Just as it’s possible to build Swift in a docker container so too is it possible to actively develop the language in a Linux environment, while still benefitting from any local affordances you may have. If there is interest from the core team, I would be happy to contribute a meta-repo of Swift, submoduleing all the repos and supplying a docker-compose file which would enable developers to provision a development machine while being able to mirror local changes over to the machine. Similarly, “preheated” images could be built which would eliminate the need for a fresh build when first starting on the project.

Deploying cloud applications in Swift

With the above infrastructure already in place and growing, deploying an application using docker is as simple as writing your Swift application, and then a Dockerfile describing how to build and start it:

FROM swiftdocker/swift
ADD . /code
WORKDIR /code
RUN swift build
CMD .build/Debug/code

The CMD line is the statement executed when the container is started. Many services already exist to support automatically running and scaling dockerized infrastructure, and it’s my hope that having Swift running in Docker will contribute to adoption by a wider community of developers.

Testing Swift applications on CI as a service providers

Whether or not you choose to deploy an application in Docker, services like Travis CI use docker extensively to automate their test machine provisioning infrastructure. If your Swift library or application can build on Linux (command line tools are an obvious target for the latter), you would be able to take advantage of a Docker environment to quickly achieve low build times, as opposed to the generally expensive and slower OS X machines that are optimised for Xcode and tend to lack the open source toolchain.

···

I hope some of you may find this information useful. I’ve posted it on dev since I believe it’s more relevant to the developers than the users, right now. Please let me know if you have any questions about Docker or how it might relate to Swift!

Oh, and Merry Christmas/have a great winter holiday!

Tom


(Max Moiseev) #2

I personally think it would be of great help to newcomers to have a containerized development environment to quickly start experimenting. Non-newcomers would also benefit from it by being able to test their changes on different versions of Linux, for example.

I would prepare a set of scripts and Dockerfile’s in a separate repo to start with, and at least add a link to it to Swift’s main ‘README’ via a PR to see what the community has to say. Maybe an extra paragraph in 'system requirements’?

On a slightly more technical note, can you elaborate a bit more on why you’d need a docker-compose? If only for volumes to be in a separate container… Maybe I’m missing something.

thanks,
max

···

On Dec 18, 2015, at 12:34 PM, swizzlr via swift-dev <swift-dev@swift.org> wrote:

Hacking on Swift

Just as it’s possible to build Swift in a docker container so too is it possible to actively develop the language in a Linux environment, while still benefitting from any local affordances you may have. If there is interest from the core team, I would be happy to contribute a meta-repo of Swift, submoduleing all the repos and supplying a docker-compose file which would enable developers to provision a development machine while being able to mirror local changes over to the machine. Similarly, “preheated” images could be built which would eliminate the need for a fresh build when first starting on the project.


(Fastmail) #3

Hi Max,

Thanks for your reply and encouragement! I will do just that.

Docker volumes allow you to specify volumes to mount inside the container; where this shines is mounting host directories into the container, so that one can edit files on the host and see the changes reflected immediately in the container.

Docker-compose does a lot of things, but in particular its benefit is that it can encapsulate a lot of configuration for a container. One example of that configuration is the above volumes feature, which is normally a pain to write out on the command line. Another is the usual dance of building an image, creating the container, and then starting and stopping it with the correct state. With docker compose, the following commands:

docker build -t swift-dev-image .
docker create -it --name development -v /my/absolute/path/to/swift:/container/mount-point swift-dev-image
docker start -ai development

Become:

docker-compose up

In my research I have however discovered that this kind of mounting is quite flaky/nonexistent for certain docker hosts, in particular OS X, however there are indeed plugins for docker that use rsync to achieve the desired effect, so it's still quite doable. I'll look into it.

Regarding alternate systems, yes, that would be amazing! Docker base images are rarely reflective of general Linux distributions, so caution is needed when trying to apply them as a way of checking compatibility, but certain fundamental things are possible such as checking Ubuntu 14 vs 15, centOS, Debian, Fedora at a core level. It would be interesting, however, to create functional test suites using full distro base images (I'm sure they exist) that can then have the packages installed on them as a way of verifying a wide array of configurations in an automated fashion.

I think for now my objective will be to provide a quickstart docker image and docker-compose, using a metarepo with submodules to achieve the desired effects.

I would be very, very interested in hearing from Swift developers about ways of accelerating time to development by providing build intermediates.

Best,

Tom

···

Sent from my iPhone

On 18 Dec 2015, at 21:25, Max Moiseev <moiseev@apple.com> wrote:

I personally think it would be of great help to newcomers to have a containerized development environment to quickly start experimenting. Non-newcomers would also benefit from it by being able to test their changes on different versions of Linux, for example.

I would prepare a set of scripts and Dockerfile’s in a separate repo to start with, and at least add a link to it to Swift’s main ‘README’ via a PR to see what the community has to say. Maybe an extra paragraph in 'system requirements’?

On a slightly more technical note, can you elaborate a bit more on why you’d need a docker-compose? If only for volumes to be in a separate container… Maybe I’m missing something.

thanks,
max

On Dec 18, 2015, at 12:34 PM, swizzlr via swift-dev <swift-dev@swift.org> wrote:

Hacking on Swift

Just as it’s possible to build Swift in a docker container so too is it possible to actively develop the language in a Linux environment, while still benefitting from any local affordances you may have. If there is interest from the core team, I would be happy to contribute a meta-repo of Swift, submoduleing all the repos and supplying a docker-compose file which would enable developers to provision a development machine while being able to mirror local changes over to the machine. Similarly, “preheated” images could be built which would eliminate the need for a fresh build when first starting on the project.


(Max Moiseev) #4

Hi Tom,

With docker compose, the following commands:

docker build -t swift-dev-image .
docker create -it --name development -v /my/absolute/path/to/swift:/container/mount-point swift-dev-image
docker start -ai development

Become:

docker-compose up

Fair enough (for some reason I’ve never thought of using compose for a single container, but it totally makes sense).
Let’s see what you’ll come up with! :+1:

max

···

On Dec 18, 2015, at 7:06 PM, Thomas Catterall <me@swizzlr.co> wrote:

Hi Max,

Thanks for your reply and encouragement! I will do just that.

Docker volumes allow you to specify volumes to mount inside the container; where this shines is mounting host directories into the container, so that one can edit files on the host and see the changes reflected immediately in the container.

Docker-compose does a lot of things, but in particular its benefit is that it can encapsulate a lot of configuration for a container. One example of that configuration is the above volumes feature, which is normally a pain to write out on the command line. Another is the usual dance of building an image, creating the container, and then starting and stopping it with the correct state. With docker compose, the following commands:

docker build -t swift-dev-image .
docker create -it --name development -v /my/absolute/path/to/swift:/container/mount-point swift-dev-image
docker start -ai development

Become:

docker-compose up

In my research I have however discovered that this kind of mounting is quite flaky/nonexistent for certain docker hosts, in particular OS X, however there are indeed plugins for docker that use rsync to achieve the desired effect, so it's still quite doable. I'll look into it.

Regarding alternate systems, yes, that would be amazing! Docker base images are rarely reflective of general Linux distributions, so caution is needed when trying to apply them as a way of checking compatibility, but certain fundamental things are possible such as checking Ubuntu 14 vs 15, centOS, Debian, Fedora at a core level. It would be interesting, however, to create functional test suites using full distro base images (I'm sure they exist) that can then have the packages installed on them as a way of verifying a wide array of configurations in an automated fashion.

I think for now my objective will be to provide a quickstart docker image and docker-compose, using a metarepo with submodules to achieve the desired effects.

I would be very, very interested in hearing from Swift developers about ways of accelerating time to development by providing build intermediates.

Best,

Tom

Sent from my iPhone

On 18 Dec 2015, at 21:25, Max Moiseev <moiseev@apple.com <mailto:moiseev@apple.com>> wrote:

I personally think it would be of great help to newcomers to have a containerized development environment to quickly start experimenting. Non-newcomers would also benefit from it by being able to test their changes on different versions of Linux, for example.

I would prepare a set of scripts and Dockerfile’s in a separate repo to start with, and at least add a link to it to Swift’s main ‘README’ via a PR to see what the community has to say. Maybe an extra paragraph in 'system requirements’?

On a slightly more technical note, can you elaborate a bit more on why you’d need a docker-compose? If only for volumes to be in a separate container… Maybe I’m missing something.

thanks,
max

On Dec 18, 2015, at 12:34 PM, swizzlr via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hacking on Swift

Just as it’s possible to build Swift in a docker container so too is it possible to actively develop the language in a Linux environment, while still benefitting from any local affordances you may have. If there is interest from the core team, I would be happy to contribute a meta-repo of Swift, submoduleing all the repos and supplying a docker-compose file which would enable developers to provision a development machine while being able to mirror local changes over to the machine. Similarly, “preheated” images could be built which would eliminate the need for a fresh build when first starting on the project.