Incremental builds with Swift Package Manager and Linux / Docker

I'm using Swift Package Manager to build my Swift server app in a Linux Docker container. Each time I rebuild it fetches all of the dependencies and rebuilds them all. Recently I started using Swift-NIO as a dependency and it takes a while to compile. Are incremental builds supported on Linux?

I tried to get incremental builds working by having my Package.swift + some dummy files copied into docker -> build -> copy in the real source files -> build. I was hoping the dependencies of the first build would be saved for the second build. Unfortunately, even if I just do two identical build commands one after another it still rebuilds all dependencies from scratch. Are incremental builds on Linux supported?

This is the docker file I'm using:

FROM bridger/swift-ubuntu:latest
MAINTAINER Bridger
LABEL Description="Docker image for building and running the Scribble server."

EXPOSE 9080

RUN mkdir /root/Scribble-Server

ADD Tests /root/Scribble-Server/Tests
ADD Package.swift /root/Scribble-Server
ADD Package.resolved /root/Scribble-Server

# Add a dummy file for resolving dependencies and pre-building first
ADD Sources/DummyForDependencies/main.swift /root/Scribble-Server/Sources/ScribbleServer/main.swift
ADD Sources/DummyForDependencies/DummyModel.swift /root/Scribble-Server/Sources/CanvasModel/DummyModel.swift

# Resolve the dependencies by building using the dummy sources
RUN cd /root/Scribble-Server && export KITURA_NIO=1 && swift build -c release

# Delete the dummy sources
RUN rm /root/Scribble-Server/Sources/ScribbleServer/main.swift
RUN rm /root/Scribble-Server/Sources/CanvasModel/DummyModel.swift

# Add in the real sources
ADD Sources /root/Scribble-Server/Sources
RUN rm -rf /root/Scribble-Server/Sources/DummyForDependencies

# Build again. This unfortunately rebuilds all dependencies, even though they haven't changed
RUN cd /root/Scribble-Server && export KITURA_NIO=1 && swift build -c release
3 Likes

Haven't played with that in a while, but previously I did something like this, and that seemed to work:

# Install swift deps to use cache
COPY ./Package.swift ./
COPY ./Package.resolved ./
RUN swift package resolve

Then copy the sources over and build the final product.

Running swift package resolve as a separate step does cache the package resolution (so future build steps don't re-download the packages from git). Unfortunately, the packages are all compiled from scratch each time. The build stage is still not incremental.

2 Likes

@Bridger_Maxwell I have run into exactly the same problem — the results of swift package resolve can be cached, but the dependencies will be rebuilt every time. My suspicion is that SwiftPM for some reason considers build artifacts in lower layers stale and thinks that it needs to rebuild them. Did you ever find a solution to this?

1 Like

Incremental builds are supported on Linux, just like macOS. Run swift build twice and the second time it will be virtually a no‐op.

However, all continuous integration environments I have ever used consistently cause the cache to be rejected as stale, and this happens on all platforms. I believe something like the environment or the file modification dates come into consideration when deciding whether the cache is still valid, such that the strategy of copying the build cache onto a new virtual machine runs afoul of it.

1 Like

The build system uses stat to determine a file modified since it last saw it. Relaunching the docker container most likely invalidates the stat information that was previously obtained.

1 Like

Hm, it looks like the stat information of cached files don't change in a docker container. Do you mind filing a JIRA for this? It might be possible to make this work.

1 Like

Thank you for looking into this! I have filed https://bugs.swift.org/browse/SR-11760.

3 Likes

Hey there! Is there any updates on this one?

1 Like

I'm commenting for visibility. This is still an issue and as a result is making Vapor builds very slow as it needs to recompile all of swift-nio each time. https://github.com/vapor/vapor/issues/2325

2 Likes

@codafi I was wondering if the recent Cross-Module Incremental Builds PR would have any effect on this issue.

All incremental builds will improve wrt 5.3(.1) as long as you ensure that as many targets as possible pass -enable-experimental-cross-module-incremental-build.

3 Likes

Any updates on this? I ran into the same issue when using aws codebuild.

Cross-module incremental builds shipped with Xcode 13 and Swift 5.5. The latest development toolchains will have these features available for users on Linux and Windows.

1 Like
Terms of Service

Privacy Policy

Cookie Policy