I'm using Swift Package Manager to build my Swift server app in a Linux Docker container. Each time I rebuild it fetches all of the dependencies and rebuilds them all. Recently I started using Swift-NIO as a dependency and it takes a while to compile. Are incremental builds supported on Linux?
I tried to get incremental builds working by having my Package.swift + some dummy files copied into docker -> build -> copy in the real source files -> build. I was hoping the dependencies of the first build would be saved for the second build. Unfortunately, even if I just do two identical build commands one after another it still rebuilds all dependencies from scratch. Are incremental builds on Linux supported?
This is the docker file I'm using:
FROM bridger/swift-ubuntu:latest
MAINTAINER Bridger
LABEL Description="Docker image for building and running the Scribble server."
EXPOSE 9080
RUN mkdir /root/Scribble-Server
ADD Tests /root/Scribble-Server/Tests
ADD Package.swift /root/Scribble-Server
ADD Package.resolved /root/Scribble-Server
# Add a dummy file for resolving dependencies and pre-building first
ADD Sources/DummyForDependencies/main.swift /root/Scribble-Server/Sources/ScribbleServer/main.swift
ADD Sources/DummyForDependencies/DummyModel.swift /root/Scribble-Server/Sources/CanvasModel/DummyModel.swift
# Resolve the dependencies by building using the dummy sources
RUN cd /root/Scribble-Server && export KITURA_NIO=1 && swift build -c release
# Delete the dummy sources
RUN rm /root/Scribble-Server/Sources/ScribbleServer/main.swift
RUN rm /root/Scribble-Server/Sources/CanvasModel/DummyModel.swift
# Add in the real sources
ADD Sources /root/Scribble-Server/Sources
RUN rm -rf /root/Scribble-Server/Sources/DummyForDependencies
# Build again. This unfortunately rebuilds all dependencies, even though they haven't changed
RUN cd /root/Scribble-Server && export KITURA_NIO=1 && swift build -c release
Running swift package resolve as a separate step does cache the package resolution (so future build steps don't re-download the packages from git). Unfortunately, the packages are all compiled from scratch each time. The build stage is still not incremental.
@Bridger_Maxwell I have run into exactly the same problem — the results of swift package resolve can be cached, but the dependencies will be rebuilt every time. My suspicion is that SwiftPM for some reason considers build artifacts in lower layers stale and thinks that it needs to rebuild them. Did you ever find a solution to this?
Incremental builds are supported on Linux, just like macOS. Run swift build twice and the second time it will be virtually a no‐op.
However, all continuous integration environments I have ever used consistently cause the cache to be rejected as stale, and this happens on all platforms. I believe something like the environment or the file modification dates come into consideration when deciding whether the cache is still valid, such that the strategy of copying the build cache onto a new virtual machine runs afoul of it.
The build system uses stat to determine a file modified since it last saw it. Relaunching the docker container most likely invalidates the stat information that was previously obtained.
Hm, it looks like the stat information of cached files don't change in a docker container. Do you mind filing a JIRA for this? It might be possible to make this work.
All incremental builds will improve wrt 5.3(.1) as long as you ensure that as many targets as possible pass -enable-experimental-cross-module-incremental-build.
Cross-module incremental builds shipped with Xcode 13 and Swift 5.5. The latest development toolchains will have these features available for users on Linux and Windows.
For me I found it was rebuilding everything when I do swift build then swift run, but if I do swift run only, then it would only build the changed files in my project and that's it. So maybe there is a bug in swift build ? I am using SPM 5.5
I just ran into this as well. 10+ minutes to build a Vapor server in a Dockerfile which makes iteration super slow for building an APNS server
fwiw I tried following Mounts | Docker Docs and it didn't seem to make any difference either, I think because I'm using the kaniko builder which doens't support BuildKit. I also tried setting --build-path /build/.artifacts on each swift build invocation, but as mentioned above this doesn't seem to address the underlying problem of timestamps changing across steps.
May need to resign to just waiting 10-15 minutes between deploys for now
FWIW, I use --mount=type=cache in my docker builds and had zero issues with it so far.
I was worried that I would need to clear the caches all the time because swift build would get confused and require a clean build - but so far it has been quite the soldier. after ~300 changes/builds I had to clear the caches exactly once (after switching git branches)
here is an excerpt of what I use:
ARG SWIFT_VERSION=5.8
FROM swift:$SWIFT_VERSION AS builder
WORKDIR /build
# Resolve all dependencies if there was a change
COPY ./Package.* ./
RUN --mount=type=cache,target=/build/.build swift package resolve
# Copy all sources and build all
COPY . .
# build with caching and copy app out of cached folder
RUN --mount=type=cache,target=/build/.build \
swift build -c release \
&& cp -a .build/release/ /release
FROM swift:$SWIFT_VERSION-slim
COPY --from=builder /release/${SERVICE_NAME} ./app
ENTRYPOINT ["./app"]
