Caching SwiftPM for iOS app build on CI env

Hello all Swifters

This would be the first time I post a topic in this forum.

I want to make a build cache for iOS app build dependencies from SwiftPM, but it doesn't seem to be supported.


https://bugs.swift.org/browse/SR-12851?jql=text%20~%20"SwiftPM%20cache"

What others have done seemed not working because of the the same problem, so I guess the --cache-build feature like Carthage is not implemented.


Is there any similar feature to carthage --cache-build in SwiftPM?

If not, could anyone please tell me how the newly built checkouts in DerivedData can be distinguished from the cached ones(moved to another directory temporarily and pasted back) and why it is not implemented?

I know SwiftPM folks such as @Aciid and @ddunbar are working towards deterministic builds & caching in the future. The llbuild2 project is one such effort.

In the meantime, I'm currently implementing a basic caching strategy based on a checksum of the Swift source file contents. The basic idea is to introduce a shell script that wraps swift run and explicitly passes --skip-build if the checksum has not changed.

#!/bin/sh
#
# bin/run: Wraps swift-run and provides caching.
#
# Usage:
#
#   bin/run <executable-product> [<args>...] # wraps 'swift run' to run an executable product
#   bin/run --build                          # run 'swift build' if the cache is invalid
#   bin/run --clean                          # run 'swift package clean' and invalidate the cache
#   bin/run --resolve                        # run 'swift package resolve'
#   bin/run --update [<dependency>]          # run 'swift package update', optionally for a specific dependency
#   bin/run --shasum                         # print out the current cache key

set -e

cache_key() {
  cat Package.resolved {Sources,.build/checkouts}/**/*.swift \
    | shasum \
    | cut -f1 -d' '
}

cache_valid() {
  [ "$IGNORE_CACHE" != 1 ] \
    && [ -f "$SWIFTPM_CACHE_KEY_PATH" ] \
    && [ "$(cat "$SWIFTPM_CACHE_KEY_PATH")" = "$(cache_key)" ]
}

write_cache_key() {
  if [ -n "$SWIFTPM_CACHE_KEY_PATH" ]; then
    mkdir -p "$(dirname "$SWIFTPM_CACHE_KEY_PATH")"
    cache_key > "$SWIFTPM_CACHE_KEY_PATH"
  fi
}

# wrap the system swift binary to set some defaults
swift() {
  env SDKROOT=macosx swift "$@"
}

build_if_necessary() {
  if ! cache_valid; then
    swift build
    write_cache_key
  fi
}

# set up a default value for SWIFTPM_CACHE_KEY_PATH which stores
# the cache key inside .build
if [ -z ${SWIFTPM_CACHE_KEY_PATH+x} ]; then
  export SWIFTPM_CACHE_KEY_PATH=.build/contents.sha
fi

if [ $# = 0 ]; then
  echo >&2 "bin/run: Specify a subcommand to run."
  exit 1
fi

case "$1" in
  --build)
    build_if_necessary
    ;;
  --clean)
    swift package clean
    ;;
  --resolve)
    swift package resolve
    ;;
  --update)
    shift
    swift package update "$@"
    ;;
  --shasum)
    cache_key
    ;;
  *)
    build_if_necessary
    swift run --skip-build "$@"
    ;;
esac
1 Like

Ah, I just realised that you're specifically talking about iOS dependencies, i.e., Xcode's SwiftPM integration. I unfortunately don't have a solution for that yet.

1 Like

Thank you for the quick response. What I am facing now is yes, about xcodebuild integration caching. Pretty sure nobody ever thought yet because it is not even implemented in the SwiftPM itself yet. If I can wrap all of the dependencies' .build portion to be built like what you are making in caching phase, would it not be built every time when the CI build for iOS app is executed?

It would definitely be possible to cache Xcode's DerivedData folder, which would presumably give you some improvements to incremental builds.

The first problem with that is that the xcodebuild archive action always performs a clean build. So without prebuilding your dependencies into a .framework or .xcframework outside of the xcodebuild process, I don't think it would be possible to meaningfully cache the build output.

The second problem is the same as with command-line use of SwiftPM, in that the build system is not fully deterministic, and so even if you cache all of DerivedData, things would likely still be rebuilt across CI runs.

One area where you can potentially make some gains though is the dependency resolution phase: if you check the output of xcodebuild -help, there are a few options in there related to manually resolving dependencies (-resolvePackageDependencies) and controlling where the packages are checked out (-clonedSourcePackagesDirPath). By caching the dependency checkouts across CI runs you could potentially remove the need for xcodebuild to resolve dependencies at all. The -disableAutomaticPackageResolution flag will let you turn that off manually to build from the cached sources.

One potential solution to caching the build products of your Swift packages would be to link them all into a single dynamic framework and then use carthage build to prebuild that framework. I haven't personally tested this setup, but it could be quite useful.

1 Like

Thanks a lot for the suggestion, maybe I should try if I can make dynamic framework per each CI pipeline of iOS app... :innocent:

This will most likely not work, unless you're building C-family language code only or are using @_implementationOnly imports for everything. When building your app, all the modules/headers will need to be visible to the compiler and the framework will only contain one.

I think I see what you mean: when such a dynamic framework is built by Xcode as a dependency in your build process, the various .swiftmodule files do appear in the build products, allowing the imports to work. But in order to prebuild this .framework, it would need to also provide the additional .swiftmodule files. Would it be feasible to add a build script phase that manually copies the .swiftmodule files into the frameworks Modules directory?

I think the Modules directory can only contain a single module.

In my testing I've been able to successfully copy the various .swiftmodule files into Packages.framework/Modules. Here's how I got it to work:

  1. Create a new Packages.xcodeproj, separate to your main project file. This is where the Swift dependencies will be added, and produces a single dynamically linked Packages.framework.

  2. Add a Run Script build phase with the following contents:

    modules_dir="$BUILT_PRODUCTS_DIR"/"$MODULES_FOLDER_PATH"
    for swiftmodule in "$BUILT_PRODUCTS_DIR"/*.swiftmodule; do
      cp -r "$swiftmodule" "$modules_dir"/
    done
    

    This will copy the generate .swiftmodule files for all your dependencies into the framework bundle.

  3. Use carthage build --no-skip-current to build Packages.framework.

  4. Link & Embed Packages.framework in your regular project.

  5. Update SWIFT_INCLUDE_PATHS to add Carthage/Build/iOS/Packages.framework/Modules so that your app can see the embedded Swift modules.

  6. (Optional) If your Swift dependency depends on any C modules, this will still not compile, because C/system modules don't produce .swiftmodule files. So, for example, I had to fork GRDB in my project and change import CSQLite to @_implementationOnly import CSQLite (and remove a bunch of @inlinable attributes that were no longer valid because the C SQLite symbols are no longer exported).

So this mostly works, depending on your specific dependencies. However I probably won't keep this setup because it adds more complexity than I'd like to my build process.

Since you have to manually adjust SWIFT_INCLUDE_PATHS anyway, I would recommend putting the extra modules into a separate directory inside the .framework, just to make sure they won't interfere with any expectations regarding the Modules directory.

Terms of Service

Privacy Policy

Cookie Policy