This would be the first time I post a topic in this forum.
I want to make a build cache for iOS app build dependencies from SwiftPM, but it doesn't seem to be supported.
What others have done seemed not working because of the the same problem, so I guess the --cache-build feature like Carthage is not implemented.
Is there any similar feature to carthage --cache-build in SwiftPM?
If not, could anyone please tell me how the newly built checkouts in DerivedData can be distinguished from the cached ones(moved to another directory temporarily and pasted back) and why it is not implemented?
I know SwiftPM folks such as @Aciid and @ddunbar are working towards deterministic builds & caching in the future. The llbuild2 project is one such effort.
In the meantime, I'm currently implementing a basic caching strategy based on a checksum of the Swift source file contents. The basic idea is to introduce a shell script that wraps swift run and explicitly passes --skip-build if the checksum has not changed.
#!/bin/sh
#
# bin/run: Wraps swift-run and provides caching.
#
# Usage:
#
# bin/run <executable-product> [<args>...] # wraps 'swift run' to run an executable product
# bin/run --build # run 'swift build' if the cache is invalid
# bin/run --clean # run 'swift package clean' and invalidate the cache
# bin/run --resolve # run 'swift package resolve'
# bin/run --update [<dependency>] # run 'swift package update', optionally for a specific dependency
# bin/run --shasum # print out the current cache key
set -e
cache_key() {
cat Package.resolved {Sources,.build/checkouts}/**/*.swift \
| shasum \
| cut -f1 -d' '
}
cache_valid() {
[ "$IGNORE_CACHE" != 1 ] \
&& [ -f "$SWIFTPM_CACHE_KEY_PATH" ] \
&& [ "$(cat "$SWIFTPM_CACHE_KEY_PATH")" = "$(cache_key)" ]
}
write_cache_key() {
if [ -n "$SWIFTPM_CACHE_KEY_PATH" ]; then
mkdir -p "$(dirname "$SWIFTPM_CACHE_KEY_PATH")"
cache_key > "$SWIFTPM_CACHE_KEY_PATH"
fi
}
# wrap the system swift binary to set some defaults
swift() {
env SDKROOT=macosx swift "$@"
}
build_if_necessary() {
if ! cache_valid; then
swift build
write_cache_key
fi
}
# set up a default value for SWIFTPM_CACHE_KEY_PATH which stores
# the cache key inside .build
if [ -z ${SWIFTPM_CACHE_KEY_PATH+x} ]; then
export SWIFTPM_CACHE_KEY_PATH=.build/contents.sha
fi
if [ $# = 0 ]; then
echo >&2 "bin/run: Specify a subcommand to run."
exit 1
fi
case "$1" in
--build)
build_if_necessary
;;
--clean)
swift package clean
;;
--resolve)
swift package resolve
;;
--update)
shift
swift package update "$@"
;;
--shasum)
cache_key
;;
*)
build_if_necessary
swift run --skip-build "$@"
;;
esac
Ah, I just realised that you're specifically talking about iOS dependencies, i.e., Xcode's SwiftPM integration. I unfortunately don't have a solution for that yet.
Thank you for the quick response. What I am facing now is yes, about xcodebuild integration caching. Pretty sure nobody ever thought yet because it is not even implemented in the SwiftPM itself yet. If I can wrap all of the dependencies' .build portion to be built like what you are making in caching phase, would it not be built every time when the CI build for iOS app is executed?
It would definitely be possible to cache Xcode's DerivedData folder, which would presumably give you some improvements to incremental builds.
The first problem with that is that the xcodebuild archive action always performs a clean build. So without prebuilding your dependencies into a .framework or .xcframework outside of the xcodebuild process, I don't think it would be possible to meaningfully cache the build output.
The second problem is the same as with command-line use of SwiftPM, in that the build system is not fully deterministic, and so even if you cache all of DerivedData, things would likely still be rebuilt across CI runs.
One area where you can potentially make some gains though is the dependency resolution phase: if you check the output of xcodebuild -help, there are a few options in there related to manually resolving dependencies (-resolvePackageDependencies) and controlling where the packages are checked out (-clonedSourcePackagesDirPath). By caching the dependency checkouts across CI runs you could potentially remove the need for xcodebuild to resolve dependencies at all. The -disableAutomaticPackageResolution flag will let you turn that off manually to build from the cached sources.
One potential solution to caching the build products of your Swift packages would be to link them all into a single dynamic framework and then use carthage build to prebuild that framework. I haven't personally tested this setup, but it could be quite useful.
This will most likely not work, unless you're building C-family language code only or are using @_implementationOnly imports for everything. When building your app, all the modules/headers will need to be visible to the compiler and the framework will only contain one.
I think I see what you mean: when such a dynamic framework is built by Xcode as a dependency in your build process, the various .swiftmodule files do appear in the build products, allowing the imports to work. But in order to prebuild this .framework, it would need to also provide the additional .swiftmodule files. Would it be feasible to add a build script phase that manually copies the .swiftmodule files into the frameworks Modules directory?
In my testing I've been able to successfully copy the various .swiftmodule files into Packages.framework/Modules. Here's how I got it to work:
Create a new Packages.xcodeproj, separate to your main project file. This is where the Swift dependencies will be added, and produces a single dynamically linked Packages.framework.
Add a Run Script build phase with the following contents:
modules_dir="$BUILT_PRODUCTS_DIR"/"$MODULES_FOLDER_PATH"
for swiftmodule in "$BUILT_PRODUCTS_DIR"/*.swiftmodule; do
cp -r "$swiftmodule" "$modules_dir"/
done
This will copy the generate .swiftmodule files for all your dependencies into the framework bundle.
Use carthage build --no-skip-current to build Packages.framework.
Link & Embed Packages.framework in your regular project.
Update SWIFT_INCLUDE_PATHS to add Carthage/Build/iOS/Packages.framework/Modules so that your app can see the embedded Swift modules.
(Optional) If your Swift dependency depends on any C modules, this will still not compile, because C/system modules don't produce .swiftmodule files. So, for example, I had to fork GRDB in my project and change import CSQLite to @_implementationOnly import CSQLite (and remove a bunch of @inlinable attributes that were no longer valid because the C SQLite symbols are no longer exported).
So this mostly works, depending on your specific dependencies. However I probably won't keep this setup because it adds more complexity than I'd like to my build process.
Since you have to manually adjust SWIFT_INCLUDE_PATHS anyway, I would recommend putting the extra modules into a separate directory inside the .framework, just to make sure they won't interfere with any expectations regarding the Modules directory.
Keen to understand the current status of this and potentially workarounds. Some of the dependencies we have take a long time to build from scratch. Considering how often we update them, it'd be great if we can cache the builds.
An alternative is to use prebuilt frameworks by specifying binaryTarget inside the package.swift file but that requires either the vendor supplies the binary or we build the binary ourselves and hosted in a separated repo.
I have finally found few months ago that there has come a new feature for SwiftPM called clonedSourcePackagePath. With this option in xcodebuild, SwiftPM finds the path first before clean-build the dependencies. Thanks a lot for the constant support, Team Swift!
is only caching the sources of your dependencies, as a result you don't need to fetch them for every build.
However, you still need to build the dependencies from the cached sources in every build - which usually takes up much more time than fetching.
the earlier discussions were about caching also build artifacts of the dependencies. That would allow you to only build your own code and link to the pre-built dependencies from a cache.
@NeoNacho@sharplet I'd love to hear your thoughts about this topic and if there were any advancements?
I thought the option makes cache like Carthage so we don’t need to build it again if we already have directory we directed in that option.
Why I thought it as a wholesome cache function is simply because the xcodebuild log shows as if it is not rebuilding its dependencies(only “resolves” and never clones&checks out all the dependencies every time we build the application that depends on Swift PM).
i wonder what the swift core team / apple think about caching build results... is it something on the roadmap? or at least on a wishlist somewhere? or maybe not even recommended to try?
