Building just SPM dependencies to cache in a Docker layer

I'm building a Swift app via Docker. It's got a decent number of SPM dependencies none of which change frequently.

While I can cache the resolution of the dependencies like so:

COPY ./Package.* ./
# packages won't be re-resolved until `Package.swift` or `Package.resolved` changes
RUN swift package resolve

I can't seem to find a way to cache the building of these packages.

Each time I build a docker image, all the package dependencies must be rebuilt.

My question is:

Is there a way to cache the built SPM dependencies in a Docker layer so that it only rebuilds if Package.swift or Package.resolved changes?

Something like swift build --packages-only or something.

This would seriously speed up my Docker build times.

Thanks in advance!