SwiftPM: How to prevent 'Resolve Packages' from stymying developer productivity [local packages]?

Development Package Manager
1

I work in a corpulent iOS monorepo with 100s of Xcode projects in an Xcode workspace, and maintaining these has been a tedious task (custom scripts to update dependencies, framework references, sort project files, etc). Therefore, like many, we thought SPM with its abstracted and simplified module management would be a viable solution to modularization in our sprawling codebase!

After a cursory spike, we started migrating our 'core' frameworks at the root of the dependency tree into a single SPM package located at the root of the repository. Essentially, all of our shared frameworks. Fast-forward to today and it's been arguably one of the most painful and frustrating experiences in my career. Surprisingly, upgrading to Xcode14.1 has only made it worse.

I won't get into the myriad of issues, but here are them at a high level:

SPM is categorically untenable for external (remote) dependency management in medium-sized and larger projects

The number of 'resolve' issues related to SPM failing to download remote dependencies due to SSH, Git, cache, random resolve errors, etc. issues is innumerable. The performance, hidden and bizarre errors, poor caching, and developer support led to us ditching remote dependencies completely.

Instead, we went back to using Carthage to download our external dependencies. We integrate them into SPM using local binaries; this provided stability, an incredible performance boost, and developer productivity increase; but also disenchantment.

Poor configuration options

With an Xcode project, you can configure your targets to be built differently depending on the environment via xcconfig files. We use these extensively to toggle settings such as "treat warnings as errors", "warn about type long type checking", etc. in different environments such as CI.

SPM does not provide an acceptable way to do this—especially if you are using SPM in a monorepo for iOS. We use test plans to build and test our packages which means we don't build our packages via swift package. We ended up reading a JSON file that only existed in CI from the Package manifest to enable "treat warnings as errors" to preclude an influx of warnings slinking in; a complete hack for something that Xcode has supported for decades.

Slow builds: Create Build Description

The 'Create Build Description' in Xcode has seemingly gotten slower now that we've started using SPM (I cannot however verify SPM is to blame).

Primary issue: 'Resolve packages...' destroys developer productivity... even when they're all local!

Previously in Xcode 13, we were able to resolve the package(s) in our workspace from the command line to prevent Xcode from spinning upon opening. With Xcode 14.1, resolving packages from the command line does nothing, and Xcode completely hangs and beach balls for ~5-20s to resolve LOCAL packages when opening Xcode. We don't have ANY remote dependencies and it can take upwards of 20 seconds to resolve our manifest, which as of today has 10 products and ~20 targets, and depends on one local package that contains 2 products and 2 targets.

Xcode has even started to hang (~10s) when being closed (could be unrelated).

However, that's not even the worst part, performing simple and common operations such as adding a file to a package target cause Xcode to start resolving again! When the 'Resolve Packages' occurs, Xcode again hangs (beach balls) for 2-3s. Note, adding a file to one of the Xcode projects in the workspace is instant.

This is all using M1 Pros and M1 Pro Maxes with a minimum of 32 GB of RAM.

One thing to note is that opening the Package outside of the Xcode workspace is faster, but 'Resolves' still occur for the same operations (e.g. adding a file).

Questions:

  1. How does one prevent Xcode from spuriously performing 'Resolve Packages' when performing basic operations like adding a file or switching branches (mind you, 0 changes to the Package.swift file)?

  2. Is there any way to disable Resolve Packages entirely and rely on manually running it? Does Xcode really need to Resolve packages every time a file is added (even when the file is outside of any of the package targets paths)?

  3. Does anyone have troubleshooting tips and ideas to understand why Xcode needs to Resolvle Packages when the manifest never changes? Spending minutes a day on an M1 Pro opening Xcode and adding files is rather frustrating.

  4. Why does Resolve Packages... cause Xcode to hang at all? Shouldn't this be a background operation?

  5. Will the performance of SPM improve (esp. with Xcode)? Should we abandon the ship and return to our custom Xcode generation scripts?

At this point, we've halted the migration of non-shared frameworks to SPM due to the increasingly high cost of stymying developer productivity.

Thanks for following along and I appreciate any advice, recommendations, or moral support.

15 Likes
2

Everyone who has tried to use Xcode's SPM integration at scale has encountered these issues. You can investigate alternate project management strategies, like using Tuist to generate your Xcode project, which you then integrate Carthage or CocoaPods on top of for external dependencies. But I don't think there's a way to fix these issues, as Xcode is a black box, especially when it comes to the SPM integration.

Redundant "Resolve Packages" actions are the performance killer, even on small projects. Even just closing your Xcode project and reopening causes reresolution. Calling xcodebuild ... clean causes reresolution. At least Xcode 14 has helped with Xcode's resilience to whatever was causing the package corruption before so we don't have to reset packages as often, but as you noted, it regressed in other ways.

Have you tried Xcode 14.3? I have noticed some major build performance improvements, so even if resolution is still slow and redundant, my builds have never been faster.

3 Likes
3

We have fought with the same problems recently and thought the same, but had some promising findings after tracing Xcode itself with Instruments. Based on my investigations, most of the slowdown happens not in the "black box" of Xcode but in the open-source SPM tool it uses.

In fact, we had barely any issues with SPM before bringing in Crashlytics along with the Firebase SDK. Since then, we've seen Xcode go "Resolving Package Graph" for up to ~40 seconds after the smallest changes to the our package, such as simply adding or removing a file!

Sampling Xcode resolving package graph

It's pretty easy to test this with the Firebase SDK which depends on some particularly heavy monorepo-style dependencies, something SPM was not well prepared for. I've attached below a sample Package.swift file which (given an empty source file at Sources/SlowSPM/dummy.swift) reproduces the slowdown both when opened in Xcode as well as in SPM command line:

Package.swift 
// swift-tools-version: 5.7.1
import PackageDescription
let package = Package(
  name: "slow-spm",
  products: [.library(name: "SlowSPM", targets: ["SlowSPM"])],
  dependencies: [
    .package(url: "https://github.com/firebase/firebase-ios-sdk", .upToNextMinor(from: "10.5.0")),
  ],
  targets: [
    .target(name: "SlowSPM", dependencies: [
      .product(name: "FirebaseCrashlytics", package: "firebase-ios-sdk"),
    ]),
  ]
)

Recording Xcode with the Sampler instrument, the heaviest stack trace is seen where SPMWorkspace (closed source?) calls to Workspace.loadPackageGraph (part of SPM). As it turns out, a lot of time is spent on accessing files (and checking whether they are symlinks):

...
19.17 s partial apply for closure #2 in SPMWorkspace.processPackageGraphActionsInBackgroundIfNeeded()
19.17 s  closure #2 in SPMWorkspace.processPackageGraphActionsInBackgroundIfNeeded()
19.15 s   Workspace.loadPackageGraph(rootInput:explicitProduct:createMultipleTestProducts:createREPLProduct:forceResolvedVersions:customXCTestMinimumDeploymentTargets:observabilityScope:)
19.05 s    static PackageGraph.load(root:identityResolver:additionalFileRules:externalManifests:requiredDependencies:unsafeAllowedPackages:binaryArtifacts:shouldCreateMultipleTestProducts:createREPLProduct:customPlatformsRegistry:customXCTestMinimumDeploymentTargets:fileSystem:observabilityScope:)
... (omit a few nested frames)
19.02 s            PackageBuilder.createTarget(potentialModule:manifestTarget:dependencies:)
18.93 s             TargetSourcesBuilder.run()
 9.79 s              TargetSourcesBuilder.computeContents()
 5.67 s               specialized LocalFileSystem.isSymlink(_:)

Testing SPM slowness outside Xcode

It is possible to test this code path outside Xcode too. First call swift package resolve to install packages. Then, to hit the code path with static PackageGraph.load(...), run the command:

$ time swift package show-dependencies
...
real    0m30.877s
user    0m15.481s
sys     0m8.859s

With a fresh build of SPM from main branch (running /path/to/spm/.build/release/swift-package show-dependencies), I get similar (if not slightly worse) execution times: 38.354s/16.313s/9.311s, respectively.

Suggestion 1: Use lstat directly

Instead of tediously retrieving all file attributes only to check if the .type is equal to .typeSymbolicLink, isSymlink(_:) should preferably just call lstat(2). Trying that with a quick hack, the command execution time went down to 10.760s/7.093s/2.275s, or a 3x speedup.

Suggestion 2: Skip unwanted files if target declares sources

Tracing swift-package with the File Activity instruments further reveals that SPM is indeed hitting a lot of files under Firebase and its dependencies. The biggest of those libraries consist of over 20,000 files, most of them (excluded code, tests and test data) irrelevant to the Swift package. TargetSourcesBuilder starts by finding all files reachable under the target's directory. But that becomes a problem if a package is making deliberate use of .target(... path: exclude: sources: ).

I'm probably making too simple an assumption here, but if TargetSourcesBuilder only included declared source files (if any) in the search, then a lot fewer paths would need to be visited. If this change is right to do (it breaks some potentially obsolete unit tests!), it could provide up to a further 4.5x speedup, evaluating show-dependencies for me in just 2.324s/0.767s/0.561s.

I'm sure there are further optimisations to be done to SPM. Testing with large real-world Swift packages could help us find and fix the bottlenecks, and directly improve the Xcode experience too. (In the meantime, we'll use Firebase via xcframeworks.)

13 Likes
4

Our project (sizeable mono repo as well) heavily depends on Realm and Firebase. We noticed long time ago that Xcode was super slow on compiling the project and SPM slow in resolving our dependencies particularly because of Realm (which even though does have a Swift interface, depends on a core C++ library), Firebase and Facebook. The commonality between those was C/C++/Objective-C code/subdependencies.

We created an in-house tool that is downloading latest versions of those libraries and create local binary packages with xcframework versions of those libs inside them. Compile time and resolving package graphs drastically improved to the point resolving package graph is almost non existent.

We now have zero non-swift dependencies imported as source and everything is bliss

5

I just reordered some local packages in my workspace. Not removed or added, just reordered.

Xcode started to Resolve Packages.

Somebody get the "this is fine" dog.

6

These are great findings and improvements! Hopefully you can get them into SPM itself. My comments about Xcode were more about why Xcode constantly reresolves dependencies after things like closing and reopening a project, or perhaps just reverting a local commit in git under Xcode, or during an xcodebuild ... clean. Some of those may have underlying SPM causes but, for instance, the clean issue seems Xcode specific, as swift package clean && swift build doesn't perform a reresolution.

7

I haven't been able to confirm why but my understanding is that the SPM integration in Xcode is simply monitoring the whole directory tree of your Swift package for changes… only to kick in the slow package re-resolution process I described earlier. :neutral_face:

8

Monitoring the entire directory tree is probably unavoidable. Because SPM effectively supports globbing, adding and removing files can result in package changes even without changes to the package manifest. Xcode would need very tight integration with SPM to know exactly when it needs to resolve packages, so it presumably just reruns that on every change under the assumption that a no-op resolve is cheap, and then runs into problems due to it not being cheap.

1 Like
9

SwiftPM has the advantage of knowing the requested operation up-front, so e.g. in this case it can shortcut directly to the build because it checks whether the structure of the package hasn't changed since the last build description was computed.

IDEs can't really do that, because they need to know about the model for various reasons, not just for conveying the structure of the project to a build system (they can invent a different caching layer of course). It also means you're not only paying the cost of resolution, but also of the build system needing to do a bunch of work again (this is probably not noticed by most folks since it gets drowned in the larger cost of performing a build).

10

Hi @johnliedtke :wave:,

I feel you pain, but I believe that some of your issues may be more related to configuration than Xcode or SPM.

I have a project in a monorepo that relies on 130 local SPM packages, along with a few external libraries like Firebase and TCA, all listed in a single Package.swift file, and it runs smoothly. However, I agree that Resolving Packages can sometimes be slow or recurrent without apparent reason, and SPM can be challenging and finicky to work with.

Can you clarify whether you have one Package.swift file or one per module? Furthermore, how have you integrated your Package.swift into your workspace?

Previously, I encountered several problems with SPM, such as difficulties fetching dependencies from repositories that required different SSH keys or when the Package.swift file did not match the local files, including missing files, incorrect or absent imports, or incorrect paths.

What I discovered to be helpful was to maintain the Package.swift file in sync, using a semi-automated approach(GitHub - mackoj/PackageGeneratorPlugin: Package Generator is a Swift Package Manager Plugin for simply updating your Package.swift file).

11

I could be wrong but pretty sure this SPM change from @fcanas that got rolled into 5.7.2 was supposed to fix the resolution issue (or at least the length of time it takes). @fcanas gave a great presentation to our local Cocoaheads on this and could probably speak better as to the logic. Xcode is definitely a black box though and I'm not seeing timing improvement even in Xcode 14.3 betas (which should include SPM 5.7.2) :frowning:

12

Maybe it fixed something but the problems I described earlier in this thread still exist(ed) in main and were due to the very high number of (ignorable) files in certain repositories and their slow processing, not cycle detection.

13

These improvements were great - out of curiosity did you open a PR with these? To get them on the radar if nothing else.

14

You're right, PRs—however incomplete—are probably the way to go. Opened apple/swift-tools-support-core#400 and apple/swift-package-manager#6267, respectively.

5 Likes