SPM unconditionally walks the dependency tree

I have a strange behavior with the Swift Package Manager. One of my projects makes swift always crawl the dependency tree – which takes quite a while even on an M1.

It goes like that:

$ swift build
Updating https://github.com/Automotive-Swift/Swift-CAN
Updated https://github.com/Automotive-Swift/Swift-CAN (0.62s)
Updating https://github.com/Automotive-Swift/Swift-TouCAN
Updated https://github.com/Automotive-Swift/Swift-TouCAN (0.55s)
Updating https://github.com/Automotive-Swift/Swift-SocketCAN
Updated https://github.com/Automotive-Swift/Swift-SocketCAN (0.55s)
Updating https://github.com/Automotive-Swift/Swift-UDS
Updated https://github.com/Automotive-Swift/Swift-UDS (0.53s)
Updating https://github.com/Automotive-Swift/ECUmulator
Updated https://github.com/Automotive-Swift/ECUmulator (0.52s)
Updating https://github.com/andybest/linenoise-swift
Updated https://github.com/andybest/linenoise-swift (0.51s)
Updating https://github.com/mickeyl/Socket.swift
Updated https://github.com/mickeyl/Socket.swift (0.51s)
Updating https://github.com/Cornucopia-Swift/CornucopiaCore
Updated https://github.com/Cornucopia-Swift/CornucopiaCore (0.53s)
Updating https://github.com/tsolomko/SWCompression
Updating https://github.com/apple/swift-crypto.git
Updating https://github.com/Zewo/CLibreSSL.git
Updated https://github.com/Zewo/CLibreSSL.git (0.61s)
Updated https://github.com/apple/swift-crypto.git (0.61s)
Updated https://github.com/tsolomko/SWCompression (0.61s)
Updating https://github.com/Quick/Nimble.git
Updating https://github.com/tsolomko/BitByteData
Updating https://github.com/apple/swift-argument-parser
Updated https://github.com/apple/swift-argument-parser (0.67s)
Updated https://github.com/Quick/Nimble.git (0.67s)
Updated https://github.com/tsolomko/BitByteData (0.67s)
Updating https://github.com/onevcat/Rainbow.git
Updated https://github.com/onevcat/Rainbow.git (0.58s)
Computing version for https://github.com/apple/swift-argument-parser
Computed https://github.com/apple/swift-argument-parser at 1.0.2 (0.24s)
Computing version for https://github.com/Quick/Nimble.git
Computed https://github.com/Quick/Nimble.git at 8.1.2 (0.19s)
Computing version for https://github.com/apple/swift-crypto.git
Computed https://github.com/apple/swift-crypto.git at 1.1.6 (0.12s)
Computing version for https://github.com/Zewo/CLibreSSL.git
Computed https://github.com/Zewo/CLibreSSL.git at 3.1.0 (0.11s)
Computing version for https://github.com/onevcat/Rainbow.git
Computed https://github.com/onevcat/Rainbow.git at 4.0.1 (0.12s)
Computing version for https://github.com/tsolomko/SWCompression
Computed https://github.com/tsolomko/SWCompression at 4.7.0 (0.12s)
Computing version for https://github.com/tsolomko/BitByteData
Computed https://github.com/tsolomko/BitByteData at 2.0.1 (0.12s)
Everything is already up-to-date
'CLibreSSL' /Volumes/Samsung/Documents/late/Automotive-Swift/ELMpersonator/.build/checkouts/CLibreSSL: warning: ignoring declared target(s) 'CLibreSSL' in the system package
warning: failed to retrieve search paths with pkg-config; maybe pkg-config is not installed
warning: you may be able to install libtls using your system-packager:
    brew install libressl

[0/0] Build complete!

Ok, now I don't change a single bit. I just call swift build again and it does the same walking.
Dito for swift run which is even more annoying, but at least I can help w/ .build/debug/whatever ....

Isn't it supposed to cache that info?

I'm not referencing any local dependencies. I did not open a bug @ swift.org yet, but if you say so, I will.

There have been a handful of different loopholes that were or are handled inconsistently, failing validity checks when loading the cache, thereby appearing stale, but nevertheless passing the fresh graph resolution triggered by a stale cache. I do not know which version of the toolchain you are using, so I do not know which particular ones to point to. However, all the bugs I know of have boiled down to the same package being referenced by subtly different URLs in different places in the graph, such as with .git in one place, and without it in another. If the mismatched reference is in your package, you can avoid the issue immediately by adjusting the reference, but if it is a disagreement between two of your dependencies about a third transitive dependency, then you may simply have to wait for new releases of either said dependencies or the toolchain. (I think all such issues should have been resolved in the main branch of SwiftPM by now.)

1 Like

I built your package with the main branch of SwiftPM. It succeeded, but flagged this warning:

'swift-socketcan': warning: 'swift-socketcan' dependency on 'https://github.com/AutomotiveSwift/Swift-CAN' conflicts with dependency on 'https://github.com/Automotive-Swift/Swift-CAN' which has the same identity 'swift-can'. this will be escalated to an error in future versions of SwiftPM.

1 Like

If you or anyone else wants to know how to debug their own package that is behaving this way, simply:

  1. git clone https://github.com/apple/swift-package-manager
  2. cd swift-package-manager
  3. swift build
  4. .build/my-target-triple/debug/swift-build --package-path /Users/Me/Wherever/My/Own/Package/Is

Then inspect the output for warnings like the one in the previous comment.

1 Like

Yikes, is this a case of typo squatting? AutomotiveSwift vs. Automotive-Swift?

Next to Xcode 13.2.1 on macOS, I'm running Swift 5.5.2 on an Ubuntu derivative here.

On Linux: Unfortunately this crashes while building with Swift 5.5.2 (Update: Also with the main toolchain).

With regards to Automotive-Swift vs. AutomotiveSwift, I have fixed the offending dependency, but I don't see any change in behavior. At least with Swift 5.5.2, the tree is still being walked every time.

I just have tried with a recent main toolchain and this does not change anything either.

On macOS: Here, I can at least successfully build GitHub - apple/swift-package-manager: The Package Manager for the Swift Programming Language, but building ELMpersonator exhibits the same behavior as on linux – walks the tree every time.

I guess it does not report every kind of problem yet. I was sure the code for that passed under my eyes recently, but maybe it is still in an unmerged PR. @tomerd?

In any case, the same information can already be fished out with the official release of Swift 5.5.2, you just have to dig deeper to get it.

swift build # Get most of the work out of the way.
swift build --verbose

Shortly before the familiar resolution log entries you always see, --verbose will also give the reason it decided resolution was necessary. In this case it reads:

Running resolver because dependency 'CornucopiaCore' was resolved to 'master' but now has a different revision-based requirement.

While the human‐readable description is a little off (the mismatch is not the requirement), it did correctly report which dependency has a pseudo‐conflict.

To quickly find where in the graph that dependency is, use this command:

swift package show-dependencies

Searching for the name reveals these entries:

.
[...]
├── swift-uds<https://github.com/Automotive-Swift/Swift-UDS@unspecified>
│   └── cornucopiacore<https://github.com/Cornucopia-Swift/CornucopiaCore@unspecified>
[...]
├── ecumulator<https://github.com/Automotive-Swift/ECUmulator@unspecified>
│   ├── cornucopiacore<https://github.com/Cornucopia-Swift/CornucopiaCore@unspecified>

From there, I can look up the respective manifests and see that one asks for .git and the other does not.

Since it short‐circuits on the first “problem” it finds, you may have to do it over again to find additional pseudo‐conflicts after you have fixed the first.

5 Likes

Amazing, thanks very much for your timely and precise help!

I can understand that a difference in the base name confuses the package resolution, but wouldn't you agree that the absence or presence of .git shouldn't make a difference – hence could be classified as a bug? After all, the git system does not make a difference either.

the presence of .git suffix should not make a difference, please file a Jira ticket with the details on this thread and we will look into it.

2 Likes

I have referenced this thread from the already existing tickets

Is that sufficient?

2 Likes

I'm not sure whether it is possible to say whether the ".git" suffix makes a difference or not, just like it isn't possible to say whether case makes a difference. Ultimately, servers are free to process requests however they want. Sometimes you can do protocol-specific normalization; for example, the HTTP standard says:

The scheme and host are case-insensitive and normally provided in lowercase; all other components are compared in a case-sensitive manner. Characters other than those in the "reserved" set are equivalent to their percent-encoded octets: the normal form is to not encode them

HTTP 1.1, 2.7.3

So we may remove redundant percent-encoding for the purposes of client-side processing (but it is not required), but ignoring case or removing a ".git" suffix is, in general, not safe and meaningfully alters the request target.

That said, there's nothing stopping SwiftPM from detecting common mistakes and presenting a diagnostic or providing some "use at your own risk" flag to enable non-standard normalization -- as long as the risks are clearly communicated.

6 Likes

Is there any way to get these diagnostics from the Xcode UI?

Not that I am aware of.

Thank you so much, I just run into using .git and not in two different places and had the same issue - this thread saved me :-)

1 Like
Terms of Service

Privacy Policy

Cookie Policy