Resolving Swift Package via Multiple Protocols

Development Package Manager
1

(Note: I am not good at editing myself and this turned out longer than I expected when I started. Sorry.)

Motivation

We are migrating our frameworks to SwiftPM packages. Most of that is going smoothly. The part that is not, and at one time was a blocker, is how to handle dependencies from private authenticated Git repos in our CI environment.

Currently we manage our project dependencies via git submodules. Submodules have their own challenges and I'm not trying to draw comparisons. They do however have one major redeeming quality in our environment. That is because the CI provider, GitLab, has good support out of the box for handling git submodule cloning.

The tl;dr from the documentation linked above: if you provide a relative repository URL then submodule resolution uses the same protocol and credentials used to clone the parent.

Why is this important? Our developers use Git over SSH to clone/push/pull the repos for their environment. While the CI uses Git over HTTPS to clone using a single use job token. What is more our developers could start using Git over HTTPS or CI Git over SSH and no changes would need to be made to the submodule configuration. Neither use case is prioritized over the other in the configuration.

SwiftPM

This brings me to SwiftPM's dependency declaration.

...
dependencies: [
  // Dependencies declare other packages that this package depends on.
  // .package(url: /* package url */, from: "1.0.0"),
  .package(url: "git@gitlab.private.host.local:path/repo.git", .revision("dfbb450d5627faba33c9cb06d203c5b930bd5f8a")),
],
...

And subsequent use of the dependency in an environment that does not use Git over SSH (e.g., our CI environment).

$ swift build
Fetching git@gitlab.private.host.local:path/repo.git
error: failed to clone; Cloning into bare repository '/Users/buildbot/builds/0/path-repo/path/repo/.build/repositories/repo-cdaa868b'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Ultimately this whole post boils down to this question. What is the best way to solve that error in SwiftPM?

Work Around

The work around we have come up with is pretty straightforward. Only use Git over SSH for developer cloning. And use sed to find the URL in Package.swift and Package.resolved and replace it with a working version of the URL for the desired environment.

For example in the CI environment where CI_SERVER_HOST (e.g., gitlab.private.host.local) and CI_JOB_TOKEN (e.g., deadbeef1234) are provided we do something like:

$ sed -i .old 's/git\@'"$CI_SERVER_HOST"':/https:\/\/gitlab-ci-token:'"$CI_JOB_TOKEN"'\@'"$CI_SERVER_HOST"'\//' Package.*

No doubt there is some edge case this will not cover but it works pretty well and has us moving forward again.

Future

Is there a better solution I can source from the community?

Assuming that there isn't already a better way to overcome this. I cannot help but wonder if there is some way for us to provide a similar functionality to the Git submodule feature where we provide a relative URL and SwiftPM resolves the relative URL using git config --get remote.origin.url (or a more robust version of the same idea).

That way in the Package.swift we could declare something like
.package(relativeURL: "../../path/repo.git", .revision("dfbb450d5627faba33c9cb06d203c5b930bd5f8a")), and have module resolution just work.

I do understand though there may be no appetite for this in the main project because either: it's not a general problem or it makes too many assumptions about the environment (namely that the Swift package is in a Git repo to begin with). I don't think it hurts to ask.

2

I moved the topic to the package manager development category, where your post is more likely to be seen by the relevant people.

(I myself know too little about those parts of the package managerā€™s internals, and thus have no meaningful comments of my own to make.)

3

Your CI system doesn't allow you to inject SSH credentials? That seems like the obvious answer.

4

It probably does allow that. Though I personally dislike the idea of there being hard-coded credentials (either injected into the environment or provided as part of the repo) out there that effectively do not expire. I really like the idea of the ephemeral credentials. And as far as I know, it does not provide ephemeral SSH keys (which would be awesome).

Regardless, I think that side-steps the fundamental question and that's my fault for focusing too much on my motivating example (e.g., CI).

How would SwiftPM resolve the same package using multiple protocols?

  • Developer A uses Git over HTTPS and a ~/.netrc file for authentication.
  • Developer B uses Git over SSH and a ~/.ssh/id_rsa file for authentication.

As far as I can tell, Package.swift file encodes the protocol therefore SwiftPM can only support one transport protocol at a time. Meaning that one of those 2 developers has to abandon their workflow.

Is that worth-while addressing? To me it is. The fact that Git itself provides a protocol agnostic option in the submodule configuration possibly implies that they also felt it worth addressing.

Then again, maybe this takes the project to being too closely dependent on Git.

5

Does the sed workaround actually work when the package graph is multiple levels deep?

Would mirroring the URLs help somehow in your environment?

6

At this point, that is not a problem we have faced. We are only probably 25% of the way through our porting. Though thinking it through it is a problem we will face in the future. I would not expect the sed hack to work. Back to the drawing board. :thinking:

I was unaware of this feature. I'm going to have to test that one. The thing I'm afraid is going to happen is I'm going to have to maintain effectively 2 dependency lists. One in the Package.swift file(s) and then again in my CI configuration so that I can call swift package config set-mirror for each dependency.

Though I think that would work around the multiple level problem.

7

Mirroring isnā€™t a feature Iā€™ve used, but what I sort of imagined was that the CI infrastructure would have a single file available outside any particular package with a global list mirroring every relevant package URL to the opposite scheme. Then the scheme could be swapped for any package by merely setting SWIFTPM_MIRROR_CONFIG so that the package manager sees the mirror list or not. The file itself could be (reā€)generated rather easily with a script that operates on a list of packages.

8

It seems that I could auto generate a mirror configuration file using something like swift package show-dependencies. That file could then be kept in the project repo. Then through a combination of .netrc and SWIFTPM_MIRROR_CONFIG use it during CI.

The trick is that we'd have to remember to up date it when a dependency is added or changed. But presumably the CI would catch it since it'd fail to build. :laughing:

It is definitely a bit clunky. Though it seems like it would work.

1 Like
9

Yeah, Iā€™m just talking in terms of workarounds. Whether or not to directly support this sort of setā€upā€”and howā€”are questions better handled by others.

1 Like
10

Frankly, this seems like something that can be handled at a variety of other levels more appropriately, rather than changing SPM.

  • Injecting credentials is standard practice, usually from a bot / CI only user with minimal privileges.
  • The CI system itself may be able fallback between different Git URLs.
  • Whether SSH or HTTPS is used is a team standard, not something left to individual choice, especially when there are shared services.

If we do want this in SPM, perhaps a better solution would be an update to the package parameters to allow something like .git(github.com/team/repo) and allow the tool to generate the appropriate URL, rather than having to mutate what the user has entered.

11

I've only worked on a few teams but I've never had a team where everyone accessed the repos via the same protocol. Even GitHub allows the same repository to be accessed via multiple protocols. As far as I know there is no way to disable one in favor of the other.

If it's the communities opinion that SwiftPM should have a convention and to only support one protocol at a time, fair enough. But personally I'd rather not pretend that's a limitation of the Git ecosystem and acknowledge that's a SwiftPM design decision.

12

Top level repos, sure, since no one needs to work on the same clone. Tool configurations, since they're shared, are commonly set to prefer one or the other, depending on the tools. As such they are subject to the team agreement, or whatever covenant governs choices like that among the team.

Except it is a limitation of the git ecosystem, as git doesn't do this remapping automatically. Are there any tools that will automatically remap these URLs? None of the package managers I've used do, largely because they just pass git URLs to git or its APIs.

13

Well as I have said git submodule does. But I'm starting to get the impression you feel like that does not count.

14

That's not remapping, just allowing the submodules to follow the same protocol as the parent. The closest thing I can think of for SPM would be for SPM to remap all git URLs to the containing repo's protocol. But again, I think we'd want an abstract .git(host/path) for that to make sense.

15

That feels like a semantic difference. Regardless of the name we assign to it, remapping or following the same protocol as the parent, the methodology allows the goal to be achieved. And, at least in the context of this discussion, I'd consider git submodule to be performing many of the same functions as a package manager. Therefore, I do see at as something for SwiftPM to use as a blueprint.

If I had to assign a name or description to what I'm proposing I'd describe it is expanding relative URLs. :man_shrugging:

All I can say is my team has done this for years and never had a problem in mixed environments; of course using submodules and not Cocoapods or Carthage or another higher level manager. But based on this feedback it is possible we are the exception and not the rule.

16

I would have also suggested using dependency mirroring. As far as I know, this use case isn't extremely common (I haven't heard it before at least), so doesn't seem like a good candidate for an explicit feature.

In theory, you can also keep using git submodules and utilise path-based dependencies to reference packages, but that would mean you will continue to have to manage the submodule references yourself.

17

Oh I like that. That's a really good suggestion.

18

Note that only works for a top level package (which is true of a lot of advanced techniques that use outside tooling or shell commands).

Attempting to depend on a package that contains submodules will not work. The package manager doesnā€™t check out submodules when it clones a dependency (unless it started and I missed the memo). You will just get missing source errors. Even if it did, you would run into trouble when you end up with a diamondā€shaped dependency tree (A ā†’ B, A ā†’ C, B ā†’ D, C ā†’ D) where the bottom is a submodule, since its source would be checked out in two locations as separate targets, leading at best to redundant portions of the binary and at worst to breaking name collisions.

19

You're right about the possibility of the diamond problem but SwiftPM does initialize and update the submodules in the dependencies: swift-package-manager/GitRepository.swift at main Ā· apple/swift-package-manager Ā· GitHub

1 Like
20

Youā€™re right, it does work for dependencies. (And Iā€™ve been wrong about it for a long time...)

Iā€™m guessing my misconception must have originated from my minimal experiments having the submodule in the root package. How much work do you think it would be to support them for the root package too, so that git clone some.url ; cd ThePackage ; swift build would work as seamlessly when submodules are involved as it does when they arenā€™t?