Mirroring isn’t a feature I’ve used, but what I sort of imagined was that the CI infrastructure would have a single file available outside any particular package with a global list mirroring every relevant package URL to the opposite scheme. Then the scheme could be swapped for any package by merely setting SWIFTPM_MIRROR_CONFIG so that the package manager sees the mirror list or not. The file itself could be (re‐)generated rather easily with a script that operates on a list of packages.
It seems that I could auto generate a mirror configuration file using something like swift package show-dependencies. That file could then be kept in the project repo. Then through a combination of .netrc and SWIFTPM_MIRROR_CONFIG use it during CI.
The trick is that we'd have to remember to up date it when a dependency is added or changed. But presumably the CI would catch it since it'd fail to build. 
It is definitely a bit clunky. Though it seems like it would work.
1 Like
Yeah, I’m just talking in terms of workarounds. Whether or not to directly support this sort of set‐up—and how—are questions better handled by others.
1 Like
Frankly, this seems like something that can be handled at a variety of other levels more appropriately, rather than changing SPM.
- Injecting credentials is standard practice, usually from a bot / CI only user with minimal privileges.
- The CI system itself may be able fallback between different Git URLs.
- Whether SSH or HTTPS is used is a team standard, not something left to individual choice, especially when there are shared services.
If we do want this in SPM, perhaps a better solution would be an update to the package parameters to allow something like .git(github.com/team/repo) and allow the tool to generate the appropriate URL, rather than having to mutate what the user has entered.
I've only worked on a few teams but I've never had a team where everyone accessed the repos via the same protocol. Even GitHub allows the same repository to be accessed via multiple protocols. As far as I know there is no way to disable one in favor of the other.
If it's the communities opinion that SwiftPM should have a convention and to only support one protocol at a time, fair enough. But personally I'd rather not pretend that's a limitation of the Git ecosystem and acknowledge that's a SwiftPM design decision.
Top level repos, sure, since no one needs to work on the same clone. Tool configurations, since they're shared, are commonly set to prefer one or the other, depending on the tools. As such they are subject to the team agreement, or whatever covenant governs choices like that among the team.
Except it is a limitation of the git ecosystem, as git doesn't do this remapping automatically. Are there any tools that will automatically remap these URLs? None of the package managers I've used do, largely because they just pass git URLs to git or its APIs.
Well as I have said git submodule does. But I'm starting to get the impression you feel like that does not count.
That's not remapping, just allowing the submodules to follow the same protocol as the parent. The closest thing I can think of for SPM would be for SPM to remap all git URLs to the containing repo's protocol. But again, I think we'd want an abstract .git(host/path) for that to make sense.
That feels like a semantic difference. Regardless of the name we assign to it, remapping or following the same protocol as the parent, the methodology allows the goal to be achieved. And, at least in the context of this discussion, I'd consider git submodule to be performing many of the same functions as a package manager. Therefore, I do see at as something for SwiftPM to use as a blueprint.
If I had to assign a name or description to what I'm proposing I'd describe it is expanding relative URLs. 
All I can say is my team has done this for years and never had a problem in mixed environments; of course using submodules and not Cocoapods or Carthage or another higher level manager. But based on this feedback it is possible we are the exception and not the rule.
I would have also suggested using dependency mirroring. As far as I know, this use case isn't extremely common (I haven't heard it before at least), so doesn't seem like a good candidate for an explicit feature.
In theory, you can also keep using git submodules and utilise path-based dependencies to reference packages, but that would mean you will continue to have to manage the submodule references yourself.
Oh I like that. That's a really good suggestion.
Note that only works for a top level package (which is true of a lot of advanced techniques that use outside tooling or shell commands).
Attempting to depend on a package that contains submodules will not work. The package manager doesn’t check out submodules when it clones a dependency (unless it started and I missed the memo). You will just get missing source errors. Even if it did, you would run into trouble when you end up with a diamond‐shaped dependency tree (A → B, A → C, B → D, C → D) where the bottom is a submodule, since its source would be checked out in two locations as separate targets, leading at best to redundant portions of the binary and at worst to breaking name collisions.
You're right about the possibility of the diamond problem but SwiftPM does initialize and update the submodules in the dependencies: https://github.com/apple/swift-package-manager/blob/master/Sources/SourceControl/GitRepository.swift#L378
1 Like
You’re right, it does work for dependencies. (And I’ve been wrong about it for a long time...)
I’m guessing my misconception must have originated from my minimal experiments having the submodule in the root package. How much work do you think it would be to support them for the root package too, so that git clone some.url ; cd ThePackage ; swift build would work as seamlessly when submodules are involved as it does when they aren’t?
I am not sure if it's right to automatically checkout submodules for the root package. In general, SwiftPM avoids performing git operations on the root package (there might not even be a git repository).
1 Like
An alternative solution might involve using URL rewriting provided in Git. We use this on developer machines to rewrite all SSH connections to HTTPS for GitHub to deal with restrictive proxies.
2 Likes
.netrc support being discussed here also: SPM support basic auth for non-git binary dependency hosts.
This is such a great solution/suggestion/workaround. For the first time my private SwiftPM dependencies resolve in my CI. A quick before_script step in my GitLab CI configuration to call git config and it works 
.
swiftpm_build:
stage: build
before_script:
- "git config --global url.$CI_SERVER_PROTOCOL://gitlab-ci-token:$CI_JOB_TOKEN@$CI_SERVER_HOST/.insteadOf git@$CI_SERVER_HOST:"
- swift package resolve
script:
- swift build -c release
tags:
- swift-5.0
Then the dark times... xcodebuild. Why xcodebuild? Because that's how you run SwiftPM tests on iOS. I do not make the rules. I just work here.
xcodebuild:
stage: build
variables:
DESTINATION: platform=iOS Simulator,name=iPad Pro (9.7 inch),OS=10.3.1
before_script:
- "git config --global url.$CI_SERVER_PROTOCOL://gitlab-ci-token:$CI_JOB_TOKEN@$CI_SERVER_HOST/.insteadOf git@$CI_SERVER_HOST:"
- xcodebuild -resolvePackageDependencies
script:
- xcodebuild -enableCodeCoverage YES -scheme "$XCODE_SCHEME" -destination "$DESTINATION" build-for-testing
tags:
- swift-5.0
- iOS-10.3.1
Resolve Package Graph
Fetching git@private.gitlab.instance.local:group/dependency.git
Resolved source packages:
SwiftPMProject: /Users/buildbot/SwiftPMProject
xcodebuild: error: Could not resolve package dependencies:
The repository could not be found. Make sure a valid repository exists at the specified location and try again.
So apparently xcodebuild does its own package resolution outside of Git.
So close to nirvana.
Regardless, thanks @monocularvision
1 Like
Passing -usePackageSupportBuiltinSCM to xcodebuild should work here. By default, Xcode uses its own SCM subsystem for fetching packages, but this option makes it use the one in libSwiftPM, so it should behave similar to swift build at that point.
2 Likes
@NeoNacho thank you; that did work.
Though for any intrepid reader that makes there way here. The -usePackageSupportBuiltinSCM did not work on Xcode/xcodebuild 11.3.1. On 11.3.1 when the tests are run apparently the dependency is not on the @rpath.
It may work on some other version but I just jumped to 11.5.0 and it worked there.