Support for Private Repositories

[GetSwifty]

With Xcode's upcoming support for SPM, I'm hoping to transition my and my organization's codebases to use Packages. The biggest complication I've had is lack of real support for anything non-OSS.

The only documentation/approach I've found is from Pulling private SPM repos in Xcode and the mentioned [IBM blogpost]*(IBM Developer) (and various reposts/stackoverflow of basically the same thing). That solution is probably possible to use but is far from user friendly, platform dependent, and not a generalized solution. Any new team member, build server, or client needing access has to go through a multi-step, manual process just to import a package.

Glancing around at few dependency managers it doesn't seem like there's a standard way of doing this well without some addition layer, so I'm guessing the problem is difficult to solve. It's also the reason I've found it simpler to deal with git-submodules rather than a "real" dependency management solution. With a (finally!) first party solution I think I can justify switching for public libraries, but I would really like to git rid of that submodule approach for iOS projects altogether.

So a few questions.

1. Are there any other approaches or services being been used?
2. Is there any official documentation on the issue?
3. Are there any plans/work toward official support?

[jack (Jack Lawrence)]

Could you describe the specific problems you're encountering using SPM vs submodules? As long as your SSH key provides access to all the repositories you depend on, it should work.

[GetSwifty]

@jack

As long as your SSH key provides access to all the repositories you depend on

That's my point: It's magic, undocumented, adds additional setup steps for every new machine, and is a process that differs between services and platforms. It appears when used through the Xcode-Beta it will at least prompt for username/password when using a private GitHub repo. The same is not true for swift build, or the private hosting I have access to.

This is something git/git-clients handle or at least surface meaningful errors with available solutions, whether for login credentials or SSH keys. My experience thus far with SPM is generic errors that don't feel actionable: fatal: could not read Username for 'https://github.com': terminal prompts disabled. Services like Github Package Registry may make this part simpler, but it's still unclear how that will integrate with SPM.

Based on what I've seen, it appears SPM relies on git's infrastructure around this (though I haven't been able to find any related documentation). Maybe better integration with that part of git is needed?

Anyway, I'm thinking what I'm really looking for is additional (official...or really any) documentation around private-repos with SPM. I've never been happy with SSL keys as a repository-access solution, but improving that is probably outside the scope of SPM.

[NeoNacho (Boris Buegling)]

Setting Up a Continuous Integration Pipeline has a "Provide Credentials" section that talks a little bit about this, it is pretty similar to what is outlined in the IBM article you mentioned.

I am curious on how this is different from Git submodules, though, presumably you also have to configure authentication for the top-level repository in a similar way. Is the issue that you will be using a lot of different hosts for your various packages, so you need to configure authentication multiple times?

[GetSwifty]

Thanks for the link, didn't look in the CI section!

I regularly use 3 different hosts, and was trying to use an as-minimal-as-possible setup in my Catalina Volume with just Xcode. Other than with Xcode+github I wasn't able to login with a username/password. swift build didn't work at all. So I setup SSH, but kept getting opaque errors (turns out I setup known_hosts wrong...like a boss ). I'm still figuring out SPM, so I expected it was something I had configured wrong there, not SSH.

When working with git and/or submodules (at least via smartgit) I'm interacting with the actual tool having problems rather than issues being obfuscated through a second tool. And once you sign in, you have the repo and you're good to go. You don't have to sign in with xcode as well, or add an SSH key. Worst-case scenario, I can just send someone the zip of the project.

I've already suffered through dealing with this kind of thing so I can figure out how to make it work, but new/inexperienced users won't be so...lucky? There are a lot of potential pitfalls in the process that can come up with a new team member or client, and explaining it isn't going to be easy.

I know you signed in to clone/download the repo, but Xcode and SPM don't work with that so you have to follow this 5+ step process that you have to just know about and then you can build the project.

What I really want is an OAuth-esque system for private repos that integrate with SPM/git. In lieu of that unlikely development, I'm guessing future services and blog posts will pop up with the adoption uptick happening, but better git integration and error messages would also be nice

[NeoNacho (Boris Buegling)]

Thanks for the clarification, so your issue isn't really lack of support for your configuration, but rather that the usability and diagnostics weren't good enough? That's helpful feedback.

One thing that could make this a bit easier for you might be either

• using xcodebuild for building on the CLI, that way you'd share the authentication configuration with Xcode
• setting up SSH keys first, that way they would be used by Xcode as well

[GetSwifty]

@NeoNacho

so your issue isn't really lack of support for your configuration, but rather that the usability and diagnostics weren't good enough?

I think that's right. If I didn't screw up the SSH setup I probably wouldn't have had any issues

Based on the lack of documentation (other than in the CICD section, which I didn't think applied), I almost concluded it wasn't supported.

More documentation for "Private Packages" or something would at least help with people in the same predicament.

E.g. here it says

By publishing your Swift packages to private Git repositories, you can manage and integrate internal dependencies across your projects, allowing you to reduce duplicate code and promote maintainability

...but there's no more information on how that's supposed to work and none of the links seem directly related

[GetSwifty]

A related note/request in case someone stumbles across this. An internal git server I'm using is accessed through a proxy. git and "swift build" work just fine when using the normal ssh-git url, but it seems Xcode doesn't respect know_hosts and needs the resolved host url (the IP address in my case).

In any case I think I'm finally fully up and running

[NeoNacho (Boris Buegling)]

I don't think we are aware of the proxy issue you are describing. Could you please file a feedback report for this?

[deepesh-vasthimal-ck (Deepesh Vasthimal Cko)]

@GetSwifty Hi, What was the real solution around accessing private repos ? I have raised a stackoverflow and the response i have got is since its a private repo you cant access through SPM, But my question is does SPM support private repos ? and if so then how can i get it working. refer : ios - Swift package manager: artifact of binary target failed download: invalid status code 404 - Stack Overflow

[GetSwifty]

It uses the built-in functionality of git. So e.g. if you set up SSH you can use that URL for a package. (e.g. for github: git@github.com:UserName/RepoName)

[nighthawk (Adrian Schoenig)]

I came across a similar issue right now in a private package A that depends on another private package B, where both are hosted on GitHub and I'm using GitHub Actions with a swift build step.

This is using shared runners, so I have limited control over them. An alternative approach that I ended up with was to to a step for checking out package B in a subfolder when package A is built, and then conditionally using that in my Package.swift:

let privateDependencies: [Package.Dependency]
if ProcessInfo.processInfo.environment["CI"] != nil {
  privateDependencies = [
    .package(name: "PackageB", path: "PackageB"),
  ]
} else {
  privateDependencies = [
    .package(name: "PackageB", url: "https://github.com/team/package-b.git", .branch("main")),
  ]
}

let package = Package(
  name: "PackageA",
  ...
  dependencies: privateDependencies + [
    ...
  ],

I wonder if that's a bad idea and I'm setting myself up for failure when A is used as a dependency in another project or package, but it's working smoothly now for building A using GitHub Actions.

[stepanulyanin]

@GetSwifty

UserName

Hi all!

Was there any movement on simplifying the private repository dependencies?

I constantly get:

error: Failed to clone repository https://github.com/....git:
    ...
    fatal: could not read Username for 'https://github.com': terminal prompts disabled

Even though the dependency is specified as ssh:

.package(url: "git@github.com:....git", from: "0.0.0")

I, personally found the support for the private dependencies quite frustrating

[swhitty (Simon Whitty)]

Since Xcode 12.5 the default for xcodebuild is to use the system git for resolving packages.

I have used git url.<base>.insteadOf to allow CI to resolve private HTTPS repos.

See more:

Support for resolving private packages through HTTPS with xcodebuild

The way we do it in our CI is "this one weird trick". We (ab)use Git's url.<base>.insteadOf to re-write the clone URL to include the appropriate credentials in our pipeline and make use of xcodebuild's -usePackageSupportBuiltinSCM. Our CI of choice is GitLab and it provides ephemeral credentials as environment variables to clone over HTTPS for each job. Specifically it provides, CI_JOB_TOKEN, CI_SERVER_PROTOCOL, and CI_SERVER_HOST. Typically, our URLs in our Xcode and Package.swift files are o…

[afeick (A)]

I've been struggling with getting private repos to work with GitHub actions and I just figured out a similar process to what @swhitty described, but it uses the github cli (gh) to help configure git for https

# ignore failures because the 'gh auth login' command returns a GraphQL API error for fine-grained personal access tokens
set +e
gh auth login --with-token <<< ${{ secrets.CUSTOM_PAT_TOKEN }}
gh auth status

# force git to use https instead of ssh so we can use gh auth for git
git config --global url."https://github.com/".insteadOf "git@github.com:"

# configure git operations for https protocol with github.com
gh auth setup-git

[TomoXD (Tomasz Lizer)]

This is quite old thread and maybe that has been resolved in some other way... Nevertheless I have faced that myself with GitHub Actions and project depending on private Swift Packages.
Obvious solution is to use SSH, however I could not force IT team in my organisation to do so (we have policy o fusing tokens only).
Looking into the problem I have came to similar solution that @afeick mentioned:

• Configure git with access token
• Force git to use https

I have wrapped that into GitHub action which handles such setup and after job is complete it also handles teardown (removes token, clears custom git setup).
If someone needs a hand, here it is:

Setup token for git host - GitHub Marketplace

Hope it will help someone ;)