# Support for resolving private packages through HTTPS with xcodebuild **URL:** \ **Category:** Package Manager\ **Tags:** packagemanager\ **Created:** [May 19, 2021, 6:54am UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581 "2021-05-19T06:54:54Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![pepicrft](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/pepicrft/32/43978_2.png) [@pepicrft](https://forums.swift.org/u/pepicrft)\ **Post date:** [May 19, 2021, 6:54am UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/1 "2021-05-19T06:54:55Z")
Hi 👋, We are using the Swift Package Manager in some of our Xcode projects to integrate packages that live in other private repos of the organization. Everything works fine, except when we run builds on CI. Despite having the right Git configuration to authenticate successfully with our Git provider, the `xcodebuild` process fails with the following error: ```bash xcodebuild: error: Could not resolve package dependencies: Failed to clone repository https://github.com/foo/bar: Cloning into bare repository '/Users/anka/Library/Developer/Xcode/DerivedData/...'... fatal: could not read Username for 'https://github.com': terminal prompts disabled ``` After some reading, I found this [blog post](https://medium.com/cristian-ortega/using-swift-package-manager-in-real-life-projects-e80d60edd860) that mentions that HTTPS is not supported, and also noticed that Apple's official documentation mentions [suggests](https://developer.apple.com/documentation/swift_packages/building_swift_packages_or_apps_that_use_them_in_continuous_integration_workflows) to use SSH. Unfortunately, using SSH is not possible with GitHub apps, and the proxy solution suggested in the former post seems too much overhead. Is there a way for xcodebuild's package resolution to successfully do authentication through HTTPS using the global git configuration? ---
**Author:** ![rlovelett](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/rlovelett/32/753_2.png) [@rlovelett](https://forums.swift.org/u/rlovelett)\ **Post date:** [June 9, 2021, 12:58pm UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/2 "2021-06-09T12:58:20Z")
> [@pepicrft](#): > > After some reading, I found this [blog post](https://medium.com/cristian-ortega/using-swift-package-manager-in-real-life-projects-e80d60edd860) that mentions that HTTPS is not supported, and also noticed that Apple's official documentation mentions [suggests](https://developer.apple.com/documentation/swift_packages/building_swift_packages_or_apps_that_use_them_in_continuous_integration_workflows) to use SSH. We use HTTPS and SSH at my work interchangeably using both Xcode and `xcodebuild` (and vanilla SwiftPM for that matter) in our CI. In the end, I think it is fair to say that package manager is not the limitation here but likely your ability to supply the correct credentials while running in your CI. How are you providing your credentials for [https://github.com](https://github.com) during your CI runs? ---
**Author:** ![NeoNacho](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/neonacho/32/1368_2.png) [@NeoNacho](https://forums.swift.org/u/NeoNacho)\ **Post date:** [June 9, 2021, 6:16pm UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/3 "2021-06-09T18:16:34Z")
If you use `xcodebuild -scmProvider xcode`, HTTPS can be used, but you would typically log in via Xcode preferences to your SCM service account. Let me try to find out whether there is a way to accomplish this on a CI system as well. ---
**Author:** ![NeoNacho](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/neonacho/32/1368_2.png) [@NeoNacho](https://forums.swift.org/u/NeoNacho)\ **Post date:** [June 9, 2021, 8:43pm UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/4 "2021-06-09T20:43:57Z")
Looks like there is no alternative to using the Xcode UI today, so you would need to login once via that on the CI machine to store the credentials and then using ` xcodebuild -scmProvider xcode` should be able to use those credentials you configured. ---
**Author:** ![pepicrft](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/pepicrft/32/43978_2.png) [@pepicrft](https://forums.swift.org/u/pepicrft)\ **Post date:** [June 14, 2021, 7:12pm UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/5 "2021-06-14T19:12:26Z")
Hi! Thanks for your answer. Adding a bit of more context, the way we authenticate on CI hosts is by using git's `GIT_ASKPASS` environment variable, and passing an executable that tells Git where to read the username and password from. That's working fine when we run Git in the system. @NeoNacho do you know if we can automate the process of storing the credentials where Xcode expects them? Unfortunately, setting the token in our CI hosts through Xcode's UI is not viable because we rotate the credentials periodically. Thanks in advance! ---
**Author:** ![NeoNacho](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/neonacho/32/1368_2.png) [@NeoNacho](https://forums.swift.org/u/NeoNacho)\ **Post date:** [June 15, 2021, 5:38pm UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/6 "2021-06-15T17:38:07Z")
> [@pepicrft](#): > > That's working fine when we run Git in the system. Oh, if git on the system works fine, `-scmProvider system` is actually the right thing. This is the default in Xcode 12.5 and later, perhaps you are using a previous version? ---
**Author:** ![pepicrft](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/pepicrft/32/43978_2.png) [@pepicrft](https://forums.swift.org/u/pepicrft)\ **Post date:** [June 17, 2021, 1:52pm UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/7 "2021-06-17T13:52:03Z")
Interesting, we are for instance using Xcode 12.5. Is it possible that the `GIT_ASKPASS` variable is not exposed to the Swift Package Manager process? We've verified that the variable works at the system level because we can clone private repositories. I don't know what else could be 😅. ---
**Author:** ![rlovelett](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/rlovelett/32/753_2.png) [@rlovelett](https://forums.swift.org/u/rlovelett)\ **Post date:** [June 17, 2021, 4:03pm UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/8 "2021-06-17T16:03:33Z")
The way we do it in our CI is "this one weird trick". We (ab)use Git's [`url..insteadOf`](https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf) to re-write the clone URL to include the appropriate credentials in our pipeline and make use of `xcodebuild`'s `-usePackageSupportBuiltinSCM`. Our CI of choice is GitLab and it provides ephemeral credentials as environment variables to clone over HTTPS for each job. Specifically it provides, [`CI_JOB_TOKEN`](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html), [`CI_SERVER_PROTOCOL`](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html), and [`CI_SERVER_HOST`](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html). Typically, our URLs in our Xcode and `Package.swift` files are of the SSH variety (e.g., `git@my.git.server.org:group/project.git`). So we use Git's [`url..insteadOf`](https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf) to re-write the URL to be HTTP(S) and to include the appropriate credentials. ### Example ```plaintext git config --global url.$CI_SERVER_PROTOCOL://gitlab-ci-token:$CI_JOB_TOKEN@$CI_SERVER_HOST/.insteadOf git@$CI_SERVER_HOST: ``` Assuming that `CI_SERVER_HOST=my.git.server.org`, `CI_SERVER_PROTOCOL=https`, and `CI_JOB_TOKEN=WF0IjoxNTE2MjM5MDIyfQ`. This translates any URL in either or Xcode or `Package.swift` that might look like `git@my.git.server.org:group/project.git` → `https://gitlab-ci-token:WF0IjoxNTE2MjM5MDIyfQ@my.git.server.org/group/project.git`. Then using `xcodebuild -usePackageSupportBuiltinSCM` along with all of our other normal flags things build just fine. That `git config` is cut-n-paste from our `.gitlab-ci.yml` file and we have found this to be a relatively painless and robust way of handling HTTPS Git authentication. YMMV. Hope it helps. ---
**Author:** ![pepicrft](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/pepicrft/32/43978_2.png) [@pepicrft](https://forums.swift.org/u/pepicrft)\ **Post date:** [June 18, 2021, 10:36am UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/9 "2021-06-18T10:36:51Z")
Thanks @rlovelett, we were not passing the `-usePackageSupportBuiltinSCM` argument when running `xcodebuild`. I added the argument, but it still fails. This is the entire `xcodebuild` command that we run. ```swift xcodebuild build-for-testing -workspace Appxcworkspace -configuration Debug -scheme Tests -scmProvider system -usePackageSupportBuiltinSCM -destination '...' ``` One thing to note, is that we don't authenticate using the `GIT_ASKPASS` variable as I mentioned, but we create a `gitconfig` file in the system, that has a credentials helper set that takes care of passing the username and password when Git needs them. The reason why we don't put a token there is because we rotate those for security reasons, and we need the Git process to get it from another process. Looking at the logs: ```swift xcodebuild: error: Could not resolve package dependencies: Failed to clone repository https://github.com/organization/repo: Cloning into bare repository '/Users/anka/Library/Developer/Xcode/DerivedData/Shopify-eotrbxjhnrnkesdptcwwzebeoowa/SourcePackages/repositories/repo-0dd4e281'... fatal: could not read Username for 'https://github.com': terminal prompts disabled ``` I wonder if the Git process fails because we disable the terminal prompts in the process run by Xcode? Shouldn't Git read the credential helper from the `gitconfig` file, and shell out to the helper instead of trying to read it from the standard input? ---
**Author:** ![rlovelett](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/rlovelett/32/753_2.png) [@rlovelett](https://forums.swift.org/u/rlovelett)\ **Post date:** [June 18, 2021, 9:35pm UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/10 "2021-06-18T21:35:12Z")
> [@pepicrft](#): > > The reason why we don't put a token there is because we rotate those for security reasons, and we need the Git process to get it from another process. This is more or less what GitLab does too. Ours rotate for every job, hence the environment variable. If it were my build environment I'd switch to using environment variables. But this is not strictly necessary I suppose. > [@pepicrft](#): > > but we create a `gitconfig` file in the system How? Are you sure the `.gitconfig` file you are creating the _right_ `.gitconfig` file? We set our `.gitconfig` properties via [`git config --global`](https://git-scm.com/docs/git-config) rather than writing the file ourselves. Ensuring that `git` should always know the correct location. ---
**Author:** ![pepicrft](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/pepicrft/32/43978_2.png) [@pepicrft](https://forums.swift.org/u/pepicrft)\ **Post date:** [June 21, 2021, 10:57am UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/11 "2021-06-21T10:57:15Z")
> [@rlovelett](#): > > How? Are you sure the `.gitconfig` file you are creating the _right_ `.gitconfig` file? We set our `.gitconfig` properties via [`git config --global`](https://git-scm.com/docs/git-config) rather than writing the file ourselves. Ensuring that `git` should always know the correct location. We create a file `/usr/local/etc/gitconfig` containing among other things: ```swift [credential "https://github.com"] helper = !path/to/binary git credentials useHttpPath = true ``` With that configuration, we can clone private repositories successfully. The problem seems to be Xcode not picking that configuration for some reason. ---
**Author:** ![tarbayev](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tarbayev/32/9470_2.png) [@tarbayev](https://forums.swift.org/u/tarbayev)\ **Post date:** [January 16, 2023, 6:11pm UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/12 "2023-01-16T18:11:43Z")
The most reliable solution for me was using netrc file: ```swift echo -e "machine github.com\n login oauth2\n password $GITHUB_TOKEN\n\n" > ~/.netrc ``` It works well without additional setup for both `xcodebuild` (even without `-scmProvider system`) and `swift build` and does not create any records in the keychain (which might fail other jobs). ---
**Author:** ![menjoo](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/menjoo/32/33974_2.png) [@menjoo](https://forums.swift.org/u/menjoo)\ **Post date:** [February 8, 2024, 3:17pm UTC](https://forums.swift.org/t/support-for-resolving-private-packages-through-https-with-xcodebuild/48581/13 "2024-02-08T15:17:12Z")
For us using the netrc file does the trick. But note that besides: ```swift machine github.com login password ``` You also need to add this if your Package.swift points to binaries stored in GitHub package registry: ```swift machine maven.pkg.github.com login password ``` Took me quite a while to figure this out, but sub domains are not included in the machine definitions, so specify them individually.