Hi ,
We are using the Swift Package Manager in some of our Xcode projects to integrate packages that live in other private repos of the organization. Everything works fine, except when we run builds on CI. Despite having the right Git configuration to authenticate successfully with our Git provider, the xcodebuild process fails with the following error:
xcodebuild: error: Could not resolve package dependencies:
Failed to clone repository https://github.com/foo/bar:
Cloning into bare repository '/Users/anka/Library/Developer/Xcode/DerivedData/...'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
After some reading, I found this blog post that mentions that HTTPS is not supported, and also noticed that Apple's official documentation mentions suggests to use SSH. Unfortunately, using SSH is not possible with GitHub apps, and the proxy solution suggested in the former post seems too much overhead.
Is there a way for xcodebuild's package resolution to successfully do authentication through HTTPS using the global git configuration?
We use HTTPS and SSH at my work interchangeably using both Xcode and xcodebuild (and vanilla SwiftPM for that matter) in our CI. In the end, I think it is fair to say that package manager is not the limitation here but likely your ability to supply the correct credentials while running in your CI.
How are you providing your credentials for https://github.com during your CI runs?
Looks like there is no alternative to using the Xcode UI today, so you would need to login once via that on the CI machine to store the credentials and then using xcodebuild -scmProvider xcode should be able to use those credentials you configured.
Hi!
Thanks for your answer. Adding a bit of more context, the way we authenticate on CI hosts is by using git's GIT_ASKPASS environment variable, and passing an executable that tells Git where to read the username and password from. That's working fine when we run Git in the system.
@NeoNacho do you know if we can automate the process of storing the credentials where Xcode expects them? Unfortunately, setting the token in our CI hosts through Xcode's UI is not viable because we rotate the credentials periodically. Thanks in advance!
Oh, if git on the system works fine, -scmProvider system is actually the right thing. This is the default in Xcode 12.5 and later, perhaps you are using a previous version?
Interesting, we are for instance using Xcode 12.5. Is it possible that the GIT_ASKPASS variable is not exposed to the Swift Package Manager process? We've verified that the variable works at the system level because we can clone private repositories. I don't know what else could be .
The way we do it in our CI is "this one weird trick". We (ab)use Git's url.<base>.insteadOf to re-write the clone URL to include the appropriate credentials in our pipeline and make use of xcodebuild's -usePackageSupportBuiltinSCM.
Our CI of choice is GitLab and it provides ephemeral credentials as environment variables to clone over HTTPS for each job. Specifically it provides, CI_JOB_TOKEN, CI_SERVER_PROTOCOL, and CI_SERVER_HOST.
Typically, our URLs in our Xcode and Package.swift files are of the SSH variety (e.g., git@my.git.server.org:group/project.git). So we use Git's url.<base>.insteadOf to re-write the URL to be HTTP(S) and to include the appropriate credentials.
Assuming that CI_SERVER_HOST=my.git.server.org, CI_SERVER_PROTOCOL=https, and CI_JOB_TOKEN=WF0IjoxNTE2MjM5MDIyfQ.
This translates any URL in either or Xcode or Package.swift that might look like git@my.git.server.org:group/project.git → https://gitlab-ci-token:WF0IjoxNTE2MjM5MDIyfQ@my.git.server.org/group/project.git.
Then using xcodebuild -usePackageSupportBuiltinSCM along with all of our other normal flags things build just fine.
That git config is cut-n-paste from our .gitlab-ci.yml file and we have found this to be a relatively painless and robust way of handling HTTPS Git authentication. YMMV.
Thanks @rlovelett, we were not passing the -usePackageSupportBuiltinSCM argument when running xcodebuild. I added the argument, but it still fails. This is the entire xcodebuild command that we run.
One thing to note, is that we don't authenticate using the GIT_ASKPASS variable as I mentioned, but we create a gitconfig file in the system, that has a credentials helper set that takes care of passing the username and password when Git needs them. The reason why we don't put a token there is because we rotate those for security reasons, and we need the Git process to get it from another process. Looking at the logs:
xcodebuild: error: Could not resolve package dependencies:
Failed to clone repository https://github.com/organization/repo:
Cloning into bare repository '/Users/anka/Library/Developer/Xcode/DerivedData/Shopify-eotrbxjhnrnkesdptcwwzebeoowa/SourcePackages/repositories/repo-0dd4e281'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
I wonder if the Git process fails because we disable the terminal prompts in the process run by Xcode? Shouldn't Git read the credential helper from the gitconfig file, and shell out to the helper instead of trying to read it from the standard input?
This is more or less what GitLab does too. Ours rotate for every job, hence the environment variable. If it were my build environment I'd switch to using environment variables. But this is not strictly necessary I suppose.
How? Are you sure the .gitconfig file you are creating the right.gitconfig file? We set our .gitconfig properties via git config --global rather than writing the file ourselves. Ensuring that git should always know the correct location.
It works well without additional setup for both xcodebuild (even without -scmProvider system) and swift build and does not create any records in the keychain (which might fail other jobs).
,
.