SPM support basic auth for non-git binary dependency hosts

eduardoperez-nbcuni · 2021-10-21T19:29:25.527Z

@sstadelman thanks for the info! I am able to see the download request from SPM using Charles Proxy, I can even see the values from my .netrc file in the request but for some reason Github always replies with Status 404: Not Found

Screen Shot 2021-10-21 at 3.21.53 PM1580×269 36.7 KB

Btw, I am testing this using both Xcode 12.5 and 13.

ksluder · 2021-10-21T19:59:04.090Z

Have you confirmed it works with curl?

eduardoperez-nbcuni · 2021-10-21T20:24:37.819Z

@ksluder thanks for your reply!
It looks like its not working with curl either:

SPM binary dependency in private GitHub release Using Swift

Unfortunately when trying that with a private repository it doesn't seem that you are able to authenticate the request using basic auth. I've attempted this with a working .netrc in my home directory using the following curl: curl -OLn "<<Private GitHub Release Page Asset Download URL>>" But I always get "Not Found".

ksluder · 2021-10-21T20:41:51.721Z

So then it sounds like the next step is following up with GitHub.

MPLewis · 2021-10-22T03:10:31.985Z

Looks like GitHub's doing its auth a little different nowadays with their new tokens, but I managed to get this working for a source code release asset download:

$ cat  ~/.netrc
machine github.com
	login {personal access token}
machine api.github.com
	login {personal access token}
$ curl --output source.zip --location --netrc https://github.com/{user}/{repository}/archive/refs/tags/{tag}.zip

Note that there's no instance of your actual username anywhere anymore, just the access token. I also didn't test what access token scopes I needed, I just gave this token the whole set of them.

theRealRobG · 2021-10-24T23:16:06.199Z

@MPLewis I tried the API you mention but unfortunately for me it seems to be downloading a zip of the entire repository (seems to be a similar functionality to the GitHub API described for Download a repository archive (zip)).

I believe the GitHub API for pulling a release asset is described here.

The tricky extra part is the requirement for a specific Accept header in order to receive the binary asset; without the header you only receive a JSON description of the asset details. @ksluder I can confirm that when passing the necessary Accept: application/octet-stream header in the curl request that it works (with ~/.netrc) so everything seems fine on the GitHub front (according to their design at least).

I raised the issue with this extra header requirement here:

SPM binary dependency in private GitHub release Using Swift

I have a project that produces an XCFramework which I zip and upload to the GitHub releases page of the private repository. I've added my personal access token to ~/.netrc and verified it working correctly against api.github.com. Following the GitHub REST reference I am able to find the relevant asset_id and declare a binary dependency as such: .binaryTarget( name: privateName, url: "https://api.github.com/repos/\(owner)/\(repo)/releases/assets/\(assetId).zip", // must have .zip extensi…

@sstadelman I'm not familiar with requirements for GitHub Enterprise; were you adding support for pulling binary assets from the GitHub releases feature? And did it follow the API guidelines set out in the repos docs for "Get a release asset"? I'm just lost on how to indicate to SPM to include the Accept: application/octet-stream header (or if there is a way around this).

If there is no current way around this, would the Swift team consider adding something like a customHeaders: [String: String] = [:] parameter to the Target.binaryTarget method? Or is there a better way to resolve this?

tallariel · 2021-10-26T14:07:45.298Z

For reference, and to clarify @theRealRobG 's doubts, here is the curl call to download release assets (not release source code) via the GitHub REST API:

curl \
    --location \
    --output release_asset.zip \
    -H "Accept: application/octet-stream" \
    -H "Authorization: token MY_GITHUB_PAT" \
    "https://api.github.com/repos/{owner}/{repo}/releases/assets/{release_id}"

Where MY_GITHUB_PAT is a GitHub PAT with full repo permissions. You can find the full URL by copying the URL of the required release asset in the release's page on GitHub, and via the GitHub CLI:

$ gh release view THE_RELEASE_TAG --json apiUrl

The curl call can leverage a netrc file thus (as @MPLewis points out):

$ cat ~/.netrc 
machine api.github.com
    login MY_GITHUB_PAT

curl \
    --location \
    --netrc \
    --output release_asset.zip \
    -H "Accept: application/octet-stream" \
    "https://api.github.com/repos/{owner}/{repo}/releases/assets/{release_id}"

It looks like Xcode 13 fails to add Swift packages that specify a binaryTarget with the URL of a GitHub asset in a private repo (even with a correctly formatted ~/.netrc file with suitable credentials):

    targets: [
        .binaryTarget(
            name: "FrameworkName",
            url: "https://api.github.com/repos/{owner}/{repo}/releases/assets/{release_id}",
            checksum: "f7c06...."

I got in touch with GitHub support and they've confirmed that the GitHub API doesn't currently support this SPM functionality.

@NeoNacho How could we go about supporting this functionality in a future release of SPM?

NeoNacho · 2021-10-26T22:08:56.376Z

I think this PR is relevant: Add accept header to download request by jimmya · Pull Request #3795 · apple/swift-package-manager · GitHub

tallariel · 2021-10-27T09:01:42.757Z

@NeoNacho yes indeed, as that header would be required. However that's insufficient as it seems like HTTPClient.Request only accepts 200 as a valid response code and the GitHub REST API initially responds with a 302 when we ask it for a release asset (see my sample curl call above).

Would it be a matter of adding 302 to the list of valid response codes?

NeoNacho · 2021-10-27T17:52:25.943Z

Ah sorry, I missed that. I think we should support various HTTP redirect methods in general, so also 301 etc. Only accepting 200 seems overly restrictive.

@tomerd wdyt?

tomerd · 2021-10-29T01:25:05.966Z

the fix here could be different - make sure the underlying (foundation) http client follow redirects as this is not normally handled at the application level

Jon_Shier · 2021-10-29T03:01:19.105Z

Barring implementation of the proper delegate method, URLSession should follow redirects by default, making the response received by SPM the 200 from the redirected address, not the 302 that was initially received. As far as I can tell SPM doesn't implement the redirect delegate. However, this behavior may be different when using swift-corelibs-foundation rather than Apple's. Additionally, handling of auth challenges and credentials is certainly different on the open source version, as I still can't get Alamofire's test suite to pass, so that part of the functionality may just be broken. I don't know how SPM handles auth right now, but it doesn't implement the URLSession delegate to do so.

tallariel · 2021-11-08T11:34:46.010Z

@tomerd @NeoNacho Any updates on this? I'd be happy to look into opening a PR if we agree on the approach... Is it a matter of adding 302 to the list of valid response codes or should the http client be revised?

tomerd · 2021-11-09T04:03:08.259Z

tallariel:

@tomerd @NeoNacho Any updates on this? I'd be happy to look into opening a PR if we agree on the approach... Is it a matter of adding 302 to the list of valid response codes or should the http client be revised?

PR is absolutely welcome, thank you for offering! as @Jon_Shier points out above this is less about adding 302 to the accepted response codes, and more about making sure SwiftPM use of URLSession is correct - in the sense of configuring URLSession to follow redirects.

Jon_Shier · 2021-11-09T04:05:40.590Z

Sorry, my point was, looking at the SPM code, SPM does not implement the delegate that would prevent automatic redirect handling, so the 302 -> 200 redirect should be working now, unless there's something I'm missing in SPM's URLSession implementation (it's surprisingly simple). However, I can't rule out issues in the open source version of URLSession or Apple's private fork of SPM.

tallariel · 2021-11-10T16:29:52.286Z

@tomerd: looking at SPM's code and as @Jon_Shier points out, it seems like SPM's use of URLSession is correct insofar as it should follow redirects. It's my understanding that that's URLSession's default behaviour, is that being overridden somewhere?

tomerd · 2021-11-11T01:20:35.978Z

tallariel:

@tomerd: looking at SPM's code and as @Jon_Shier points out, it seems like SPM's use of URLSession is correct insofar as it should follow redirects. It's my understanding that that's URLSession's default behaviour, is that being overridden somewhere?

not that I know, SwiftPM's wrapper of URLSession is pretty straight forward

tomerd · 2021-11-11T01:21:46.657Z

However, I can't rule out issues in the open source version of URLSession or Apple's private fork of SPM.

@tallariel is this happening on macOS or Linux?

tomerd · 2021-11-11T01:24:34.585Z

NeoNacho:

I think this PR is relevant: Add accept header to download request by jimmya · Pull Request #3795 · apple/swift-package-manager · GitHub

@tallariel did we confirm that this does not solve the issue you are seeing? or iow are we sure its a redirect problem?

tallariel · 2021-11-11T14:51:15.172Z

@tomerd This is happening on Xcode 13. The issue is that it fails to add Swift packages that specify a binaryTarget with the URL of a GitHub asset in a private repo (even with a correctly formatted ~/.netrc file with suitable credentials). I suspect this could be because

A) the ensuing redirect isn't followed
B) Xcode doesn't read the netrc file