SPM support basic auth for non-git binary dependency hosts

@sstadelman thanks for the info! I am able to see the download request from SPM using Charles Proxy, I can even see the values from my .netrc file in the request but for some reason Github always replies with Status 404: Not Found

Btw, I am testing this using both Xcode 12.5 and 13.

Have you confirmed it works with curl?

@ksluder thanks for your reply!
It looks like its not working with curl either:

So then it sounds like the next step is following up with GitHub.

Looks like GitHub's doing its auth a little different nowadays with their new tokens, but I managed to get this working for a source code release asset download:

$ cat  ~/.netrc
machine github.com
	login {personal access token}
machine api.github.com
	login {personal access token}
$ curl --output source.zip --location --netrc https://github.com/{user}/{repository}/archive/refs/tags/{tag}.zip

Note that there's no instance of your actual username anywhere anymore, just the access token. I also didn't test what access token scopes I needed, I just gave this token the whole set of them.

@MPLewis I tried the API you mention but unfortunately for me it seems to be downloading a zip of the entire repository (seems to be a similar functionality to the GitHub API described for Download a repository archive (zip)).

I believe the GitHub API for pulling a release asset is described here.

The tricky extra part is the requirement for a specific Accept header in order to receive the binary asset; without the header you only receive a JSON description of the asset details. @ksluder I can confirm that when passing the necessary Accept: application/octet-stream header in the curl request that it works (with ~/.netrc) so everything seems fine on the GitHub front (according to their design at least).

I raised the issue with this extra header requirement here:

@sstadelman I'm not familiar with requirements for GitHub Enterprise; were you adding support for pulling binary assets from the GitHub releases feature? And did it follow the API guidelines set out in the repos docs for "Get a release asset"? I'm just lost on how to indicate to SPM to include the Accept: application/octet-stream header (or if there is a way around this).

If there is no current way around this, would the Swift team consider adding something like a customHeaders: [String: String] = [:] parameter to the Target.binaryTarget method? Or is there a better way to resolve this?

For reference, and to clarify @theRealRobG 's doubts, here is the curl call to download release assets (not release source code) via the GitHub REST API:

curl \
    --location \
    --output release_asset.zip \
    -H "Accept: application/octet-stream" \
    -H "Authorization: token MY_GITHUB_PAT" \
    "https://api.github.com/repos/{owner}/{repo}/releases/assets/{release_id}"

Where MY_GITHUB_PAT is a GitHub PAT with full repo permissions. You can find the full URL by copying the URL of the required release asset in the release's page on GitHub, and via the GitHub CLI:

$ gh release view THE_RELEASE_TAG --json apiUrl

The curl call can leverage a netrc file thus (as @MPLewis points out):

$ cat ~/.netrc 
machine api.github.com
    login MY_GITHUB_PAT

curl \
    --location \
    --netrc \
    --output release_asset.zip \
    -H "Accept: application/octet-stream" \
    "https://api.github.com/repos/{owner}/{repo}/releases/assets/{release_id}"

It looks like Xcode 13 fails to add Swift packages that specify a binaryTarget with the URL of a GitHub asset in a private repo (even with a correctly formatted ~/.netrc file with suitable credentials):

    targets: [
        .binaryTarget(
            name: "FrameworkName",
            url: "https://api.github.com/repos/{owner}/{repo}/releases/assets/{release_id}",
            checksum: "f7c06...."

I got in touch with GitHub support and they've confirmed that the GitHub API doesn't currently support this SPM functionality.

@NeoNacho How could we go about supporting this functionality in a future release of SPM?

I think this PR is relevant: Add accept header to download request by jimmya · Pull Request #3795 · apple/swift-package-manager · GitHub

@NeoNacho yes indeed, as that header would be required. However that's insufficient as it seems like HTTPClient.Request only accepts 200 as a valid response code and the GitHub REST API initially responds with a 302 when we ask it for a release asset (see my sample curl call above).

Would it be a matter of adding 302 to the list of valid response codes?

Ah sorry, I missed that. I think we should support various HTTP redirect methods in general, so also 301 etc. Only accepting 200 seems overly restrictive.

@tomerd wdyt?

the fix here could be different - make sure the underlying (foundation) http client follow redirects as this is not normally handled at the application level

Barring implementation of the proper delegate method, URLSession should follow redirects by default, making the response received by SPM the 200 from the redirected address, not the 302 that was initially received. As far as I can tell SPM doesn't implement the redirect delegate. However, this behavior may be different when using swift-corelibs-foundation rather than Apple's. Additionally, handling of auth challenges and credentials is certainly different on the open source version, as I still can't get Alamofire's test suite to pass, so that part of the functionality may just be broken. I don't know how SPM handles auth right now, but it doesn't implement the URLSession delegate to do so.

@tomerd @NeoNacho Any updates on this? I'd be happy to look into opening a PR if we agree on the approach... Is it a matter of adding 302 to the list of valid response codes or should the http client be revised?

PR is absolutely welcome, thank you for offering! as @Jon_Shier points out above this is less about adding 302 to the accepted response codes, and more about making sure SwiftPM use of URLSession is correct - in the sense of configuring URLSession to follow redirects.

Sorry, my point was, looking at the SPM code, SPM does not implement the delegate that would prevent automatic redirect handling, so the 302 -> 200 redirect should be working now, unless there's something I'm missing in SPM's URLSession implementation (it's surprisingly simple). However, I can't rule out issues in the open source version of URLSession or Apple's private fork of SPM.

@tomerd: looking at SPM's code and as @Jon_Shier points out, it seems like SPM's use of URLSession is correct insofar as it should follow redirects. It's my understanding that that's URLSession's default behaviour, is that being overridden somewhere?

not that I know, SwiftPM's wrapper of URLSession is pretty straight forward

However, I can't rule out issues in the open source version of URLSession or Apple's private fork of SPM.

@tallariel is this happening on macOS or Linux?

@tallariel did we confirm that this does not solve the issue you are seeing? or iow are we sure its a redirect problem?

@tomerd This is happening on Xcode 13. The issue is that it fails to add Swift packages that specify a binaryTarget with the URL of a GitHub asset in a private repo (even with a correctly formatted ~/.netrc file with suitable credentials). I suspect this could be because

A) the ensuing redirect isn't followed
B) Xcode doesn't read the netrc file