SPM support basic auth for non-git binary dependency hosts

Depending on the macOS version used, the auth mechanism being used by the SPM http client may be flaky, because from what I can see, it doesn't use the URLSession authorization mechanism correctly: [SR-15731] swift-package-manager HTTPClient misuses URLSession reserved Authorization header - Swift

While technically true according to the documentation, that's not really true in practice. In common usage URLSession only cares about the Authorization header when the server is using one of the HTTP auth mechanisms that expects an authorization response, like HTTP Basic. In that case URLSession will try to use a provided URLCredential to create the Authorization header. If that's not the case, providing your own Authorization header works fine. It has to, otherwise various authorization mechanisms wouldn't work at all. Depending on the server, providing an Authorization header immediately may preempt any server side challenge and should work just fine. It's only when the server insists on issuing the challenge that you may see URLSession do something to the header.

1 Like

@tomerd It seems GitHub has two different behaviors depending on who owns the repository. Scenario 1, which you tested, is personal ownership. Scenario 2 is organization ownership.

Scenario 1 worked fine for me just now with Xcode 13.2.1 when I tried it with both my personal account with a private repository and my work account with a private repository that is not tied to my company.

However, in Scenario 2, when I try it with a repository owned by my company (organization in GitHub parlance), it fails. Instead I have to get the API URL using the GitHub command line app:

gh release view 1.0.0 --json assets | jq '.assets[]'

In the JSON returned I take the apiUrl value, append a .zip to the end and change the host to a proxy I setup. Then I use this in my Package.swift binary target.

When Swift PM resolves packages, it hits that proxy which then rewrites the request to add a Accept: application/octet-stream header and strips the .zip and then forwards the request to GitHub.

This proxied setup does work with Swift Package Manager today, but it would be nice not to have to run it.

I am using a local .netrc for authentication for the record.

@afarnham surprised it worked for you without the patch linked to above. can you share the url you used for your tests (just the shape of the url is enough if the url itself is sensitive), note in my test i’ve used the github’s release zip file url, not a github api based url

https://github.com/<username>/<repo name>/releases/download/1.0.0/myframework.xcframework.zip

That link is from a GitHub release for the given tag and when creating that release I drag the binary from desktop to the Release creation form.

@afarnham the steps I took and should work (with the patch above) for both users and orgs:

  1. create a private repo
  2. commit the xcframework at the root of the repo
  3. tag the repo
  4. use a https://github.com/<user>/<repo>/archive/refs/tags/<tag>.zip url

could that work for you? looking for feedback before going forward with that patch.

@tomerd I'm trying to replicate the steps you mention, I've a couple of questions..

A) Where can I learn more about running SPM from source? Couldn't find anything to that effect in the Documentation directory... I'm trying to run your patch so as to pass it a Package.swift file without going through the tests but can't figure out how..
B) SPM can already deal with xcframeworks placed at the root of the repo:

targets: [
            name: "Foo",
            path: "foo.xcframework"

How does this relate to SPM downloading GH release assets (which may not be checked into the repo)?

I don't think checking these into the root of a repo would work for us. GitHub charges based on bandwidth used in git LFS. The only way we can check some of these binaries into a repo is with LFS since GitHub has file size limits for non-LFS artifacts. This would explode our bandwidth costs with our CI pulling these things all the time.

Also, as far as I know, SwiftPM does not support Git LFS.

1 Like

+1 for not being able to check in the xcframework into the repo because of filesize and git lfs.

I'm running into the same error that was reported in SPM support basic auth for non-git binary dependency hosts - #59 by tallariel

failed downloading 'https://api.github.com/repos/xxx/xxx/releases/assets/xxx.zip' which is required by binary target 'xxx': badResponseStatusCode(404)

I have read through most of the thread, but I'm still unsure: Will SPM support downloading private Github release assets with the next Xcode/Swift version? @tomerd

it basically boils down to cloning the SwiftPM repo locally, building it and then running the relevant executable

documented in swift-package-manager/CONTRIBUTING.md at main · apple/swift-package-manager · GitHub

feel free to DM if need more guidance on that

SwiftPM supports two sources for binary artifacts:

  1. URL based: the artifact is delivered archived as a zip file from a web-server. private GitHub repo is one option to host such archives which has been the focus of this thread, but it can be behind any we-service, with or without auth.
  2. Path based: the artifact is checked in next to the consuming package sources (which is your example above). In this case the artifact can be the xcframework directly or a zip archive containing the xcframework. the path must be relative to the project root.
1 Like

@afarnham can you share instructions on you set up you binary artifact private repo? you said something about "I drag the binary from desktop to the Release creation form." is there a set of git based CLI instructions you can provide?

@mathaeus once we get a confirmation that the proposed solution in SPM support basic auth for non-git binary dependency hosts - #74 by tomerd is useful for folks, we can merge the after-mentioned PR which will also be included in the next schedule release of Xcode.

1 Like

We already have this functionality by specifying a binary target and a relative path to an xcframework:

targets: [
            name: "Foo",
            path: "foo.xcframework"

So I don't see the added benefit of specifying a https://github.com/<user>/<repo>/archive/refs/tags/<tag>.zip url vs. just a git url and a tag.

This is already working in Xcode 13+ for both private and public repos. How would this address the original issue – downloading release assets from private github repos?

This uses the GitHub CLI. First, cd into your local repo, then:

echo "my release asset" > release_asset.txt
zip my_archive.zip release_asset.txt
gh release create 9.9.9 --title "9.9.9" --notes ""
gh release upload 9.9.9 my_archive.zip

This creates a release at https://github.com/<user>/<repo>/releases/tag/9.9.9 (Notice that the release asset doesn't need to be checked into the repo). To then view the URL of release asset my_archive.zip:

gh release view 9.9.9 --json assets

If you then download my_archive.zip you'll see that the first level component is release_asset.txt, so this doesn't seem like a "strip first level component" issue.

I do not have any git CLI instructions for this as it is not really a git process. I create a release as demonstrated in these documents: Managing releases in a repository - GitHub Docs

Step 8 specifically is how I attach our binary builds to the release. Currently, I am building most of the binaries manually, but working on GitHub actions that will build and release them automatically.

If you do not publish the release but have it setup as a "draft release" (e.g. you have not published it yet, but have filled out all the info and saved it), you can get the URLs to the binaries via the GitHub cli app. I do that, update my Package.swift, push the Package.swift changes, then publish the Release on Github.

@tallariel @afarnham I think I got this to also work using the "release upload" technique you are using. Here are the steps I took:

  1. create a personal access token and add it to ~/.netrc. note we are going to use api.github.com based URLs so need an entry for that host.
  2. create a private repo
  3. create a release using the upload technique with gh cli tool provided by @tallariel (which sounds like is the same thing @afarnham reported doing from the UI):
  $ zip test.zip test.xcframework
  $ gh release create 1.0.0 --title "1.0.0" --notes ""
  $ gh release upload 1.0.0 test.zip
  1. extract the asset url
$ gh release view 1.0.0 --json assets
  "assets": [
      "apiUrl": "https://api.github.com/repos/tomerd/test-binary-dep-2/releases/assets/54425621",
      "contentType": "application/zip",
      "createdAt": "2022-01-20T01:30:04Z",
      "downloadCount": 7,
      "id": "RA_kwDOGtEVq84DPngV",
      "label": "",
      "name": "test.zip",
      "size": 184,
      "state": "uploaded",
      "updatedAt": "2022-01-20T01:30:04Z",
      "url": "https://github.com/tomerd/test-binary-dep-2/releases/download/1.0.0/test.zip"

the tricky part is that the "apiUrl" is the one that works, and we need tack a "zip" extension at the end to make SwiftPM happy (at least for now), so the manifest for this example would look like:

    name: "test",
    url: "https://api.github.com/repos/tomerd/test-binary-dep-2/releases/assets/54425621.zip",
    checksum: "..."

@tomerd I followed your steps and I get the following error message in Xcode 13.2: failed downloading 'https://api.github.com/repos/<owner>/<repo>/releases/assets/<assetid>.zip' which is required by binary target...

we could rule out this is an auth issue as I can download the asset via curl:

curl \
    --location \
    --netrc \
    --output release_asset.zip \
    -H "Accept: application/octet-stream" \

@tomerd @tallariel I now retried again as well following the steps provided by @tomerd. I made sure my .netrc config is valid (by trying the raw curl command and that works).

For computing the checksum I followed Apple Developer Documentation.

At first I got:

checksum of downloaded artifact of binary target 'foobar' (bb035bxxx) does not match checksum specified by the manifest (68dddxxx)

Does that mean it really downloaded something? I can't find anything in derived data at least.
Also, I'm pretty sure the checksum is correct, though.

Still I went ahead and changed the checksum to the one that Xcode told me it computed.
Then I'm getting this error:

failed extracting 'https://api.github.com/repos/foo/bar/releases/assets/54280443.zip' which is required by binary target 'foobar': End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive.

unzip: cannot find zipfile directory in one of /Users/mathaeus/Library/Developer/Xcode/DerivedData/xxx/SourcePackages/artifacts/foobar/54280443.zip or

/Users/mathaeus/Library/Developer/Xcode/DerivedData/xxx/SourcePackages/artifacts/foobar/54280443.zip.zip, and cannot find /Users/mathaeus/Library/Developer/Xcode/DerivedData/xxx/SourcePackages/artifacts/foobar/54280443.zip.ZIP, period.

Which makes me think it actually didn't download anything, but Xcode believes it did.

p.s. I'm on Xcode 13.2.1. @tallariel since you wrote you are using 13.2

I've seen the same issues as @mathaeus with Xcode 13.2.1 when following your steps @tomerd. Like @tallariel, once I add the octet-stream header it works.

Instead of testing via curl, I pointed the url in my Package.swift to my proxy which forwards the request to api.github.com and adds the application/octet-stream Accept header. This works for me.

The shape of the proxy url is https://proxy.mydomain.com/repos/<owner>/<repo>/releases/assets/<assetid>.zip. Same shape as the GitHub url, but different host.

sorry, to be clear this requires the application/octet-stream accept header change which is not in any released version of Xcode yet, so this should work using the two techniques I described above on the next version of Xcode. I suggest we re-group here once the betas are out. Or you can test this with SwiftPM directly using the nightly 5.6 toolchain (outside Xcode)