SPM support basic auth for non-git binary dependency hosts

Using Swift
#81

We already have this functionality by specifying a binary target and a relative path to an xcframework:

targets: [
        .binaryTarget(
            name: "Foo",
            path: "foo.xcframework"
        )
    ]

So I don't see the added benefit of specifying a https://github.com/<user>/<repo>/archive/refs/tags/<tag>.zip url vs. just a git url and a tag.

This is already working in Xcode 13+ for both private and public repos. How would this address the original issue – downloading release assets from private github repos?

#82

This uses the GitHub CLI. First, cd into your local repo, then:

echo "my release asset" > release_asset.txt
zip my_archive.zip release_asset.txt
gh release create 9.9.9 --title "9.9.9" --notes ""
gh release upload 9.9.9 my_archive.zip

This creates a release at https://github.com/<user>/<repo>/releases/tag/9.9.9 (Notice that the release asset doesn't need to be checked into the repo). To then view the URL of release asset my_archive.zip:

gh release view 9.9.9 --json assets

If you then download my_archive.zip you'll see that the first level component is release_asset.txt, so this doesn't seem like a "strip first level component" issue.

#83

I do not have any git CLI instructions for this as it is not really a git process. I create a release as demonstrated in these documents: Managing releases in a repository - GitHub Docs

Step 8 specifically is how I attach our binary builds to the release. Currently, I am building most of the binaries manually, but working on GitHub actions that will build and release them automatically.

If you do not publish the release but have it setup as a "draft release" (e.g. you have not published it yet, but have filled out all the info and saved it), you can get the URLs to the binaries via the GitHub cli app. I do that, update my Package.swift, push the Package.swift changes, then publish the Release on Github.

#84

@tallariel @afarnham I think I got this to also work using the "release upload" technique you are using. Here are the steps I took:

  1. create a personal access token and add it to ~/.netrc. note we are going to use api.github.com based URLs so need an entry for that host.
  2. create a private repo
  3. create a release using the upload technique with gh cli tool provided by @tallariel (which sounds like is the same thing @afarnham reported doing from the UI):
  $ zip test.zip test.xcframework
  $ gh release create 1.0.0 --title "1.0.0" --notes ""
  $ gh release upload 1.0.0 test.zip
  1. extract the asset url
$ gh release view 1.0.0 --json assets
{
  "assets": [
    {
      "apiUrl": "https://api.github.com/repos/tomerd/test-binary-dep-2/releases/assets/54425621",
      "contentType": "application/zip",
      "createdAt": "2022-01-20T01:30:04Z",
      "downloadCount": 7,
      "id": "RA_kwDOGtEVq84DPngV",
      "label": "",
      "name": "test.zip",
      "size": 184,
      "state": "uploaded",
      "updatedAt": "2022-01-20T01:30:04Z",
      "url": "https://github.com/tomerd/test-binary-dep-2/releases/download/1.0.0/test.zip"
    }
  ]
}

the tricky part is that the "apiUrl" is the one that works, and we need tack a "zip" extension at the end to make SwiftPM happy (at least for now), so the manifest for this example would look like:

.binaryTarget(
    name: "test",
    url: "https://api.github.com/repos/tomerd/test-binary-dep-2/releases/assets/54425621.zip",
    checksum: "..."
)
#85

@tomerd I followed your steps and I get the following error message in Xcode 13.2: failed downloading 'https://api.github.com/repos/<owner>/<repo>/releases/assets/<assetid>.zip' which is required by binary target...

we could rule out this is an auth issue as I can download the asset via curl:

curl \
    --location \
    --netrc \
    --output release_asset.zip \
    -H "Accept: application/octet-stream" \
    "https://api.github.com/repos/<owner>/<repo>/releases/assets/<assetid>.zip"
#86

@tomerd @tallariel I now retried again as well following the steps provided by @tomerd. I made sure my .netrc config is valid (by trying the raw curl command and that works).

For computing the checksum I followed Apple Developer Documentation.

At first I got:

checksum of downloaded artifact of binary target 'foobar' (bb035bxxx) does not match checksum specified by the manifest (68dddxxx)

Does that mean it really downloaded something? I can't find anything in derived data at least.
Also, I'm pretty sure the checksum is correct, though.

Still I went ahead and changed the checksum to the one that Xcode told me it computed.
Then I'm getting this error:

failed extracting 'https://api.github.com/repos/foo/bar/releases/assets/54280443.zip' which is required by binary target 'foobar': End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive.

unzip: cannot find zipfile directory in one of /Users/mathaeus/Library/Developer/Xcode/DerivedData/xxx/SourcePackages/artifacts/foobar/54280443.zip or

/Users/mathaeus/Library/Developer/Xcode/DerivedData/xxx/SourcePackages/artifacts/foobar/54280443.zip.zip, and cannot find /Users/mathaeus/Library/Developer/Xcode/DerivedData/xxx/SourcePackages/artifacts/foobar/54280443.zip.ZIP, period.

Which makes me think it actually didn't download anything, but Xcode believes it did.

p.s. I'm on Xcode 13.2.1. @tallariel since you wrote you are using 13.2

#87

I've seen the same issues as @mathaeus with Xcode 13.2.1 when following your steps @tomerd. Like @tallariel, once I add the octet-stream header it works.

Instead of testing via curl, I pointed the url in my Package.swift to my proxy which forwards the request to api.github.com and adds the application/octet-stream Accept header. This works for me.

The shape of the proxy url is https://proxy.mydomain.com/repos/<owner>/<repo>/releases/assets/<assetid>.zip. Same shape as the GitHub url, but different host.

#88

sorry, to be clear this requires the application/octet-stream accept header change which is not in any released version of Xcode yet, so this should work using the two techniques I described above on the next version of Xcode. I suggest we re-group here once the betas are out. Or you can test this with SwiftPM directly using the nightly 5.6 toolchain (outside Xcode)

2 Likes
#89

Great! I really appreciate all the time you took here to understand the issue and work with us. Thank you so much!

#90

@afarnham @tallariel either of you able to make it work with Xcode 13.3 BETA? Because I'm not :pensive:. I tried with two different repositories and verified that the respective release assets can be downloaded with my ~/.netrc configuration.

@tomerd from what I understood Xcode 13.3 BETA should support downloading private github release assets, because it adds the application/octet-stream header, correct? I tried both, via the Xcode UI and the command line, no luck. Not sure if I'm still missing something, any help appreciated.

#91

hi @mathaeus unfortunately this did not make it into the first beta. will update here when available.

2 Likes
#92

This should be included in beta 3. Also note that we support the keychain which is a much safer alternative to .netrc

2 Likes
#93

Xcode 13.3 BETA 3 seems to do the job for me. Thanks a lot @tomerd

Also note that we support the keychain

:+1:

1 Like
#94

I independently confirm that with Xcode 13.3 Beta 3 it is possible to download an asset (some.xcframework.zip) of a GitHub release in a private GitHub repository.

As outlined in this thread the asset url needs to be used and the .netrc file needs to contain an entry for machine api.github.com, GitHub user name for login and a personal access token scoped to access private repos as password.

I wrote a blog post as summary as this feature is not mentioned in the Xcode 13.3 Beta 3 release notes.

5 Likes
#95

Nice summary in the blog article @MarcoEidinger :+1:

2 Likes
#96

@MarcoEidinger’s blog post gives great instructions for using .netrc, but how would I use the keychain for this @tomerd?

#97

@spacecrafter3d I was able to to achieve the same download by creating an Internet password in the keychain. To create such use "New Password Item" (⌘N) and specify the the following attributes

Keychain Item Name https://api.github.com
Account Name GitHub user name
Password personal access token

I updated my blog post with the information and also added screenshots for better understanding.

3 Likes
#98

That’s perfect, thanks!

#99

I was able to get this working with the keychain by following @MarcoEidinger's instructions. One issue that tripped me up for a long time was that apparently you have to quit Keychain Access before the new entry is available to Xcode.

Is there any chance this could work using Xcode's built in account system (for those using Xcode) instead of adding the password to Keychain?

#100

Note not to use --json apiUrl as mentioned in one of the posts above, this would give you the release url, not the asset url. You want to use --json assets as mentioned in other posts.

For reference, the error I got was: failed downloading 'https://api.github.com/repos/<org>/<repo>/releases/<release-id>.zip' which is required by binary target 'XX': badResponseStatusCode(415)

1 Like