SPM support basic auth for non-git binary dependency hosts

@tomerd @NeoNacho Any updates on this? I'd be happy to look into opening a PR if we agree on the approach...

@tallariel @NeoNacho @afarnham based on the thread it sounds like the remaining issue is that SwiftPM requires the artifact (asset) URL to end in .zip. If so, a PR that lifts this restriction but validates the downloaded file is a zip file (perhaps via the content-type header?) could do the trick?

@tomerd @NeoNacho To write and debug my implementation I was hoping to use the testBinaryTargetsValidation unit test, but this test currently fails on main (Every single one of its assertions fails). That unit test code needs to be updated and/or the actual manifest loading functionality isn't stable right now. Either way, I can't run or test basic manifest loading functionality, so shall I wait before continuing work on a PR?

I tried checking working on top of release/5.4, but that has diverged significantly from main.

@tallariel yes main is the branch we work on day to day and all the tests pass there. sounds like something is going on your machine / setup that causes the tests to fail. can you share the failure information? also make sure to follow the setup instructions on the contribution guide if you have done so already.

@tomerd

  • checked out revision 6e3c2ba1f363601116de0a05a65f9ed96994b3e7
  • opened it in Version 13.2 beta 2 (13C5081f)
  • selected the swift-package scheme
  • Product --> Test

The tests hit a runtime error on PD_4_2_LoadingTests.swift, line 597

Thread 1: Fatal error: 'try!' expression unexpectedly raised an error: PackageLoading.ManifestParseError.invalidManifestFormat("error: link command failed with exit code 1 (use -v to see invocation)\nld: library not found for -lPackageDescription", diagnosticFile: nil)

If I run just testBinaryTargetsValidation, which is the one I'm interested in, I get this result:

If I run swift test --filter testBinaryTargetsValidation the tests pass. But I'd like to run them from Xcode in order to facilitate debugging. Any thoughts?

It seems like the package manifest dylib is not getting built. Does it work better with SwiftPM-Package scheme? that should build the entire project when running test, including those dylibs

Yes, thanks. I've sent in a small PR to clarify the documentation accordingly. I'll carry on with the rest.

1 Like

@tallariel @afarnham not sure if this helps, but I just did a small test with a private GitHub.com repo hosting a binary artifact and got it to work with a small change to SwiftPM. here is the test setup:

  1. created a private repo on GitHub.com to host the binary artifact, i.e it has a single XCFramework file in it. in my case named test.xcframework.
  2. tagged the above private repo as 1.0.0
  3. used the GitHub.com release url for the binary artifact, e.g.:
.binaryTarget(
    name: "test",
    url: "https://github.com/tomerd/test-binary-dep/archive/refs/tags/1.0.0.zip",
    checksum: "..."
)

the missing part for this setup to work is that SwiftPM expects the downloaded zip to contain the framework directly while GitHub.com release zip files have a top level directory. e.g.

GitHub provides

1.0.0.zip
|- test-binary-dep-1.0.0
    |- test.xcframework

SwiftPM expects:

1.0.0.zip
|- test.xcframework

this is a classic "strip first level component" archive issue, and with a small patch to SwiftPM we can support both modes: [wip] support binary artifact archives that require first level component stripping by tomerd · Pull Request #4020 · apple/swift-package-manager · GitHub

does this help?

Depending on the macOS version used, the auth mechanism being used by the SPM http client may be flaky, because from what I can see, it doesn't use the URLSession authorization mechanism correctly: [SR-15731] swift-package-manager HTTPClient misuses URLSession reserved Authorization header - Swift

While technically true according to the documentation, that's not really true in practice. In common usage URLSession only cares about the Authorization header when the server is using one of the HTTP auth mechanisms that expects an authorization response, like HTTP Basic. In that case URLSession will try to use a provided URLCredential to create the Authorization header. If that's not the case, providing your own Authorization header works fine. It has to, otherwise various authorization mechanisms wouldn't work at all. Depending on the server, providing an Authorization header immediately may preempt any server side challenge and should work just fine. It's only when the server insists on issuing the challenge that you may see URLSession do something to the header.

1 Like

@tomerd It seems GitHub has two different behaviors depending on who owns the repository. Scenario 1, which you tested, is personal ownership. Scenario 2 is organization ownership.

Scenario 1 worked fine for me just now with Xcode 13.2.1 when I tried it with both my personal account with a private repository and my work account with a private repository that is not tied to my company.

However, in Scenario 2, when I try it with a repository owned by my company (organization in GitHub parlance), it fails. Instead I have to get the API URL using the GitHub command line app:

gh release view 1.0.0 --json assets | jq '.assets[]'

In the JSON returned I take the apiUrl value, append a .zip to the end and change the host to a proxy I setup. Then I use this in my Package.swift binary target.

When Swift PM resolves packages, it hits that proxy which then rewrites the request to add a Accept: application/octet-stream header and strips the .zip and then forwards the request to GitHub.

This proxied setup does work with Swift Package Manager today, but it would be nice not to have to run it.

I am using a local .netrc for authentication for the record.

@afarnham surprised it worked for you without the patch linked to above. can you share the url you used for your tests (just the shape of the url is enough if the url itself is sensitive), note in my test i’ve used the github’s release zip file url, not a github api based url

https://github.com/<username>/<repo name>/releases/download/1.0.0/myframework.xcframework.zip

That link is from a GitHub release for the given tag and when creating that release I drag the binary from desktop to the Release creation form.

@afarnham the steps I took and should work (with the patch above) for both users and orgs:

  1. create a private repo
  2. commit the xcframework at the root of the repo
  3. tag the repo
  4. use a https://github.com/<user>/<repo>/archive/refs/tags/<tag>.zip url

could that work for you? looking for feedback before going forward with that patch.

@tomerd I'm trying to replicate the steps you mention, I've a couple of questions..

A) Where can I learn more about running SPM from source? Couldn't find anything to that effect in the Documentation directory... I'm trying to run your patch so as to pass it a Package.swift file without going through the tests but can't figure out how..
B) SPM can already deal with xcframeworks placed at the root of the repo:

targets: [
        .binaryTarget(
            name: "Foo",
            path: "foo.xcframework"
        )
    ]

How does this relate to SPM downloading GH release assets (which may not be checked into the repo)?

I don't think checking these into the root of a repo would work for us. GitHub charges based on bandwidth used in git LFS. The only way we can check some of these binaries into a repo is with LFS since GitHub has file size limits for non-LFS artifacts. This would explode our bandwidth costs with our CI pulling these things all the time.

Also, as far as I know, SwiftPM does not support Git LFS.

1 Like

+1 for not being able to check in the xcframework into the repo because of filesize and git lfs.


I'm running into the same error that was reported in SPM support basic auth for non-git binary dependency hosts - #59 by tallariel

failed downloading 'https://api.github.com/repos/xxx/xxx/releases/assets/xxx.zip' which is required by binary target 'xxx': badResponseStatusCode(404)


I have read through most of the thread, but I'm still unsure: Will SPM support downloading private Github release assets with the next Xcode/Swift version? @tomerd

it basically boils down to cloning the SwiftPM repo locally, building it and then running the relevant executable

documented in swift-package-manager/CONTRIBUTING.md at main · apple/swift-package-manager · GitHub

feel free to DM if need more guidance on that

SwiftPM supports two sources for binary artifacts:

  1. URL based: the artifact is delivered archived as a zip file from a web-server. private GitHub repo is one option to host such archives which has been the focus of this thread, but it can be behind any we-service, with or without auth.
  2. Path based: the artifact is checked in next to the consuming package sources (which is your example above). In this case the artifact can be the xcframework directly or a zip archive containing the xcframework. the path must be relative to the project root.
1 Like

@afarnham can you share instructions on you set up you binary artifact private repo? you said something about "I drag the binary from desktop to the Release creation form." is there a set of git based CLI instructions you can provide?

@mathaeus once we get a confirmation that the proposed solution in SPM support basic auth for non-git binary dependency hosts - #74 by tomerd is useful for folks, we can merge the after-mentioned PR which will also be included in the next schedule release of Xcode.

1 Like
Terms of Service

Privacy Policy

Cookie Policy