SPM support basic auth for non-git binary dependency hosts

@tallariel I think we should come back and test this when a beta of the next Xcode is out

:+1:

Even if the redirects are followed in the next Xcode beta, there is also this problem due to the GitHub asset URL not ending in .zip:
unsupported extension for binary target 'TestBinaryTarget'; valid extensions are: 'zip'

I got the above error when I tested this with Xcode 13.2 beta 2, which I assume is what you meant by a beta of the next Xcode.

I've found that GitHub API will disregard a file extension if you add it, so you can just append .zip to the URL declaration to appease SPM (though perhaps the need to do this should also be revised).

1 Like

I just tested on Xcode 13.2 beta and wasn't successful.

To recap: I created a release on my private github repo, added the zipped xcframework as a release asset, and updated the Package.swift file thus:

    targets: [
        .binaryTarget(
            name: "FrameworkName",
            url: "https://api.github.com/repos/{owner}/{repo}/releases/assets/{release_id}",
            checksum: "f7c06...."

When resolving packages, Xcode sends out the following error:

unsupported extension for binary target 'ClarityMiddleware'; valid extensions are: 'zip' in git@github.com:shapedbyiris/clarity-middleware.git

could be due to:

1 Like

I tried doing this and Xcode 13.2 produced the following error:

failed downloading 'https://api.github.com/repos/{owner}/{repo}/releases/assets/{release_id}.zip' which is required by binary target 'TargetName': badResponseStatusCode(404)

1 Like

We ended up setting up a proxy on AWS that rewrites requests from a custom domain to the GitHub domain.
https://customdomain.com/repos/{owner}/{repo}/releases/assets/{release_id}.zip

to

https://api.github.com/repos/{owner}/{repo}/releases/assets/{release_id}

We had to rewrite the Accept header to Accept: application/octet-stream as well. Our .netrc contains an entry with our custom domain but uses a GitHub login and token for the auth.

It feels like it's held together by duct tape, but it does work.

2 Likes

@tomerd @NeoNacho Any updates on this? I'd be happy to look into opening a PR if we agree on the approach...

@tallariel @NeoNacho @afarnham based on the thread it sounds like the remaining issue is that SwiftPM requires the artifact (asset) URL to end in .zip. If so, a PR that lifts this restriction but validates the downloaded file is a zip file (perhaps via the content-type header?) could do the trick?

@tomerd @NeoNacho To write and debug my implementation I was hoping to use the testBinaryTargetsValidation unit test, but this test currently fails on main (Every single one of its assertions fails). That unit test code needs to be updated and/or the actual manifest loading functionality isn't stable right now. Either way, I can't run or test basic manifest loading functionality, so shall I wait before continuing work on a PR?

I tried checking working on top of release/5.4, but that has diverged significantly from main.

@tallariel yes main is the branch we work on day to day and all the tests pass there. sounds like something is going on your machine / setup that causes the tests to fail. can you share the failure information? also make sure to follow the setup instructions on the contribution guide if you have done so already.

@tomerd

  • checked out revision 6e3c2ba1f363601116de0a05a65f9ed96994b3e7
  • opened it in Version 13.2 beta 2 (13C5081f)
  • selected the swift-package scheme
  • Product --> Test

The tests hit a runtime error on PD_4_2_LoadingTests.swift, line 597

Thread 1: Fatal error: 'try!' expression unexpectedly raised an error: PackageLoading.ManifestParseError.invalidManifestFormat("error: link command failed with exit code 1 (use -v to see invocation)\nld: library not found for -lPackageDescription", diagnosticFile: nil)

If I run just testBinaryTargetsValidation, which is the one I'm interested in, I get this result:

If I run swift test --filter testBinaryTargetsValidation the tests pass. But I'd like to run them from Xcode in order to facilitate debugging. Any thoughts?

It seems like the package manifest dylib is not getting built. Does it work better with SwiftPM-Package scheme? that should build the entire project when running test, including those dylibs

Yes, thanks. I've sent in a small PR to clarify the documentation accordingly. I'll carry on with the rest.

1 Like

@tallariel @afarnham not sure if this helps, but I just did a small test with a private GitHub.com repo hosting a binary artifact and got it to work with a small change to SwiftPM. here is the test setup:

  1. created a private repo on GitHub.com to host the binary artifact, i.e it has a single XCFramework file in it. in my case named test.xcframework.
  2. tagged the above private repo as 1.0.0
  3. used the GitHub.com release url for the binary artifact, e.g.:
.binaryTarget(
    name: "test",
    url: "https://github.com/tomerd/test-binary-dep/archive/refs/tags/1.0.0.zip",
    checksum: "..."
)

the missing part for this setup to work is that SwiftPM expects the downloaded zip to contain the framework directly while GitHub.com release zip files have a top level directory. e.g.

GitHub provides

1.0.0.zip
|- test-binary-dep-1.0.0
    |- test.xcframework

SwiftPM expects:

1.0.0.zip
|- test.xcframework

this is a classic "strip first level component" archive issue, and with a small patch to SwiftPM we can support both modes: [wip] support binary artifact archives that require first level component stripping by tomerd · Pull Request #4020 · apple/swift-package-manager · GitHub

does this help?

Depending on the macOS version used, the auth mechanism being used by the SPM http client may be flaky, because from what I can see, it doesn't use the URLSession authorization mechanism correctly: [SR-15731] swift-package-manager HTTPClient misuses URLSession reserved Authorization header - Swift

While technically true according to the documentation, that's not really true in practice. In common usage URLSession only cares about the Authorization header when the server is using one of the HTTP auth mechanisms that expects an authorization response, like HTTP Basic. In that case URLSession will try to use a provided URLCredential to create the Authorization header. If that's not the case, providing your own Authorization header works fine. It has to, otherwise various authorization mechanisms wouldn't work at all. Depending on the server, providing an Authorization header immediately may preempt any server side challenge and should work just fine. It's only when the server insists on issuing the challenge that you may see URLSession do something to the header.

1 Like

@tomerd It seems GitHub has two different behaviors depending on who owns the repository. Scenario 1, which you tested, is personal ownership. Scenario 2 is organization ownership.

Scenario 1 worked fine for me just now with Xcode 13.2.1 when I tried it with both my personal account with a private repository and my work account with a private repository that is not tied to my company.

However, in Scenario 2, when I try it with a repository owned by my company (organization in GitHub parlance), it fails. Instead I have to get the API URL using the GitHub command line app:

gh release view 1.0.0 --json assets | jq '.assets[]'

In the JSON returned I take the apiUrl value, append a .zip to the end and change the host to a proxy I setup. Then I use this in my Package.swift binary target.

When Swift PM resolves packages, it hits that proxy which then rewrites the request to add a Accept: application/octet-stream header and strips the .zip and then forwards the request to GitHub.

This proxied setup does work with Swift Package Manager today, but it would be nice not to have to run it.

I am using a local .netrc for authentication for the record.

@afarnham surprised it worked for you without the patch linked to above. can you share the url you used for your tests (just the shape of the url is enough if the url itself is sensitive), note in my test i’ve used the github’s release zip file url, not a github api based url

https://github.com/<username>/<repo name>/releases/download/1.0.0/myframework.xcframework.zip

That link is from a GitHub release for the given tag and when creating that release I drag the binary from desktop to the Release creation form.