SPM support basic auth for non-git binary dependency hosts

This should be included in beta 3. Also note that we support the keychain which is a much safer alternative to .netrc

2 Likes

Xcode 13.3 BETA 3 seems to do the job for me. Thanks a lot @tomerd

Also note that we support the keychain

:+1:

1 Like

I independently confirm that with Xcode 13.3 Beta 3 it is possible to download an asset (some.xcframework.zip) of a GitHub release in a private GitHub repository.

As outlined in this thread the asset url needs to be used and the .netrc file needs to contain an entry for machine api.github.com, GitHub user name for login and a personal access token scoped to access private repos as password.

I wrote a blog post as summary as this feature is not mentioned in the Xcode 13.3 Beta 3 release notes.

5 Likes

Nice summary in the blog article @MarcoEidinger :+1:

2 Likes

@MarcoEidinger’s blog post gives great instructions for using .netrc, but how would I use the keychain for this @tomerd?

@spacecrafter3d I was able to to achieve the same download by creating an Internet password in the keychain. To create such use "New Password Item" (⌘N) and specify the the following attributes

Keychain Item Name https://api.github.com
Account Name GitHub user name
Password personal access token

I updated my blog post with the information and also added screenshots for better understanding.

3 Likes

That’s perfect, thanks!

I was able to get this working with the keychain by following @MarcoEidinger's instructions. One issue that tripped me up for a long time was that apparently you have to quit Keychain Access before the new entry is available to Xcode.

Is there any chance this could work using Xcode's built in account system (for those using Xcode) instead of adding the password to Keychain?

Note not to use --json apiUrl as mentioned in one of the posts above, this would give you the release url, not the asset url. You want to use --json assets as mentioned in other posts.

For reference, the error I got was: failed downloading 'https://api.github.com/repos/<org>/<repo>/releases/<release-id>.zip' which is required by binary target 'XX': badResponseStatusCode(415)

1 Like

Good to see this is now in Xcode 13.3!

But now it seems there's a chicken and egg problem when using this with GitHub and SPM:

  • Package.swift must specify the file's URL
  • We know that URL only once the file has been added to a GitHub release as an asset
  • That GitHub release is only created after we push a tagged commit, in which the Package.swift file specifies the... file's URL, which doesn't exist at that stage as the release hasn't been created yet

@MarcoEidinger @GevaZeichner how did you get this to work ?

@tallariel you need another dedicated repo just for the Swift package for making this flow work.

e.g.
You have a repo MyAwesomeFramework which publishes the xcframework as a release asset.
And then you need another repo MyAwesomePackage that contains the Package.swift and references the release asset URL of MyAwesomeFramework.

I was in the same position and based on other answers somewhere in the net and my understanding of how the whole thing works, this is the best solution so far.

EDIT: We have also automated that process for some of our internal packages. If there's enough interest I can maybe summarise this in some blog post.

You can create a draft release with your binary assets attached. When that release moves from draft to published the asset urls remain the same. Our flow is:

  • Point the Package.swift to the draft release assets
  • Push the updated Package.swift + tag
  • Update the draft release to use the new tag
  • Move the release from draft to published
1 Like

Interesting @afarnham, but that wouldn't work if one would want to trigger a CI pipeline based on a git tag, if I understand correctly?

A CI pipeline does complicate it quite a bit. If you could edit the Package.swift file programmatically to update the asset URLs you could probably accomplish it. Maybe by integrating libSwiftPm? Last I checked that library was still marked as unstable so probably not a great solution.

@afarnham we already have it automated on CI, see my other comment :)
What I meant was that the flow you suggested would not be possible technically if the CI pipeline is kicked off by a git tag. Because based on the git tag also the Github release is created and I wouldn't know how to do it otherwise - and in your flow the Github draft release exists before the git tag is created.
Anyway I hope @tallariel has enough now to get started.


p.s.
Replacing the asset URL and checksum in an automated way we do with sed, no need for another library :)

For this to work for example we define two lets in the Package.swift to unambiguously identify the proper url and checksum

let assetUrl = "https://api.github.com/repos/myorg/mypackage/releases/assets/12345.zip"
let checksum = "abcde12345"

let package = Package(name: "..."
...

and then one can easily replace those values:

# Use '#' as a delimiter since the asset URL contains slashes, and this would break the 'sed' command otherwise
sed -i '' "s#let assetUrl.*#let assetUrl = \"$ASSET_URL\"#" ./Package.swift
sed -i '' "s#let checksum.*#let checksum = \"$CHECKSUM\"#" ./Package.swift
1 Like

Here's a quick javascript to do the same thing (replace an asset URL/checksum in a Package manifest), but in a more "formatting-agnostic" way. It also supports updating multiple binary targets in a Package manifest. I've omitted error handling for brevity.

#!/usr/bin/env node

const exec = require('util').promisify(require('child_process').exec);
const fs = require('fs').promises;

async function run(targetName, newUrl, newChecksum) {
    const { stdout } = await exec('swift package dump-package');
    const packageJson = JSON.parse(stdout);
    const { url, checksum } = packageJson.targets.find(t => t.name === targetName);

    let packageSwift = await fs.readFile('Package.swift', 'utf8');

    packageSwift = packageSwift
        .replace(url, newUrl)
        .replace(checksum, newChecksum);

    await fs.writeFile('Package.swift', packageSwift, 'utf8');
}

run('YourBinaryTarget', 'https://url/to/asset.zip', 'new-checksum');

@bricker nice, thanks for sharing.

In my case I haven't had multiple binary targets, that's why I went with this "simplified approach". I'll keep your solution in mind though for the future in case I need to support multiple binary targets :+1:

for CI testing, you can also potentially use an env variable to "feed" into the manifest instead of generating a complete manifest

I'm not sure which solution you are referring to, but both our approaches are just replacing the URL and checksum in an existing manifest, and not "generating a complete manifest" :)

That having said, I'd love to see more support from SPM directly for achieving such things.
Another use case is updating a single(!) package within a package that is used in an Xcode project. I also only see a way of achieving this with custom code. But I'm going off-topic here, so I'll stop :)

@tomerd is your patch in the Xcode 14 betas?

Using a private company repo with both the .netrc and Keychain approaches setup on my machine, I can curl to fetch the asset:

curl \
    --location \
    --netrc \
    --output release_asset.zip \
    -H "Accept: application/octet-stream" \
    "https://api.github.com/repos/:owner/:repo/releases/assets/:id.zip"

Using that url in SPM however fails: failed downloading 'Name' which is required by binary target 'other name': badResponseStatusCode(404).

.binaryTarget(
    name: "Name",
    url: "https://api.github.com/repos/:owner/:repo/releases/assets/:id.zip",
    checksum: "checksum"
),