Support for rewriting package urls?

Development Package Manager
1

Hi,
I am working for government and we cannot use GitHub. In order to use SPM I wrote a script that mirrors git repositories from GitHub into our internal bitbucket server. There, I can only use a single space "Vendors" so I have to change the name from i.e. "apple/swift-syntax" to "Vendor/apple.swift-syntax".

While this works for a single package, it won't work for dependencies. I found no way to add some kind of rewrite mechanism to SPM so the next best thing was for me to rewrite the git repositories (EXT -> rewrite -> INT-READONLY). While this works it is very cumbersome and I have to constantly update my binary project that does the rewriting. At least I found a reasonable approach by appending some swift code to the end of each Package*.swift file in each commit in each branch of a repository.

The question I have is:

  • Am I missing something here? Is there a way to use Xcode, or swift package on a development machine that has no access to the internet?

  • If not, what is the best way to suggest a feature request and to whom should I send it?

Thanks
Andreas Pardeike
Swedish Police IT Department

1 Like
2

I haven’t used it personally, but I believe SPM’s dependency mirroring will accomplish what you’re looking for.

3

It is my understanding that this works on the package level only. How would I tell our app developers to use a common package (like pointfreeco/swift-composable-architecture) in their Xcode projects?

4

Again, without having ever actually used the feature, I believe the workflow looks like:

  1. You mirror https://github.com/pointfreeco/swift-composable-architecture to, say, https://internal-bitbucket-server/pointfreeco-swift-composable-architecture
  2. Your developer runs the following command on their workstation (probably as part of a setup script for the app):
swift package config set-mirror \
    --package-url https://github.com/pointfreeco/swift-composable-architecture \
    --mirror-url https://internal-bitbucket-server/pointfreeco-swift-composable-architecture
  1. Your developers add https://github.com/pointfreeco/swift-composable-architecture to their Package.swift like normal
  2. When they build their app, Xcode/SPM instead reach out to https://internal-bitbucket-server/pointfreeco-swift-composable-architecture

Note that you'll also have to repeat steps 1 and 2 for every single dependency for https://github.com/pointfreeco/swift-composable-architecture, but the mirroring configuration is on the top level for the app they're building and should apply to all parts of the builds kicked off from that context.

5

As I said earlier, this only works if you use SPM. When running swift package config set-mirror it will tell you:

error: Could not find Package.swift in this directory or any of its parent directories

which is to be expected since a normal Xcode project does not have a Package.swift.

6

This is difficult, as Xcode doesn't support mirroring. You'll need to break all of your app's dependencies into a separate Package.swift and then use that file as a package in your app. You'd do all the mirroring in that file.

This is trivial in VSCode.

7

It does, but not in a very user friendly way.

If you place a mirror config in xcshareddata/swiftpm/configuration of the project/workspace in use, it'll be picked up and applied by libSwiftPM. This should work in pretty much any Xcode version for any dependency declared by a package, so any transitive dependencies.

There was an issue applying them to dependencies declared in Xcode projects specifically which has been fixed a few months ago in https://github.com/apple/swift-package-manager/pull/6765, but I am not 100% sure if that has made it into any Xcode releases, yet.

1 Like
8

This here is the information I was looking for. Thank you, I will try to write some helper binary/script that makes it easier is necessary.

I’ll see if this way actually works.

9

Btw this stackoverflow topic adds some more information to the details: xcode - Check in xcworkspace for swiftpm to git? - Stack Overflow

10

Hi again,
I now had time to test this but I cannot get it to work. Here is what I tried so far: created a new spm package and added the mirror there, in this case using GitHub - apple/swift-argument-parser: Straightforward, type-safe argument parsing for Swift

I then created a new app project in Xcode and copied mirrors.json from the spm package over, resulting in this file structure:

├── test
│   ├── Assets.xcassets
│   │   └── Contents.json
│   ├── ContentView.swift
│   ├── Preview Content
│   │   └── Preview Assets.xcassets
│   │       └── Contents.json
│   └── testApp.swift
└── test.xcodeproj
    ├── project.pbxproj
    ├── project.xcworkspace
    │   ├── contents.xcworkspacedata
    │   ├── xcshareddata
    │   │   ├── IDEWorkspaceChecks.plist
    │   │   └── swiftpm
    │   │       └── configuration
    │   │           └── mirrors.json
    │   └── xcuserdata
    │       └── u0035718.xcuserdatad
    │           └── UserInterfaceState.xcuserstate
    └── xcuserdata
        └── u0035718.xcuserdatad
            └── xcschemes
                └── xcschememanagement.plist

I then opened the Xcode project and used File > Add Package Dependencies... to add https://github.com/apple/swift-argument-parser.git but Xcode just asks me to log into GitHub (which via a proxy will fail). So it looks like the mirror is not used.

In contrast, if I go into Package.swift in the spm package project and add the dependency to https://github.com/apple/swift-argument-parser.git then it will successfully use our internal mirror from the command line:

> swift package update
Fetching ssh://git@internal.bitbucket.redacted.se:1234/vendor/apple.swift-argument-parser.git from cache

but not if I use Xcode to edit Package.swift:

Conclusion: To me this still looks like Xcode does its very own thing and I cannot use mirroring to solve my proxy problem.

1 Like
11

Note the second part of the answer:

2 Likes
12

Hi do you have any idea if this does works for nested dependancies? :thinking: Because I have xcworkspace/xcshareddata/swiftpm/configuration/mirrors.json and seems like Xcode doesn't respect it, at lest not for nested dependancies.

13

After a lot of back and forwards, I finally landed on a practical solution: Using git's “insteadOf” functionality.

https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf

As long as the change from official (public) repository url to internal url is done by a change in prefix, this method is dead simple. Just add a rule to your .gitconfig file:

[url “git://internal-git-server.intranet.com”]
insteadOf = https://github.com

And git will always switch the prefix "https://github.com/" with i.e "git://internal-git-server.intranet.com”.

We then run "git daemon" to get a git server for free and point its root to a directory containing mirrored repositories and keep them up to date. I tested it with Xcode and with SPM and it works direct or with transient dependencies.

/cheers

2 Likes
14

Yup, I'm also using insteadOf to rewrite SSH URIs to https with tokens for GitHub Actions. This took a bit of hair-tearing to figure out, so pasting in the magic below in case it's useful to anyone.

Steps:

  • Create a GitHub App for your organisation (this is necessary for the create-github-app-token action you will use
  • Generate and export a PEM private key for the app
  • For each repository with private dependencies, add the App's ID and the exported private key to the APP_ID and APP_PEM Actions repository secrets, respectively
      - name: Get token from Github App
        uses: actions/create-github-app-token@v1
        id: app-token
        with:
          app-id: ${{ secrets.APP_ID }}
          private-key: ${{ secrets.APP_PEM }}
          owner: ${{ github.repository_owner }}
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
          token: ${{ steps.app-token.outputs.token }}
          submodules: true
      - name: Fix permissions
        run: chown -R $(id -u):$(id -g) $PWD
      - name: Remap Git URLs
        run: |
          git config --global url."https://x-access-token:${{ steps.app-token.outputs.token }}@github.com/${{ github.repository_owner }}/".insteadOf git@github.com:${{ github.repository_owner }}/
3 Likes
15

Thank you both for the suggestions, but I need a mechanism that enforce the mirror url like SwiftPM mirroring. Using git's "insteadOf" potentially could do the work if its able to add it into the git repo, as I understand I can add it either to ~/. gitconfig or .git/config both not bing able to be added under version control. Or maybe I miss something?

16

Hello again, seem like swift package config set-mirror works, just that the mirrored urls are not reflected in Package.resolved.

1 Like