SPM support for dependency transformation plugin

Development Package Manager
#1

We have a particular use case where security/availability requirements within our organisation prevent us from depending on external services (such as Github) during the build process. We want to be able to depend on the original Github repositories during development however to avoid setting up mirrors for dependencies we may never take to production.

We currently manage this requirement by building a shim layer around Git that transforms the dependency urls as SPM is asking for them. This transform follows some basic transform logic for different url domains. This mechanism is fragile and not really ideal as it requires setting up the shim externally to SPM before a build.

SPM's current dependency mirroring mechanism - defined in SE-0219 - is also not well suited for this situation as it is an explicit mapping from package-url to mirror-url. Either we set up this config manually and risk it becoming out of date as our dependencies change or have a different automated process external to SPM before a build to setup all the mirrors required.

I have been considering ideas around how to incorporate the url transformation logic more officially into SPM. My current thoughts are something like this-

let package = Package(
    name: "SomePackage",
    platforms: [
      .macOS(.v10_13), .iOS(.v10)
    ],
    products: [
        .library(
            name: "SomePackage",
            targets: ["SomePackage"]),
    ],
    dependencies: [
        .package(url: "https://github.com/org/some-dependency.git", from: "1.0.0"),
        .package(url: "https://github.com/org/dependency-transform.git", from: "1.0.0"),
    ],
    plugins: [
        .dependencyTransform(executable: .product(name: "DependencyTransform", package: "dependency-transform"),
                             condition: .when(environment: ["MIRROR_DEPENDENCIES"]))
    ],
    targets: [
        .target(
            name: "SomePackage", dependencies: [
                .product(name: "Dependency", package: "some-dependency"),
            ]),
    ],
    swiftLanguageVersions: [.v5]
)

That is, there is a build plugin of type dependencyTransform that is either defined in the same package or another SPM package and can be restricted to being used in particular environments. The plugin would have to point to an executable product that responds to a simple command-line protocol; package-url -> mirror-url/error.

Thoughts on the feasibility/desirability of incorporating this into SPM are welcome along with thoughts on the general approach.

Extensions

There is an advantage to keeping the solution generic to allow for additional build plugins in the future, for example something like code generation-

let package = Package(
    name: "SomePackage",
    platforms: [
      .macOS(.v10_13), .iOS(.v10)
    ],
    products: [
        .library(
            name: "SomePackage",
            targets: ["SomePackage"]),
        .library(
            name: "GeneratedPackage",
            targets: ["GeneratedPackage"]),
        .library(
            name: "ConsumingPackage",
            targets: ["ConsumingPackage"]),
        .library(
            name: "ConsumedPackage",
            targets: ["ConsumedPackage"]),
    ],
    dependencies: [
        .package(url: "https://github.com/org/some-dependency.git", from: "1.0.0"),
        .package(url: "https://github.com/org/dependency-transform.git", from: "1.0.0"),
        .package(url: "https://github.com/org/model-generation.git", from: "1.0.0"),
    ],
    plugins: [
        .dependencyTransform(executable: .product(name: "DependencyTransform", package: "dependency-transform"),
                             condition: .when(environment: ["MIRROR_DEPENDENCIES"])),
        .codeGeneration(name: "ModelGeneration",
                        executable: .product(name: "ModelGeneration", package: "model-generation"),
                        parameters: [
                            "SWAGGER_PATH": "Swagger.yml"
                        ],
                        dependencies: [
                            "OPERATIONS_PACKAGE": .target(name: "ConsumedPackage"),
                        ])
    ],
    targets: [
        .target(
            name: "SomePackage", dependencies: [
                .product(name: "Dependency", package: "some-dependency"),
            ]),
        .target(
            name: "GeneratedPackage", .generatedBy("ModelGeneration")),
        .target(
            name: "ConsumingPackage", dependencies: [
                .target(name: "GeneratedPackage"),
            ]),
        .target(
            name: "ConsumedPackage", dependencies: [
            ])
    ],
    swiftLanguageVersions: [.v5]
)

Which would follow a similar model; a codeGeneration plugin would have to point to an executable product that responds to a different command-line protocol in order to interact with the build process.

#2

There's an existing thread on extensible build tools which is similar to the code-generation plugins that you mention.

For the URL transformation idea specifically, it feels a bit off to me to couple the requirements of a particular build environment with the description of the package like this. Presumably, different organisations would need different transformations, so how would that work if they collaborate on a package together, does everyone create their own fork with their particular transformation added?

Maybe we should instead have a way to describe transformations in the mirror configuration?

Package Manager Extensible Build Tools
#3

Thats a fair point. Something like this proposal doesn't prevent such a scenario where multiple dependencyTransform plugins could be listed and activated for different environments.

In terms of coupling the requirements of a particular build environment to the package definition, doesn't the package manifest file already describe such requirements - whether it is requirements for different platforms or different configurations?

#4

That's true, maybe a better way to put my concern is saying that this is adding something that's largely external and orthogonal to the individual package into its core description. This would need to be added to each package that is ever being used as a root package by an organization and there would need to be one plugin listed for each such organization. This makes it seem that it would be better to find a solution which doesn't require for this to exist inside the package manifest.

Do you think it would be feasible to instead extend the mirror configuration into supporting some kind of rule mechanism? Are the rules you need typically very complex so that they need custom logic or could they be expressed e.g. as regular expressions?

#5

Although this probably won’t help with the actual question, maybe it will help with the implementation. I have in the past used this command to redirect GitHub URLs to SSH instead of HTTPS; you could probably do something similar to replace GitHub with your own mirror.

git config --global --add url."git@github.com:".insteadOf "https://github.com/"