Package built with dependency on another package that uses a build tool plugin fails to find the build tool product

hassila · June 23, 2022, 11:01am

Run into another issue with build tool plugins that I am a bit stumped by;

I have the following packages:

package-data-model - defines a build tool plugin that will generate sources for the data model defined therein
api-core - uses the above build tool plugin (and that is the only dependency from package-data-model)
prototype-app - depends on the api-core packages Core target

Checking out api-core and building now works fine and the relevant source code is generated and included in the Core target.

Now comes the issue - when building the prototype-app I would expect SPM to check out the Core product from api-core, generate the source code with the build tool plugin from package-data-model and link everything, but instead I get a package resolve error:

'api-core': **error:** product 'DataModel' required by package 'api-core' target 'Core' not found in package 'package-data-model'.

The "DataModel" product is defined in package-data-model as:

        .plugin(name: "DataModel",
                targets: ["DataModelBuildTool"]),

so the product should be there?!

Looking in .build/, the package-data-model is not checked out

hassila@max ~/G/prototype-app (feature/sc-347/add-build-tool-for-data-model-generation)> ls .build/checkouts
api-app/           api-core/          flatbuffers/       package-plugin/    swift-docc-plugin/

Any hints for troubleshooting appreciated!

hassila · June 23, 2022, 3:53pm

Ok, I managed to get this to work - the error message is misleading - the problem is that for some reason the command line tool DataModelGenerator that the plugin depends on:

    targets: [
        .plugin(
            name: "DataModelBuildTool",
            capability: .buildTool(),
            dependencies: ["DataModelGenerator"]
        ),

is not available for consumers of the Core library unless specifically expressed as a dependency:

    targets: [
        .target(
            name: "Core",
            dependencies: [
                .product(name: "DataModelGenerator", package: "package-data-model"), // adding this made it work
                .product(name: "FlatBuffers", package: "flatbuffers"),
            ],
            plugins: [
                .plugin(name: "DataModelBuildTool", package: "package-data-model")
            ]
        ),

Wouldn't it be reasonable to expect that if the final prototype-app depends on api-core which depends on DataModelBuildTool (which has an explicit dependency on the DataModelGenerator) that it would 'just work' when trying to run the plugin?

Worth filing a bug for, or am I misunderstanding how it should work?

FranzBusch · June 23, 2022, 6:08pm

I think this is somewhat expected. You could think of the plug-in being a completely different product that is build and run. The output of the plug-in is then shared with your other target.

This goes hand in hand with the discussions around having separate dependency trees for plugins. Which makes sense if you think about it because the plug-in might run on your Mac whereas the target your are building might be for iOS.

hassila · June 23, 2022, 6:11pm

But shouldn't I then define the dependencies for the plugin and not for the 'Core' target? Something seems a bit off? [edit, I guess that those are the discussions that have been had somewhere that you are alluding to]

FranzBusch · June 23, 2022, 6:31pm

If I get your example right, you are doing the following:

You have a plugin DataModelBuildTool that is depending on DataModelGenerator
You have a target Core that uses the DataModelBuildTool plugin
You have a target prototype-app that uses Core

Looking at this you can think about these as two separate builds(dependency trees) happening within a single build invocation. First your DataModelGenerator executable is build, then the build for the prototype-app is kicked off which uses the built executable during its build.

I assume your generated code is importing DataModelGenerator somewhere and that's why you wanted the dependency to be automatically picked up. However, this is not always the case so SPM can't make that assumption and is keeping the dependencies separate.

Note: @NeoNacho and @abertelrud are the better people to comment on this and separate dependency trees are not a thing yet AFAIK.

NeoNacho · June 23, 2022, 7:45pm

Thanks Franz, what you're saying is exactly right.

I do think Joakim brings up a good point, though, and Anders has probably thought about this already. If the plugin generated code requires a dependency, it doesn't feel quite right for the client to have to declare it, since the indented experience is that adding the plugin usage is sufficient. So it would make sense to have a concept that does allow for the plugin to add dependencies to the client automatically, if needed. It would have to be explicitly declared though rather than just adding all the dependencies of the plugin target since those are intentionally separate.

icanzilb · June 23, 2022, 7:48pm

I'm with Joakim on this one — it feels like data duplication to have to both 1) add the plugin and 2) add its dependencies as well to the target, given that the dependencies are already declared on the plugin target.

I've hit the same problem and I just assumed this was a bug in the beta version.

hassila · June 23, 2022, 8:00pm

Thanks, as I see it there are two 'strange' things here which are similar but not the same:

I need to add the Flatbuffers dependency (which is what you discuss - basically the generated code is dependent on that)

but also needed is:

I need to add the DataModelGeneratorTool which is the dependency used by the plugin itself to actually generate the code (it's not a binary dependency currently but a SPM .executable dependency)

I don't know what happens if the DataModelGeneratorTool would have been a binary target dependency - maybe it would have worked then (as it seems the common sample example)?

Regardless, it seems that any client declaring a dependency on a plugin should be insulated to not have to manually duplicate that plugins dependencies (regardless of whether its a build tool or generated source code package dependency). It's a bit fragile also, as it's an implementation detail of the plugin that's leaking to its consumers unnecessarily.

hassila · June 23, 2022, 8:08pm

Just to be super clear, here's the setup with three separate SPM packages:

DataModel (providing the plugin, the test targets uses the plugin internally here but also explicitly adds dependency on flatbuffers)

// swift-tools-version: 5.6

import PackageDescription

let package = Package(
    name: "package-data-model",
    platforms: [
        .macOS(.v12),
    ],
    products: [
          .executable(
            name: "DataModelGenerator",
            targets: ["DataModelGenerator"]
        ),
        .plugin(name: "DataModel",
                targets: ["DataModel"]),
    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-argument-parser.git", .upToNextMajor(from: "1.1.0")),
        .package(url: "https://github.com/apple/swift-log", .upToNextMajor(from: "1.0.0")),
        .package(url: "https://github.com/apple/swift-system", .upToNextMajor(from: "1.0.0")),
        .package(url: "https://github.com/apple/swift-docc-plugin", from: "1.0.0"),
        .package(url: "https://github.com/mustiikhalil/flatbuffers", branch: "swift"),
    ],
    targets: [
        .plugin(
            name: "DataModel",
            capability: .buildTool(),
            dependencies: ["DataModelGenerator"]
        ),
        .executableTarget(
            name: "DataModelGenerator",
            dependencies: [
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
                .product(name: "Logging", package: "swift-log"),
                .product(name: "SystemPackage", package: "swift-system"),
                "DataModelDefinition",
            ],
            path: "Sources/DataModelGenerator"
        ),
        .target(name: "DataModelDefinition", path: "Sources/DataModelDefinition"),
        .testTarget(
            name: "DataModelGeneratorTests",
            dependencies: ["DataModelGenerator",
                           "DataModelDefinition",
                           .product(name: "FlatBuffers", package: "flatbuffers")],
            plugins: [
                .plugin(name: "DataModel")
            ]
        ),
        .testTarget(
            name: "DataModelTests",
            dependencies: [
                .product(name: "FlatBuffers", package: "flatbuffers"),
            ],
            plugins: [
                .plugin(name: "DataModel")
            ]
        ),
    ]
)

Core API - se comments for the dependencies, two slightly different issues that are unexpected

// swift-tools-version: 5.6
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "api-core",
    platforms: [
        .macOS(.v12),
    ],
    products: [
        .library(
            name: "Core",
            targets: ["Core"]
        ),
    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-docc-plugin", from: "1.0.0"),
        .package(url: "https://github.com/ordo-one/package-data-model", branch: "feature/sc-347/add-build-tool-for-data-model-generation"),
        .package(url: "https://github.com/mustiikhalil/flatbuffers", branch: "swift"),
    ],
    targets: [
        .target(
            name: "Core",
            dependencies: [
                .product(name: "DataModelGenerator", package: "package-data-model"), // this tool is needed to build the build plugin source
                .product(name: "FlatBuffers", package: "flatbuffers"), // this is needed to be able to build the generated code
            ],
            plugins: [
                .plugin(name: "DataModel", package: "package-data-model")
            ]
        ),
        .testTarget(
            name: "CoreTests",
            dependencies: ["Core"]
        ),
    ]
)

Final app consuming the Core API

// swift-tools-version: 5.6

import PackageDescription

let package = Package(
    name: "prototype-app",
    platforms: [
        .macOS(.v12),
    ],
    products: [
        .library(
            name: "xxx",
            type: .dynamic,
            targets: ["xxx"]
        ),
    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-docc-plugin", from: "1.0.0"),
        .package(url: "https://github.com/ordo-one/api-app", branch: "feature/sc-347/add-build-tool-for-data-model-generation"),
        .package(url: "https://github.com/ordo-one/api-core", branch: "feature/sc-347/add-build-tool-for-data-model-generation"),
    ],
    targets: [
        .target(
            name: "xxx",
            dependencies: [
                .product(name: "Application", package: "api-app"),
                .product(name: "Core", package: "api-core"),
            ]
        ),
        .testTarget(
            name: "xxxTests",
            dependencies: ["xxx"]
        ),
    ]
)

SDGGiesbrecht · June 23, 2022, 11:39pm

I could swear there are tests that ensure this. I remember fighting with precisely this use case when reconciling plugins with target‐based dependencies.

Right now I can find this, which may only be in main and not released. However, it only tests depth to one degree, and you said the first degree worked and only the second failed. It also differs in that the executable was not even exposed as a product.

That last difference pretty much proves it is intended to work as you assumed, and therefore what you see is an unintended bug.

hassila · June 27, 2022, 7:11am

For future googlers, I opened an issue on the DataModelGenerator dependency failure here:

https://github.com/apple/swift-package-manager/issues/5630