Difficulty Sharing Code Between Swift Package Manager Plugins

Using Swift
1

I'm currently writing an SPM package to house several utility plugins for a codebase, and these plugins overlap in functionality. Because of this, they currently duplicate code verbatim that I'd love to factor out — but I've run into a few difficulties in doing so.

My initial approach was to factor out the common code into a shared "Common" target that could be exposed to these plugins:

// Package.swift
import PackageDescription

let package = Package(
    name: "…",
    products: [
        .plugin(name: "Plug1", targets: ["Plug1"]),
        .plugin(name: "Plug2", targets: ["Plug2"]),
    ],
    targets: [
        .target(name: "Common"),
        .plugin(name: "Plug1", capability: .buildTool(), dependencies: ["Common"]),
        .plugin(name: "Plug2", capability: .command(intent: ...), dependencies: ["Common"]),
    ]
)

The file hierarchy:

.
├── Package.swift
├── Plugins
│   ├── Plug1
│   │   └── plugin.swift
│   └── Plug2
│       └── plugin.swift
├── README.md
└── Sources
    └── Common
        └── Common.swift

Unfortunately, with this setup, it appears that neither plugin is able to see or import a Common module, nor use any of its contents: attempting to do so just results in No such module 'Common'.

  • The Common target is being built: if I #error in one of its source files, the build fails as expected
  • All of the contents of Common are marked as public
  • If I add another target, it's able to see, import, and use the Common module even if I don't add Common as a dependency; the package builds just fine if neither plugin attempts to use the shared code
  • Adding Common as a public product of the package yields no change
  • Moving the module sources on disk (adjusting path and sources for the target) doesn't appear to change visibility

So, am I doing something wrong here? Is this scenario expected to work, or is this being disallowed for reasons I'm unaware of?

As an alternative approach, I tried sharing code more directly by directly listing target sources and including Common.swift in both Plug1 and Plug2, as well as attempting to merge Plug1 and Plug2 by making one type which offers both BuildToolPlugin and CommandPlugin interfaces, but both of these attempts failed as SPM doesn't allow overlapping sources to be used in multiple targets.

While I understand this restriction for regular source libraries (where multiple targets can be linked together to lead to linking conflicts), should this apply for plugin targets? My gut feeling says "no" — as far as I can tell, plugins can't reasonably be linked together in a way that makes overlapping sources a concern. But if I'm wrong on this, I'd love to learn something new!

Happy to file issues / Feedback, but wanted to make sure first this wasn't user error, and that these scenarios are ones we'd actually like to support.

Tangential Follow-Up Question

Some of the code I was hoping to share in Common actually contains extensions to types vended by PackagePlugin and XcodeProjectPlugin — but it appears that those modules are only exposed to .plugin targets, and are inaccessible anywhere else (No such module 'PackagePlugin'). Is this also intentional? (I can sort of see this making sense as those modules don't make much sense for use outside of .plugin targets, but this also feels like a limitation that would be useful to lift without being risky.)

2

possibly related: SPM plugin dependency on ArgumentParser?

3

Interesting. It sounds, then, that the expected approach might be to declare two .plugin targets which dispatch to a central .executableTarget which itself contains the shared code? (Feels unintuitive! Wish this were documented somewhere.)

I'll try to see if I can get this to work. The plugins right now do little but process a bit of info from the PluginContext, looking up a tool from another binaryTarget that I expect to use, and just dispatching off to that. I'd need to pass a lot of context to the executable target, and I hope I can run the external tool from in there. (And hopefully I'll be able to execute the tool from in there, given that plugins are sandboxed.)

[If this is indeed the expected approach, it feels very limiting — I have all of the context I need from within the plugin, and having to pass all of that in a digestible way to another tool just for the benefit of sharing a source file feels like overkill.]

4

That should have errored already when the manifest was loaded, because such dependencies are explicitly not supported.

Yes. The point in the separation was that all actual work would be part of the build plan to take advantage of caching, etc. The plug‐in definition, like the manifest, is supposed to be a minimal isolate containing only what is needed to create the build plan in the first place. But that design resulted in a lot of unforeseen, unintuitive roadblocks like this that make the initial release nearly useless for many anticipated use cases.

5

Indeed, I would have found this scenario easier to diagnose if the manifest reported an error along the lines of Plugins can only depend on executable targets or similar. (Though it would be pretty difficult to fit an explanation of where to go from there in the error message.)

Is there any official documentation/guidance you can point me towards to learn more about this? I've been having a hard time finding documentation on SPM plugins beyond this year's WWDC videos:

Having watched both, it feels like unless I've missed something obvious (entirely possible!), neither has seemed to hint at this sort of design or restriction in implementation, so I'm wondering how one is supposed to figure all of this out.

6

I tried refactoring in this direction, with the following (simplified) setup:

// Package.swift
import PackageDescription

let package = Package(
    name: "…",
    products: [
        .plugin(name: "Plugin", targets: ["Plugin"]]
    ],
    targets: [
        .executableTarget(name: "Executable"),
        .plugin(
            name: "Plugin",
            capability: .buildTool(),
            dependencies: ["Executable"]
        )
    ]
)

The file hierarchy:

.
├── Package.swift
├── Plugins
│   └── Plugin
│       └── plugin.swift
├── README.md
└── Sources
    └── Executable
        └── main.swift

The plugin is set up to return a .prebuildCommand which dispatches to Executable — and I can confirm that the plugin runs. However: the path to Executable that's found is not an absolute path; unlike the path that I get when trying to look up a .binaryTarget, the path I get back from context.tool(named:) is /${BUILD_DIR}/${CONFIGURATION}/Executable.

When this path is returned to .prebuildCommand, these variables aren't expanded as expected (as far as I can tell), and the invocation that's run is

/usr/bin/sandbox-exec -p "(version 1)
(deny default)
(import \"system.sb\")
(allow file-read*)
(allow process*)
(allow file-write*
    (subpath \"/private/tmp\")
    (subpath \"/private/var/folders/…\")
)
(deny file-write*
    (subpath \"…\")
)
(allow file-write*
    (subpath \"…\")
)
" "/${BUILD_DIR}/${CONFIGURATION}/Executable" <arguments>

The result of this invocation is The file “Executable” doesn’t exist.

I do see that an Executable binary has been built in ~/Library/Developer/Xcode/DerivedData/<project dir>/Build/Products/Develop Debug/Executable, as well as in a few places in Intermediates.noindex/<package name>.build/Develop Debug/Executable.build, so my guess is that BUILD_DIR and CONFIGURATION may not be getting set or expanded in executing this.

Is there anything explicit I should be doing to get this invocation to work? On the face of it, it doesn't appear that there's enough info in PluginContext to expand these variables myself if I wanted.

7

I wonder if there might be an issue with custom build configurations. IIRC, we do some trickery to map the configuration specified by the scheme to the ones existing in a package (always just "Release" and "Debug") to make built products get found in their expected location, but I am not sure whether that extends to CONFIGURATION having the correct value.

8

Ah, hmm. The Xcode project I'm building this in has its build configurations defined through xcconfig files — I wonder if that's enough to trip this up. I'll try in a bare Xcode project to see if there's a difference.

9

Sadly, no luck in a brand new project:

  1. Xcode > File > New > Project
  2. Created an iOS "App" project
  3. File > Add Packages… > Add Local… → added the plugin package
  4. In the project "Targets > Build Phases", added the build tool plugin to the "Run Build Tool Plug-Ins" build phase
  5. Built successfully

The result is the same: same invocation (/${BUILD_DIR}/${CONFIGURATION}/Executable), same error (The file “Executable” doesn’t exist.). The executable was indeed built in the Debug products directory.

10

In the end, going with an executable target feels like a lot of overhead and complexity, for little-to-no benefit in my case. A simple and effective, if somewhat "improper", workaround: symlinking the files into each target from a shared directory.

.
├── Package.swift
├── Plugins
│   ├── LintCommand
│   │   ├── File1.swift -> ../Common/File1.swift
│   │   ├── File2.swift -> ../Common/File2.swift
│   │   └── plugin.swift
│   ├── LinterBuildTool
│   │   ├── File1.swift -> ../Common/File1.swift
│   │   ├── File2.swift -> ../Common/File2.swift
│   │   └── plugin.swift
│   └── Common
│       ├── File1.swift
│       └── File2.swift
└── README.md

This works exactly as expected, working around the "overlapping sources" restriction on plugins.

1 Like
11

Not that I know of. My initial vague understanding came from the Swift Evolution proposals, where there is enough information to deduce the intent and most of the resulting restrictions, but only if you know what you are looking for. I still reached for the wrong way the first few times, before I had a better idea of how it was supposed to work. Unfortunately, I think having named the definition “plug‐in” instead of the workhorse probably sends every new user in the wrong direction.

But the fact that I understand it confidently enough to post an answer to your question comes only from deeper knowledge gleaned by working with the SwiftPM source files and test suite in order to reconcile bugs found in the overlap with other features.

I thought about suggesting this workaround in my first post. I didn’t because I doubt it works on Windows, it cannot share sources between packages, and some naïve formatters overwrite symlinks with the actual contents of the modified file. But if those sorts of things are not an issue, then yes, it is the simplest workaround I know of.

12

Thanks for your insights, and for the confirmation! I'm at least unblocked in the meantime.

While there's room for improvement in this area, luckily it doesn't feel insurmountable to get there. At the very least, more visible documentation could improve things significantly.

1 Like
13

The same limitations are also preventing me from vending a utility library for working around bugs in SPM, FWIW. At the moment, the best I can do is distribute a source file that people copy into each plugin.

3 Likes
14

Any updates on this?
I just want to share a target between my plugins

15

Great workaround!

I wanted to report for anyone stumbling across this that it also works with symlinked folders like this:

.
├── Package.swift
├── Plugins
│   ├── LintCommand
│   │   ├── Shared -> ../Shared
│   │   └── plugin.swift
│   ├── LinterBuildTool
│   │   ├── Shared -> ../Shared
│   │   └── plugin.swift
│   └── Shared
│       ├── File1.swift
│       └── File2.swift
└── README.md

So for each of the plugin targets, you would simply

cd <plugin_target_dir>
ln -s ../Shared Shared

Hopefully dependencies for plugins can be made a bit more related to other target types in the future.

16

Keep in mind that symlinks are not supported on Windows, at least to the same degree they are supported on other platforms. The recommended workaround is to create a separate executableTarget instead, to which you can then delegate from plugins.

1 Like
17

Some context, this is a common extension I like to put in my plugins to help target certain filetypes within Xcode targets that would be nice to not have to copy pasta repeatedly per plugin

import XcodeProjectPlugin

extension XcodeTarget {
    
    func sourceFiles(withSuffix suffix: String) -> [FileList.Element] {
        inputFiles.filter { $0.type == .source && $0.path.extension == suffix }
    }
}

This is just a minor example of what would go into a swift file within the Shared folder. There are other "path helper" extensions that I like to use for syntactic sugar purposes.

1 Like
18

In my experience symlinks work perfectly well for this and many other things on windows. For some reason they’re disabled by default but once you enable them, there’s no problem.