SE-0303: Package Manager Extensible Build Tools

Thanks for the detailed and thoughtful feedback!

As you note, the source files of dependencies are available to the plugin, but currently the outputs defined by other plugins aren't. But this is a great point. The proposal could define that plugins are invoked in dependency order (with independent ones being allowed to be invoked in parallel, of course), and provide for a way of passing the results of those invocations on to downstream plugins.

The idea with the current proposal was that the TargetBuildContext would contain enough information to allow each plugin to construct a module mappings file for the closure of dependency targets, and the plugin would be able to write this out directly to the output directory (and not have to use a separate external command) since the information is known before any commands are run. But as you point out it might be better to allow access to the outputs of the other plugins, so that the component module mappings could be reused.

Would it be important for the plugin to be able to distinguish between original source files and those produced by another plugin? In the case of protoc this might not matter, but for different kinds of plugins (e.g. linters) there might be a need to distinguish original sources from generated sources.

Having a custom type for a file set would make sense. There should also perhaps be a way for the plugin to provide a filter for the set of input files it sees. Having access to all the files is flexible but it might make sense to specify a filename filter up-front (in the declaration of the plugin, for example).

In this proposal the plugins are invoked before the build (while the build plan is being created), so if the proposal is amended to define plugins as being invoked in dependency DAG order (and the TargetBuildContext is extended with the appropriate fields), the plugins for a target would have access to any intermediates written by plugins for its dependencies. Even as written, in this proposal the commands produced by a plugin would have access to the outputs of other commands run for the dependency targets.

So this would not work if a plugin needs to see the outputs of running build-time commands from other plugins, but it does work if, for example, the plugins themselves write out the module map files (which was the intent in the example in the proposal).

This is one of the simplifications that this proposal makes, with an eye toward being able to loosen that restriction in the future. Ideally a plugin would be able to be invoked as needed by a build system capable of doing that, and as long as the inputs and outputs of the plugin are well defined.

This initial proposal is focused on source generation, and initially any platform differences would need to be written into the generated source files using conditionals. To support tools that do actual compilation, or to allow more flexible source generation, I would expect this proposal to be extended in the future to let the plugin have access to target platform information (and to declare whether it wants to be invoked once for all the platforms, or once per platform, etc).

The current approach does let the plugins write "up-front" information that can be derived just from the configuration and structure of the target to which it's being applied and it's dependency closure (such as the module mappings). But it's a good point that it might be better to require it to use custom APIs to provide the contents of any such files, if that makes the invocation more portable. The idea would be that a plugin would always generate the same outputs (in terms of command lines defined using addCommand as well as intermediate inputs to those commands written up-front), but it might be better to extend the API to support that rather than allow the output directory to be mutable.

Thanks for reply,

I don't know what systemModule is. Is it something like systemLibrary target? When I read the proposal, I made this assumption.

About GLib/GObject/GTK...
The usual way of obtaining the libraries on macOS is using brew. Brew downloads the library and installs it into /usr/local/Cellar/gtk+3/3.24.25/lib on my mac. The .gir file is than located in /usr/local/Cellar/gtk+3/3.24.25/share/gir-1.0 and is part of the downloaded package.

On previous versions of macOS, all required gir files were automatically stored at /usr/local/share/gir-1.0 , on distributions like Ubuntu, gir files are also stored in one directory. We would need SPM extensible build tools to allow us read this concrete directory.

Several users complained, that on the lates macOS 11 and computers with M1 chips, the /usr/local/share/gir-1.0 directory DOES NOT contain all required gir files.
Therefore, we need to get the gir file from the directory where library is installed by homebrew. I have came up with solution, that involves using pkg-config like this pkg-config --variable=libdir gtk+-3.0 appending path /../share/gir-1.0/.
Therefore, on the new macOS 11, we would not only need access to any directory on the filesystem, we would also need the ability to initiate process and call shell/pkg-config.

I understand that this post might be a bit confusing, but I wanted to keep the post short. Let me know if you would like me to clarify something :)

1 Like

Hopefully not, generally, not exposing "everything" and making them declare what they need access too keeps down what you'd have to copy if you ever did want to run these tasks remotely.

Slight tangent - if one did declare needed inputs/output, would prebuild/postbuild be needed? Or would that become implicitly based on taking no inputs or on having a wildcard for all outputs as the input?

And a second tangent - on the postbuild - is the tool allowed to modify the things in the output directory or just add things? i.e. - could it say remove unneeded localizations? In anycase, when does it run in relation to something like codesigning? I don't think I saw that mentioned.

The source generation is a good point, what about resource generation? Say a 3d library might want to have a plugin for generating textures out of images? Or maybe something to extract images/icons directly from photoshop files? This also sorta brings up something else I didn't realized until re-reading, currently all the sources are exposed to every plugin and every plugin then has to filter them. Since the target also implies a module, does this force folks to put everything for each plugin into it's own module, or should there be a way in the Package file to say which files should be sent to each plugin?

It certainly makes sense.
Thanks for answering.

More reason for a +1


What is your evaluation of the proposal?
-0.5. It does bring value to SPM but the "scripting part" seem quite complex IMHO.

Is the problem being addressed significant enough to warrant a change to Swift?


Does this proposal fit well with the feel and direction of Swift?

IMO no.
I like the simplicity of added attributes in Package.swift but plugin scripting is about building a builder tool and not actually scripting.
I think it’s important that we keep simplicity and that we ask people to actually write a script and not define any kind of dsl/rules for the swift compiler which is the way it is going.

Also I’m not sure with current implementation plugins can be runned manually?
For instance I have a project where I generate some translations. I want to be able to run it while building and manually when I want. I think it’s important to not be tied to the compilation phase for these things.

To me we already have a library which could help us in doing what we want: Swift Argument Parser. Was there any thoughts about using (and enhancing) it to make the plugin itself? For instance taking SwiftGen example:

struct SwiftGenPlugin: PluginCommand {
    let buildContext: BuildContext

    var generatedSourceFiles: [FilePath] = []
    lazy var prebuildGeneratedSourceDirectory: [FilePath] = [

    func run() {
        .lookupTool(named: "swiftgen")
            arguments: "config", "run", "...", // don't really see the point of a Dictionary here, what about using Variadic?
            environment: [...]

This would allow:

  • to run scripts outside of the build phase (though this would still require some logic from SPM to set the buildContext)
  • to have better SoC between running the script and any mandatory data (like generated source files or directories)
  • There could be an easy migration from “scripts” using Swift Argument Parser to this
  • We could give arguments both manually and through Package.swift (adding a arguments: [String:String] property)
  • IMHO, it's easier to read as we have a run() method

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

Used Cocoapods and NPM and it seemed simpler with these tools.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

A (multiple) quick reading

API naming nits:

"Lookup" is the noun or adjective, "look up" the verb. It sounds like the intention is to have a method named lookUpTool (rather than lookupTool, which implies that you're returning a "lookup tool," i.e., a tool for looking things up). Another consideration is whether this API should have a "noun" name or a "verb" name, but in either case, it wouldn't be lookupTool.

I don't know of other uses of "Dir" as an abbreviation in Swift APIs; would prefer "Directory" here.

  • What is your evaluation of the proposal?

I think this is an excellent proposal that will greatly expand the capabilities of the package manager in really important ways. Enthusiastic +1, although I do have some ideas for improvements.

  • Is the problem being addressed significant enough to warrant a change to Swift?

Absolutely. The proposal does a good job of illustrating some of the ways developers use tools such as source generation today. I'm generally quite averse to adding dependencies just for developer QoL features, but after trying one of these out (I think it was R.swift) a few years ago, I became convinced that they really are quite valuable. I really like the way they're able to bridge your code with your resources, and make them feel like more of a cohesive whole.

  • Does this proposal fit well with the feel and direction of Swift?

I think it does - Swift is all about reducing runtime errors by being smarter at build time (typically compile-time, but no reason not to extend that to the package build process).

  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I think this is easier for beginners to set up than Xcode's build phase scripts, and much more convenient for linux users, who otherwise might have to build via a bash script.

  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I read the proposal and did a bit of research.

As for improvements - I think the exposed API for build tools could be improved.

  1. Target system information (not "build target" - "target" as in host/build/target machines). When cross-compiling, a plugin may very well need to know the architecture, OS (and possibly minimum deployment... uh... target), and other information in order to generate the correct files or pass the correct flags to external tools. This API doesn't provide that, meaning developers will have to rely on build-machine information using conditionals such as #if os(macOS). Sysroot is also very useful.

  2. File information. It's a bit unfortunate how the API globs sources, resources, and other files in the same list, with nothing more than a FilePath given to the tool. It might be nice to make this an enum, or to wrap it in a struct with some type of enum Kind property.

     /// Absolute paths of the files in the target being built, including the
     /// sources, resources, and other files. This also includes any source
     /// files generated by other plugins that are listed earlier than this
     /// plugin in the `plugins` parameter of the target being built.
     var inputFiles: FileList { get }
     /// Provides information about the input files in the target being built,
     /// including sources, resources, and other files. The order of the files
     /// is not defined but is guaranteed to be stable.
     /// This allows the implementation to be more efficient than a static
     /// file list.
     protocol FileList: Sequence {
         func makeIterator() -> FileListIterator
     struct FileListIterator: IteratorProtocol {
         mutating func next() -> FilePath?
  3. Convenience APIs. I understand that this is V1 of this feature, but the API is pretty verbose - for instance, having to separately look up a tool and add the command which uses it. Lots of plugins will probably end up looking like the protobuf example - grabbing all files which match a simple pattern and passing them to a tool with a couple of flags.

    It may seem like a bit of a luxury feature, but it also means plugin scripts would be easier to read and audit. Take a look at the protobuf example again, and it's hard not to notice just how much ceremony there is to construct the invocation.


That sounds quite interesting as a future direction.

One thing I've also been wondering about is if we could add support for configure scripts. Essentially recycling my idea for edit-scripts from the manifest editing commands proposal. Unfortunately, scripts are not a good fit for that particular proposal, since they would lead to a loss of non-code information such as comments - but they'd be great for making tweaks to a package manifest at build time, or even generating them entirely at build-time.

The output of the configure script would still be a Package object, which could be serialised and available for inspection by the user, but it would be a better solution for a lot of the automation use-cases that the new CLI commands attempt to help with.

Not sure if this was just an oversight, but is there a reason the public PackageDescription API does not provide a way to specify a path: sources directory?

public static func plugin(
        name: String,
        capability: PluginCapability,
        dependencies: [Dependency] = []
    ) -> Target

I see in the current implementation, the path: argument is just being set to nil. Is there a reason why this isn’t exposed?

Question about the swiftgen example in the proposal:

// swift-tools-version: 999.0
import PackageDescription

let package = Package(
    name: "SwiftGen",
    targets: [
        /// Package plugin that tells SwiftPM how to run `swiftgen` based on
        /// the configuration file. Client targets use this plugin by listing
        /// it in their `plugins` parameter. This example uses the `prebuild`
        /// capability, to make `swiftgen` run before each build.
        /// A different example might use the `builtTool` capability, if the
        /// names of inputs and outputs could be known ahead of time.
            name: "SwiftGenPlugin",
            capability: .prebuild(),
            dependencies: ["SwiftGen"]
        /// Binary target that provides the built SwiftGen executables.
            name: "SwiftGen",
            url: "https://url/to/the/built/",
            checksum: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

Is it possible for a prebuild plugin to build and run a swift target, as opposed to a binary target? or would it be necessary for the plugin to clone its associated tool from a git repository into its output directory, and then invoke swift build / swift run on its own?

Does the current design require every build tool to learn to conditionally write out their outputs only when they have changed? Or does SwiftPM/Xcode already has file contents vs. us modification times to decide what to rebuild?

Since the current design exposes all files as inputs, it seems like the tools would get run for every build if any file was edited, and if they then generate the exact same thing, this could cascade into causing all the generated sources to be compiled again.

I’m super happy to see movement on this front. I like the proposal.

The only thing I’d like to approach differently would be plugin options. Since a typed solution seems complex we can’t expect it happening any time soon. It actually might be a very very long time until we get it. I think that’s unfortunate.

I understand the reason why you would not do it, but in my opinion, a string dictionary would be preferable to proliferation of configuration files.

This wouldn’t be foreign to Package.swift since there is already a lot of unsafe referencing going on. Products, targets and dependencies are referenced by string identifiers.

This is answered in the proposal itself [1], and depends if the plugin is a prebuild/postbuild or a buildTool:

Because prebuild and postbuild commands are run on every build, they can negatively impact build performance. Such commands should do their own dependency analysis and use caching to avoid any unnecessary work. Caches can be written to the output directory given as input to the plugin when it is invoked, as described earlier.

Command invocations emitted by plugins that have the buildTool capability can additionally specify input and output dependencies. These commands are incorporated into the build graph, and are only run when their outputs are missing or their inputs have changed.

Short version:

  • prebuild / postbuild would have to implement a caching/detection scheme to avoid redundant work
  • buildTool are only invoked when inputs change

This is a limitation of the current day build system infrastructure and can be fixed in future releases though, by adding more capabilities to spm and the xcode buildsystems. Please consider this a "first step" towards this :slight_smile:

[1] swift-evolution/ at main · apple/swift-evolution · GitHub

1 Like

Overall I think the proposal looks like a decent first step, I have some specific remarks though:

I think Target.plugin(...) should include a sources parameter at least, if not more of the parameters the other target types have. With the sources list, packages with a single, simple plugin could just use a single file at the package root, which I personally would quite like.

If we can run basically arbitrary shell commands as prebuild actions, can we use swift run on some package in a subdirectory (or even generated from scratch using the new package editing commands)? This could serve as a workaround to avoid the (I think rather unfortunate) reliance on downloading a binary from somewhere if it could be avoided, if maybe in a hackish way, until prebuild plugins depending on other targets can be supported natively.

The security section states "In addition, SwiftPM and IDEs that use libSwiftPM should run each command generated by an plugin in a sandbox". How does this work when tools from the system search path are used, which could require pretty much arbitrary files on the system to be accessible in order to function properly?

Wrt. future directions, does 'the ability for package plugin scripts to use libraries provided by other targets' imply implementing build steps using these libraries directly in the plugin, probably in form of a closure? Or are the libraries only meant to help creating commands? (I hope it's the former, as I'd rather get rid of the indirection through a command line invocation as soon as may be)

For the type-safe configuration idea, I think it would be unfortunate if every line in a plugin list ended up as .plugin(name: "X", options: XOptions(...)), instead of something like X(...) or .x(...), avoiding the repetition and string name, but I guess we'll get to that when we get to that :D

Lastly, should DiagnosticsEmitter integrate with SwiftLog in some way? This could then dovetail nicely once build steps can be defined in Swift, and depend on libraries that might be using SwiftLog already.

1 Like

At all, big 1+.

Coming from Kotlin and "its" build system Gradle (
How can I define multiple plugins, which depends on each other?
In prebuild:

  1. Init code generators
  • protobuf
  • SwiftGen
  1. Further generator
  • my custom plugin, which needs both protobuf and SwiftGen first.

Currently, plugins: [PluginUsage] = [] is a flat array, which cannot store this plugin graph/tree, like in Gradle: What is Gradle?

Doh, sorry about that, clearly skimmed too quickly (again).

1 Like

if you change the plugin type to buildTool(), it can already delegate to another package target. it appears to me that you can make any buildTool() behave like a prebuild plugin by just passing the entire targetBuildContext.sourceFiles list to the package plugin’s inputPaths.

unfortunately, the path to the swift toolchain used to kickstart the original build is not available to the plugin, so you need to delegate to an environment tool like swiftenv to point the swift command to the right toolchain.

1 Like

I am a little late but wanted to say this is a great proposal that I am a big +1 for. It fits well with Swift and SwiftPM and I feel like it leaves things open enough to extend it very well with other functionality.


SE-0303 review has concluded. While the proposed idea was accepted, the proposal was returned for revision to refine some of the API details.

Thank you to everyone for the feedback and contributions to this proposal.

Terms of Service

Privacy Policy

Cookie Policy