SE-0303: Package Manager Extensible Build Tools

Does the current design require every build tool to learn to conditionally write out their outputs only when they have changed? Or does SwiftPM/Xcode already has file contents vs. us modification times to decide what to rebuild?

Since the current design exposes all files as inputs, it seems like the tools would get run for every build if any file was edited, and if they then generate the exact same thing, this could cascade into causing all the generated sources to be compiled again.

I’m super happy to see movement on this front. I like the proposal.

The only thing I’d like to approach differently would be plugin options. Since a typed solution seems complex we can’t expect it happening any time soon. It actually might be a very very long time until we get it. I think that’s unfortunate.

I understand the reason why you would not do it, but in my opinion, a string dictionary would be preferable to proliferation of configuration files.

This wouldn’t be foreign to Package.swift since there is already a lot of unsafe referencing going on. Products, targets and dependencies are referenced by string identifiers.

This is answered in the proposal itself [1], and depends if the plugin is a prebuild/postbuild or a buildTool:

Because prebuild and postbuild commands are run on every build, they can negatively impact build performance. Such commands should do their own dependency analysis and use caching to avoid any unnecessary work. Caches can be written to the output directory given as input to the plugin when it is invoked, as described earlier.

Command invocations emitted by plugins that have the buildTool capability can additionally specify input and output dependencies. These commands are incorporated into the build graph, and are only run when their outputs are missing or their inputs have changed.

Short version:

  • prebuild / postbuild would have to implement a caching/detection scheme to avoid redundant work
  • buildTool are only invoked when inputs change

This is a limitation of the current day build system infrastructure and can be fixed in future releases though, by adding more capabilities to spm and the xcode buildsystems. Please consider this a "first step" towards this :slight_smile:

[1] https://github.com/apple/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md#plugin-invocation

1 Like

Overall I think the proposal looks like a decent first step, I have some specific remarks though:

I think Target.plugin(...) should include a sources parameter at least, if not more of the parameters the other target types have. With the sources list, packages with a single, simple plugin could just use a single file at the package root, which I personally would quite like.

If we can run basically arbitrary shell commands as prebuild actions, can we use swift run on some package in a subdirectory (or even generated from scratch using the new package editing commands)? This could serve as a workaround to avoid the (I think rather unfortunate) reliance on downloading a binary from somewhere if it could be avoided, if maybe in a hackish way, until prebuild plugins depending on other targets can be supported natively.

The security section states "In addition, SwiftPM and IDEs that use libSwiftPM should run each command generated by an plugin in a sandbox". How does this work when tools from the system search path are used, which could require pretty much arbitrary files on the system to be accessible in order to function properly?

Wrt. future directions, does 'the ability for package plugin scripts to use libraries provided by other targets' imply implementing build steps using these libraries directly in the plugin, probably in form of a closure? Or are the libraries only meant to help creating commands? (I hope it's the former, as I'd rather get rid of the indirection through a command line invocation as soon as may be)

For the type-safe configuration idea, I think it would be unfortunate if every line in a plugin list ended up as .plugin(name: "X", options: XOptions(...)), instead of something like X(...) or .x(...), avoiding the repetition and string name, but I guess we'll get to that when we get to that :D

Lastly, should DiagnosticsEmitter integrate with SwiftLog in some way? This could then dovetail nicely once build steps can be defined in Swift, and depend on libraries that might be using SwiftLog already.

1 Like

At all, big 1+.

Coming from Kotlin and "its" build system Gradle (gradle.org):
How can I define multiple plugins, which depends on each other?
Eg:
In prebuild:

  1. Init code generators
  • protobuf
  • SwiftGen
  1. Further generator
  • my custom plugin, which needs both protobuf and SwiftGen first.

Currently, plugins: [PluginUsage] = [] is a flat array, which cannot store this plugin graph/tree, like in Gradle: What is Gradle?

Doh, sorry about that, clearly skimmed too quickly (again).

1 Like

if you change the plugin type to buildTool(), it can already delegate to another package target. it appears to me that you can make any buildTool() behave like a prebuild plugin by just passing the entire targetBuildContext.sourceFiles list to the package plugin’s inputPaths.

unfortunately, the path to the swift toolchain used to kickstart the original build is not available to the plugin, so you need to delegate to an environment tool like swiftenv to point the swift command to the right toolchain.

1 Like

I am a little late but wanted to say this is a great proposal that I am a big +1 for. It fits well with Swift and SwiftPM and I feel like it leaves things open enough to extend it very well with other functionality.

2 Likes

SE-0303 review has concluded. While the proposed idea was accepted, the proposal was returned for revision to refine some of the API details.

Thank you to everyone for the feedback and contributions to this proposal.

2 Likes