SwiftPM included as part of Xcode 16b3 has suddenly started adding a new "destination" folder to the directory hierarchy of build tool plugin outputs when run from the command-line. Files generated by our plugin are now going to:
.build/plugins/outputs/package-name/TargetName/destination/plugin-name/
whereas all prior versions of SwiftPM would save plugin outputs to:
.build/plugins/outputs/package-name/TargetName/plugin-name/
This breaks our plugin for all our users who updated to the latest Xcode, because we rely on setting the inputs of the plugin based on the outputs of the plugin being run by dependencies, and so we can no longer depend on ../../../other-package-name/OtherTargetName/plugin-name/expected-output-file. The plugin fails with: "error: couldn't build [plugin output] because of missing inputs: [plugin input]".
Is this an intentional change? I notice a relevant feature listed in the Xcode release notes at Xcode 16 Beta 3 Release Notes | Apple Developer Documentation
Resolved Issues in Xcode 16 Beta
* Fixed: Files generated by plugins are available in plugins run at later stages through two new APIs on `SourceModuleTarget` - `pluginGeneratedSources` and `pluginGeneratedResources`. (103808529)
And while this new pluginGeneratedSources API might be useful for supporting dependent plugin output files for Swift 6 onwards, we still need to support Swift 5.x, so we can't rely on that API being available. We need to support both Swift 5 and Swift 6 plugin outputs, and we don't see any way do this.
Interestingly, this only affects the plugin outputs for the command-line swift build. Building from Xcode itself still uses the same directory hierarchy in the DerivedData folder for the target, and so our plugin continues to work when run in Xcode. But we're worried that it will break in a future version of Xcode without warning, which will break the projects of all our users.
TL;DR
The contents and layout of .build directory is an implementation detail, which can and will differ between versions and build environments (Xcode vs swift build). You should not rely on its layout and it's subject to change without warning.
This is an intentional change that allows us to fix cross-compilation when plugins and macros are used. Without these changes we can't make SwiftPM more flexible and support more advanced use cases.
Implementation Details
Previously, all nodes in the build graph had to use the same build parameters throughout the whole graph, with no clean separation between host and target architectures. There was no namespacing, and thus plugins, macros, and their dependencies built for the target platform could overwrite build artifacts for the host platform, and vice versa, which led to nasty and subtle bugs. In some cases this behavior was non-deterministic due to randomized ordering of Set and Dictionary types during iteration.
In SwiftPM 6.0 we started separating host and target build artifacts into separate directories. This allows us to use different build parameters for host and target platforms. For example, in theory one could build macros, plugins, and their dependencies (SwiftSyntax is a common case here) for the host platform in release mode for performance, while building same dependencies in debug mode (again SwiftSyntax if also needed for the target platform) for easier debugging. These separate debug/release build configuration controls are not enabled or exposed to users yet, but at least the underlying logic behaves correctly now to allow that.
In the future I'd like to see even more separation for build artifacts, as we should be able to build different parts of the build graph for different platform at the same time (e.g. arm64 and x86_64 macOS, with some artifacts cross-compiled to Linux). For that, ad-hoc destination directory naming won't work, we'll probably have to hash build parameters used for a module and create a new directory based on that hash.
4 Likes
Thanks for the reply. For the time being, we're working around the issue by manually checking to see if the destination folder for the plugin contains "destination", and, if so, adjust the expected inputFiles parameter of Command.buildCommand accordingly. Clearly, this will break when further changes are made to the output folder layout, which will again strand our users with a broken project.
What is the recommended way for a plugin on target B to reference files that were output by dependent target A? The pluginGeneratedSources referenced in the Xcode release notes sounded promising, but it doesn't actually seem to exist in the current PackagePlugin module.