For in-build plugins adding new files to the Target's source is expensive (plugin and tool-commands all run fresh)

Development Package Manager
, ,
1

My in build plugin runs from scratch every time a new file is created in Target's source directory even if it doesn't affect the plugin's input list.

About the plugin: Using swift package init --type build-tool-plugin as a starting point I made a build plugin and a CLI that imports it. The build plugin creates a Command for every text file in the Source folder. The tool in the command just creates a .swift file of the same name. Super minimum viable build plugin.

The trick is back in the CLI that imports the plugin (swift-tools-version: 5.9, vanilla swift run):

  • If I add a .swift file to the Source folder the tool still reruns for each and every text file even though .swift files shouldn't generate new commands.
  • If I just edit one of the text files the tool only runs for that text file as expected.
  • If I add a file OUTSIDE of the Source folder like a README.md the tools do not rerun.
  • If I ad a README.md in an EXCLUDED folder inside the source directory, the tools rerun.

My understanding was that in-build tools should only run if their inputs have changed or the outputs are missing. My inputs have not changed. Presumably the outputs haven't been zotted either?

I can understand rechecking when the Source directory changes, I'd rather rerun by mistake than not run. That said if it was A LOT of files / expensive processes it'd be annoying.

If I wanted to see if I could change the behavior what should I do next?

  • A: Nothing - This is the expected behavior / known issue and unavoidable / changed in 5.10, etc. :+1: Carry on.
  • B: Keep trying to twiddle the code -
    • For example: Presumably the number of .nones in the returned [Command] changes, but I tried filtering on text files in filesFromDirectory and that didn't seem to have an effect. Also presumably .nones get compactMapped out? I can try again if that's the ticket.
  • C: Look into if any of the options/flags on swift build or swift run could impact this.
  • D: File an issue.
  • E: Something else.

Thanks so much for this incredibly powerful system. Cheers.

Side Note: In an Xcode project (Just the default multi-platform template, Xcode 15.2, basic built for MacOS and iOS 15 in the simulator) the build plugin just seems to run every time, no matter what but I care less about Xcode projects over churning.

Test Projects

import PackagePlugin
import Foundation

@main
struct FruitStoreBuild: BuildToolPlugin {
    /// Entry point for creating build commands for targets in Swift packages.
    func createBuildCommands(context: PluginContext, target: Target) async throws -> [Command] {

        // Find the code generator tool to run (This is what we named our actual one.).
        let generatorTool = try context.tool(named: "my-code-generator")
        
        // Still ensures that the target is a source module.
        guard let target = target as? SourceModuleTarget else { return [] }

        let filesToProcess = try filesFromDirectory(path: target.directory, shallow: false)
        
        // Construct a build command for each source file with a particular suffix.
        return filesToProcess.compactMap {
            createBuildCommand(for: $0, in: context.pluginWorkDirectory, with: generatorTool.path)
        }
    }
}

#if canImport(XcodeProjectPlugin)
import XcodeProjectPlugin

extension FruitStoreBuild: XcodeBuildToolPlugin {
    // Entry point for creating build commands for targets in Xcode projects.
    func createBuildCommands(context: XcodePluginContext, target: XcodeTarget) throws -> [Command] {
        // Find the code generator tool to run (replace this with the actual one).
        let generatorTool = try context.tool(named: "my-code-generator")

        // Construct a build command for each source file with a particular suffix.
        return target.inputFiles.map(\.path).compactMap {
            createBuildCommand(for: $0, in: context.pluginWorkDirectory, with: generatorTool.path)
        }
    }
}

#endif

extension FruitStoreBuild {
    /// Shared function that returns a configured build command if the input files is one that should be processed.
    func createBuildCommand(for inputPath: Path, in outputDirectoryPath: Path, with generatorToolPath: Path) -> Command? {
        // Skip any file that doesn't have the extension we're looking for (replace this with the actual one).
        guard inputPath.extension == "txt" else { return .none }
        
        print("PROOF OF PLUGIN LIFE from createBuildCommand")

        // Return a command that will run during the build to generate the output file.
        let inputName = inputPath.lastComponent
        let outputName = inputPath.stem + ".swift"
        let outputPath = outputDirectoryPath.appending(outputName)
        return .buildCommand(
            displayName: "------------ Generating \(outputName) from \(inputName) ------------",
            executable: generatorToolPath,
            arguments: ["\(inputPath)", "-o", "\(outputPath)"],
            inputFiles: [inputPath],
            outputFiles: [outputPath]
        )
    }
}

func filesFromDirectory(path providedPath:Path, shallow:Bool = true) throws -> [Path] {
    if shallow {
        return try FileManager.default.contentsOfDirectory(atPath: providedPath.string).compactMap { fileName in
            providedPath.appending([fileName])
        }
    } else {
        let dataDirectoryURL = URL(fileURLWithPath: providedPath.string, isDirectory: true)
        var allFiles = [Path?]()
        let enumerator = FileManager.default.enumerator(at: dataDirectoryURL, includingPropertiesForKeys: [.isRegularFileKey], options: [.skipsHiddenFiles, .skipsPackageDescendants])
        
        while let fileURL = enumerator?.nextObject() as? URL {
            if let regularFileCheck = try fileURL.resourceValues(forKeys:[.isRegularFileKey]).isRegularFile, regularFileCheck == true {
                allFiles.append((Path(fileURL.path())))
            }
        }
        return allFiles.compactMap({$0})
    }
}

import Foundation

let arguments = ProcessInfo().arguments
if arguments.count < 4 {
    print("missing arguments")
}

// print("ARGUMENTS")

// arguments.forEach {
//     print($0)
// }

let (input, output) = (arguments[1], arguments[3])

//Added for ease of scanning for our output.
print("FIIIIIIIIIIIIIINNNNNNNNNDDDMMMEEEEEEEEEEEEEEEEE")
print("from MyBuildPluginTool:", input)
print("from MyBuildPluginTool:", output)
var outputURL = URL(fileURLWithPath: output)

let contentsOfFile = "//nothing of importance"

try contentsOfFile.write(to: outputURL, atomically: true, encoding: .utf8)
2 Likes
2

Just wanted to say a million thanks for this code example!!! Pulling my hair out getting this to work. Now I have a working plugin for my TypeLocXC project. Thank you.

1 Like