Pitch: SwiftPM Extensible Build Tools

[tomerd]

@georgebarnett

I'd like this area expanded upon as I don't think configuration files work well when configuration is different per-target (as the configuration format must be rich enough to support per-target option or the author of the package extension target must define some convention on naming configuration files in a per-target way

@AliSoftware

I get (and share) the frustrations of not being able to pass options to the extensions in the first iteration. I think I'd still prefer to have the pitch go forward and the feature implemented first even without support for options in its first version, as we could always document for adopters of our extensions to use JSON/YAML configuration files that could follow a specific standard (eg swiftgen-$targetName.yml or similar)

For configuration, we need to weight the tradeoffs between three main options:

1. Typesafe configuration directly in the package manifest itself
2. Non typesafe configuration (e.g. some kind of unstructured dictionary) directly in the package manifest itself
3. External configuration file(s) defined by each extension if/when needed

The sentiment in the thread, which I personally share, is that option #1 would give SwiftPM users the best experience and most safety. The technical challenge with achieving such experience is that the package manifest is Swift code, and as such the SwiftPM/Xcode build systems would need to pre-compile the extensions before the package manifest can make use of types the extensions define. Some build tools e.g. gradle, sbt, there is a "build for a build" for exactly this reason. Since neither SwiftPM's nor Xcode's build system are set up for this today, this could be a pretty significant undertaking, so the main downside with choosing this option is that it will delay the extensibility feature in a significant way.

Option #2 - non typesafe configuration in the manifest - is nice in that it keeps the configuration close to the plugin binding in the manifest, and does not suffer from the chicken and egg issue of option #1. The downside is that it is not type safe and as such could lead to subpar and frustrating user experience when making configuration mistakes. It would also likely need to be limited to a somewhat flat list of key values which is not very flexible. The maven build tool takes this approach, and suffers for these shortcomings, but is still fairly successful in it's ecosystem. We should also consider the evolution of the Swift packages ecosystem: Since we want to get to option #1 eventually, the transition from non-typesafe configuration in the manifest to typesafe one could be painful given that SwiftPM must support older versions of the manifest and as such would need to support a "mixed mode" for a long time.

Option #3 - external configuration files - is nice in that it gives the extension author full flexibility in how to design their configuration, and does not suffer from the chicken and egg issue of option #1. The downsides have already been brought up in this thread earlier but the tl;dr is that configuration files are arm-length from the manifest so less intuitive, requires the extension author to advertise information about how to use them outside of the manifest validation, and it also puts more responsibility on the extension author for something that SwiftPM could theoretically provide as a utility for those authors. When it comes to evolution of the proposal / feature, external files may have an advantage over non-typesafe configuration because the extensions can evolve to accept manifest based configuration (option #1) when such is available in addition to configuration files, which makes for a natural transition. With regards to the location and structure of the external configuration - for some plugins it may be enough to have single file at the root of the project, and some may require one per target which can be achieved by naming convention or by putting one at the root of each target.