Test dependencies was removed because the implementation was broken at some point during very early stage of SwiftPM. We decided that it is better to remove the API and land it back with a proper swift-evolution process.
I am always ready to help out with code-reviews but currently, the major blocker is the lack of design discussion and evolution proposal. If you are interested in driving this, feel free to open a new thread with a draft for discussion!
Hi folks,
Let me provide some background on the original reason we used Swift as the manifest format. I'm actually just going to provide some quotes here from the original discussion, before SwiftPM was open sourced, that led to this decision. (This probably should have been included in the documentation with the original open-source release of SwiftPM!)
The manifest should be machine readable and writeable format. We envision a variety of tools that may want to inspect the contents of packages (for example, to build information for an index) or make automatic edits to the project structure. For example, when introducing a new library dependency via adding an import statement, we would like it if a tool could, after a user prompt, automatically update the manifest to specify the new dependency.
By writing the manifest in Swift, we ensure a consistent development experience across not only authoring their source code, but also their project metadata. This means developers will have a consistent environment with all of the development conveniences they expect: syntax coloring, code completion, API documentation, and formatting tools. This also ensures that new developers to Swift can focus on learning the language and its tools, not another custom package description format.
The package definition format being written in Swift is problematic for tools that wish to perform automatic updates to the file (for example, in response to a user action, or to bind to a user interface), or for situations where dealing with executable code is problematic.
To that end, the declarative package specification portion of the file is "Swift" in the same sense that "JSON is Javascript". The syntax itself is valid, executable, Swift but the tools that process it will only accept a restricted, declarative, subset of Swift which can be statically evaluated, and which can be unambiguously, automatically rewritten by editing tools. We do not intend to define a "standard" syntax for "Swift Object Notation", but we intend to accept a natural restriction of the language which only accepts literal expressions. We do intend to allow the restricted subset to take full advantage of Swift's rich type inference and literal convertible design to allow for a succinct, readable, and yet expressive syntax.
We decided to use a Swift-based format for the manifest because we believe it gives developers the best experience for working with and describing their project. The primary alternative we considered was to use a declarative format encoded in a common data format like JSON. Although that would simplify implementation of the tooling around the manifest, it has the downside that users must then learn this additional language, and the development of high quality tools for that (documentation, syntax coloring, parsing diagnostics) isn't aligned with our goal of building great tools for Swift. In contrast, using the Swift language means that we can leverage all of the work on Swift to make those tools great.
The decision to use a restricted subset of Swift for the primary package definition is because we believe it is important that common tasks which require the manifest be able to be automated or surfaced via a user interface.
Clearly this is not how the Package.swift manifest works today; you can currently write arbitrary Swift code, and we just run it through the interpreter. However, that's intended to be a short term situation, until this desired behavior is implemented. (There will have to be a backwards compatibility story for the existing freeform manifests, but that's solvable).
In the meantime, I think we should be really cautious about encouraging / enabling anything that runs counter to this goal (unless we as a community decide that this wasn't the right goal after all). It's true that you could write code inline in your manifest today that e.g. parses a YAML file, but I haven't seen anyone do that yet, and I really wouldn't want SwiftPM to encourage it by making that easier.
Beyond the reasons for using Swift described above, one might want to write real, non-declarative swift code for more complicated needs. While I think we might want to re-asses whether that's really important to support, the original idea here did allow for it. Here's what we said about that:
We intend for the declaration package definition to cover 80%+ of the use cases for modifying the convention based system. Nevertheless, there are some kinds of legitimate project structures which are difficult or cumbersome to encode in a purely declarative model. For example, designing a general purpose mechanism to cover all the ways in which users may wish to divide their source code is difficult.
Instead, we allow users to interact with the Package object using its native Swift APIs. The package declaration in a file may be followed by additional code which configures the package using a natural, imperative, Swifty API.
It is important to note that even when using this feature, the package manifest still must be declarative. That is, the only output of a manifest is a complete description of the package, which is then operated on by the package manager and build tools. For example, a manifest must not attempt to do anything to directly interact with the build output. All such interactions must go through a documented, public API vended by the package manager libraries and surfaced via the package manager tools.
This customization section will not be written in the restricted Swift subset syntax. Instead, the customization section will be clearly demarcated in the file. The leading file section up to the first '// MARK:' will be processed as part of the restricted declarative specification. All subsequent code must be honored by tools which only need to consume the output of the specification, and should be displayed by tools which present an editor view of the manifest, but should not be automatically modified. The semantics of the APIs will be specifically designed to accommodate the expected use case of editor support for the primary data with custom project-specific logic for special cases.
All tools which process the package manifest must validate that the declaration portion of the specification fits into the restricted language subset, to ensure a consistent user experience.
We decided to allow additional customization of the package via imperative code because we do not anticipate that the convention based system will be able to cover all possible uses cases. When users need to accommodate special cases, we want them to be able to do so using the most natural and expression medium, by writing Swift code. By explicitly designing in a customization system, we believe we will be able to deliver a higher quality set of core conventions -- there is an escape hatch for the special cases that allows us to focus on only delivering conventions (and core APIs) for the things that truly merit it.
We've informally referred to the purely-declarative Swift subset portion as the "top half" and the wild-west real Swift code portion as the "bottom half". The expectation was that most packages wouldn't need the "bottom half" at all (and you have an incentive to not do that, if you want to remain machine-editable and not just machine-readable).
I also expect we'd want to put APIs in the SwiftPM library interface to let anyone build tools for machine-editing manifests, and not require all machine-editing to use built-in SwiftPM commands. Once we have real machine editability, we'll unlock a whole world of custom workflows and package maintenance tools, but they'll all still speak a common lingo through the standard package manifest format.
I'd love to see us start to work on implementing all this behavior. If there's a real need for a short-term workaround for the pain we have from not having machine-editing yet, more ideas are welcome, but let's make sure that we don't do anything that compromises simplicity, clarity, and performance, or would prevent us from implementing future features we should have, like manifest caching.
One other side-thing to note:
The only reason we used a comment for that is because you have to know what version of the package manager the manifest is from before you can interpret anything else about it, so we stuck it in a comment that can easily be read in a format-agnostic way (I don't think Swift is going to obsolete the existing comment syntax 
). I'd expect that the manifest format version is the only property that concern applies to, since once we know how to interpret the manifest, we can do so using the actual language.
(not a technical contribution to this thread, just a drive-by comment from an interested reader)
What's striking about everything you just said is that it sounds a lot like a kind of Swift playground. This might not be a bad paradigm for what you have in mind.
The source code is the manifest, the output is ⌠(?) the actual build script ... (?) the build ⌠(?) the build log ⌠something like that.
There are currently 2 editors for playgrounds (Xcode and the iPad app). You envisage multiple editors on multiple platforms for manifests too. Most interesting.
TL;DR: I support the idea of "Swift Object Notation" DSL. I wish SPM made that goal more clear in the first place. I also really hope that SPM prioritizes providing tooling for programmatically editing the manifest. This is something every other dependency manager I know of has provided since day one. It's not fair that package maintainers like me are pushed to finding hacks for programmatically editing the manifest just so our users can have a decent experience adding/removing our packages to their projects.
That is definitely a huge piece of knowledge I was missing. I had no idea that the .swift file was supposed to be "Swift Object Notation" not just regular old Swift.
To be clear, I have only ever wanted a purely declarative package manifest. This entire proposal was centered around my desire to eschew the Package.swift and the arbitrary Swift code execution that comes along with it.
The regex hacking can work for the time being. It is a really terrible solution, but the alternative is worse: It is really difficult (especially as a new programmer) to be required to understand SPM, GitHub, semver, etc just to add a simple package to your project.
I still can't imagine how this will end up working. Even if you really pare down Swift into this JSON-like DSL, there are so many different ways you could write the manifest. For example:
// swift-tools-version:4.0
import PackageDescription
let package = Package(
name: "VaporApp",
dependencies: [
.package(url: "https://github.com/vapor/vapor.git", from: "3.0.0-beta"),
],
targets: [ ... ]
)
versus:
// swift-tools-version:4.0
import PackageDescription
let package = Package(
name: "VaporApp",
dependencies: [ ],
targets: [ ... ]
)
package.dependencies.append(.package(url: "https://github.com/vapor/vapor.git", from: "3.0.0-beta"))
I guess it would be fine for these subtle differences to converge into the "standard" way of defining the manifest upon serialization. But it seems like many underwater rocks will be encountered when the team actually sits down to implement this one.
That is correct, and also largely my take on this proposal.
I would rather solve this class of problems with a well-defined plugin model which would have some elements of this proposal, but in a more structured fashion. It won't be as simple to create a "plugin", but it will give SwiftPM much more knowledge about what is happening and thus ability to control the complexity the user experiences.
Can we call it a "swon" file? :)
Do you know if there an existing proposal that captures this conversation? My searches say no.
I think that's a big enough difference from what's being discussed here that a new proposal may be in order to start talking that through, if one doesn't already exist.
There isn't... I have started writing down a few minimal thoughts, but there is nothing substantial enough to be worth sharing.
Ok, well, nevermind I'll just post what I have anyway!
Looks like a solid start, @ddunbar. I'll try to read it over in more detail this evening and see if I can start to think out some ideas for implementation, to maybe help flesh that part out! Something tells me that before it becomes a full-fledged reviewable proposal, we'll want to add a path forward for backwards compatibility and a way to ensure SPM maintains a relatively simplistic user experience for users who don't necessarily want to opt-in to build-time extensibility (and the complexity that potentially comes with it)
Are PRs welcome to your forked evolution repo? :)
Thanks @ddunbar! I'm guessing this a better link for the doc, since you'll continue to update that branch: https://github.com/ddunbar/swift-evolution/blob/extensible-build-tools/proposals/NNNN-swiftpm-extensible-build-tools.md
I'm closing my proposal in favor of yours: Update NNNN-spm-static-deps.md ¡ tanner0101/swift-evolution@ccd7f5c ¡ GitHub
Sure! Although it might be good to keep as much of the discussion here as possible.
I really appreciate you bringing this issue up; active users raising their biggest pain-points is a great way to organize the community to prioritize the most pressing problems.
One of those ways involves a straightforward declaration, while the other requires a follow-on imperative statement. The latter could probably be restricted to "bottom half" non-editable manifest code.
No one's fleshed out exactly what the rules should be for a machine-editable Swift syntax, but when we were first considering this route, Chris Lattner vouched for its feasibility, and I figured he knew what he was talking about ;-)
Just to get the ideas started on a plug-in architecture, hereâs a suggestion. We add a directory âBuildSupportâ (needs bikeshedding) to the package with the following properties:
The BuildSupport package could provide the following:
As an example, letâs say you need a kernel version to make some dependency decisions. We keep the Package.swift file purely declarative as was suggested. Everything that needs more Swift than is allowed in the Package.swift (the so called bottom half) is moved into the BuildSupport package. So, to add functionality for obtaining the kernel version, the following is created:
BuildSupport/Package.swift
BuildSupport/Sources/KernelInfo/KernelInfo.swift
Now in the main Package.swift âimport KernelInfoâ can be used, which provides the required information. Of course, the declarative Package.swift file should provide enough flexibility to be able to express build decisions based upon the obtained value.
If the KernelInfo functionality is common enough, it can be moved to its own repository and just creating BuildSupport/Package.swift with a dependency to the KernelInfo package will be sufficient.
This is just a simple example of importing specific values into the Package.swift file, but plugins could actually add functionality to SPM. A plugin could for example enable support for a protobuf build stage that can be configured in Package.swift. How the SPM hooks for plugins are defined needs to be figured out, I just wanted to provide a suggestion for how to set-up a project with plugins.
Using a BuildSupport package provides a standard way to write either a simple local extension or depend on a large common plugin from the community. If you want a simple local extension youâll have to do a little bit of extra work compared to just adding regular Swift code into Package.swift, but I think this separation could help to keep the intentions of the system clear and steer people into the right direction.
I've been pondering this a bit, along with the question of how to do build configuration settings in spm.
It feels to me that the goal of keeping spm simple and sticking to the DSL as much as possible is a worthy one, so I was wondering how much could be achieved with a tiny extra bit of layering (another level of indirection 
).
Along the lines of some of the suggestions above could we add a new meta-build command which would build, then execute, a target (or product?) with a standard name, defined in the Package.swift file along with the real targets and products (maybe called prebuild or configure or... I'm not quite sure).
This target would be just another executable, written in Swift, and depending on whatever it needed to, including any tool packages, which would be fetched and built as part of the normal process of building the executable.
The executable would then be executed, and expected to output the information needed to configure and build everything else in some stable format such as JSON (swon?). This might include:
swift build calls via -Xswiftc etcI can think of some potential negatives:
I think that these problems could be overcome however.
An advantage is that it adds no complexity to the package file, and the bootstrap process to get it working is pretty simple:
swift build command, then the build executableswift build and other built executablesJust to clarify something about what I wrote above.
The products, dependencies and targets would remain entirely static and defined by the manifest.
The dynamic information returned by the "prebuild" executable would be the tools to run, and the settings to apply when running them.
This could perhaps be seen as edging towards trying to separate the "package manager", and "build tool" aspects of spm. Possibly. If you squint and don't look too closely at the details... ;)
I hacked together a small proof-of-concept: GitHub - elegantchaos/Builder: Experimental build system on top of spm.
Unless I'm misunderstanding something, that seems to move a lot of the configuration out of the Package.swift file. I think we could leverage the power of the Package.swift manifest, as the proposal that started this thread suggested, while still limiting the syntax to a machine editable subset of Swift and without using other configuration files (for most common cases).
I hacked something together as well as an example to what I suggested earlier:
https://github.com/orobio/spm-plugins
Sort of, yes.
I think the scope of my prototype drifted slightly, and is perhaps now more aimed at addressing ways to provide build settings and custom build phases.
It does feel quite clean, to me, to separate out what to build, which I think is what the Package.swift file is primarily for, from how to build it, which is arguably what the settings/phases are for. I started to see this as an actual advantage, but that may be just because my understanding of the reasons behind the current design isn't good enough.
I do see what you mean, and the conditional dependency solution in your prototype isn't directly possible with mine - which is perhaps what the original point of this thread was - sorry! Though I think you might be able to achieve something similar in a different way.
The Configure tool that mine builds returns a list of products to build, so I think you'd have to specify multiple alternate products with the different dependencies and then have it choose one. That could suffer from some sort of combinatorial explosion I guess, but on the other hand the Package.swift file would then list all the potential products.
I guess we can go in many directions with this. The suggestion I did was what came to my mind when I was reading this thread. It seems to me a straightforward, clean, and very powerful solution, but I agree that similar functionality could be accomplished in a different way.
The important question is probably, where do we want to go? And it might be good to know what roadblocks people are hitting at the moment. To be honest Iâm not someone that uses Swift daily, so I might not be the best person to say something about this.