Spm static dependencies

TL;DR: I support the idea of “Swift Object Notation” DSL. I wish SPM made that goal more clear in the first place. I also really hope that SPM prioritizes providing tooling for programmatically editing the manifest. This is something every other dependency manager I know of has provided since day one. It’s not fair that package maintainers like me are pushed to finding hacks for programmatically editing the manifest just so our users can have a decent experience adding/removing our packages to their projects.

That is definitely a huge piece of knowledge I was missing. I had no idea that the .swift file was supposed to be “Swift Object Notation” not just regular old Swift.

To be clear, I have only ever wanted a purely declarative package manifest. This entire proposal was centered around my desire to eschew the Package.swift and the arbitrary Swift code execution that comes along with it.

The regex hacking can work for the time being. It is a really terrible solution, but the alternative is worse: It is really difficult (especially as a new programmer) to be required to understand SPM, GitHub, semver, etc just to add a simple package to your project.

I still can’t imagine how this will end up working. Even if you really pare down Swift into this JSON-like DSL, there are so many different ways you could write the manifest. For example:

// swift-tools-version:4.0
import PackageDescription

let package = Package(
    name: "VaporApp",
    dependencies: [
        .package(url: "https://github.com/vapor/vapor.git", from: "3.0.0-beta"),
    ],
    targets: [ ... ]
)

versus:

// swift-tools-version:4.0
import PackageDescription

let package = Package(
    name: "VaporApp",
    dependencies: [ ],
    targets: [ ... ]
)
package.dependencies.append(.package(url: "https://github.com/vapor/vapor.git", from: "3.0.0-beta"))

I guess it would be fine for these subtle differences to converge into the “standard” way of defining the manifest upon serialization. But it seems like many underwater rocks will be encountered when the team actually sits down to implement this one.

That is correct, and also largely my take on this proposal.

I would rather solve this class of problems with a well-defined plugin model which would have some elements of this proposal, but in a more structured fashion. It won’t be as simple to create a “plugin”, but it will give SwiftPM much more knowledge about what is happening and thus ability to control the complexity the user experiences.

4 Likes

Can we call it a “swon” file? :)

1 Like

Do you know if there an existing proposal that captures this conversation? My searches say no.

I think that’s a big enough difference from what’s being discussed here that a new proposal may be in order to start talking that through, if one doesn’t already exist.

There isn’t… I have started writing down a few minimal thoughts, but there is nothing substantial enough to be worth sharing.

Ok, well, nevermind I’ll just post what I have anyway!

3 Likes

Looks like a solid start, @ddunbar. I’ll try to read it over in more detail this evening and see if I can start to think out some ideas for implementation, to maybe help flesh that part out! Something tells me that before it becomes a full-fledged reviewable proposal, we’ll want to add a path forward for backwards compatibility and a way to ensure SPM maintains a relatively simplistic user experience for users who don’t necessarily want to opt-in to build-time extensibility (and the complexity that potentially comes with it)

Are PRs welcome to your forked evolution repo? :)

Thanks @ddunbar! I’m guessing this a better link for the doc, since you’ll continue to update that branch: https://github.com/ddunbar/swift-evolution/blob/extensible-build-tools/proposals/NNNN-swiftpm-extensible-build-tools.md

I’m closing my proposal in favor of yours: https://github.com/tanner0101/swift-evolution/commit/ccd7f5c4e851add6e37f3bab2c6efc386b0ea9a6

Sure! Although it might be good to keep as much of the discussion here as possible.

I really appreciate you bringing this issue up; active users raising their biggest pain-points is a great way to organize the community to prioritize the most pressing problems.

One of those ways involves a straightforward declaration, while the other requires a follow-on imperative statement. The latter could probably be restricted to “bottom half” non-editable manifest code.

No one’s fleshed out exactly what the rules should be for a machine-editable Swift syntax, but when we were first considering this route, Chris Lattner vouched for its feasibility, and I figured he knew what he was talking about ;-)

Just to get the ideas started on a plug-in architecture, here’s a suggestion. We add a directory ‘BuildSupport’ (needs bikeshedding) to the package with the following properties:

  • The BuildSupport directory is a full Swift package that is built before the main Package.swift is processed.
  • It provides plugins that can hook into SPM to provide additional functionality for building the main package.
  • Naturally, the BuildSupport package is optional. Most trivial packages shouldn’t need it.

The BuildSupport package could provide the following:

  • Simple modules that can be imported in Package.swift.
  • Full plugins that hook into SPM to provide additional build functionality.
  • Other tools used during the build.

As an example, let’s say you need a kernel version to make some dependency decisions. We keep the Package.swift file purely declarative as was suggested. Everything that needs more Swift than is allowed in the Package.swift (the so called bottom half) is moved into the BuildSupport package. So, to add functionality for obtaining the kernel version, the following is created:
BuildSupport/Package.swift
BuildSupport/Sources/KernelInfo/KernelInfo.swift

Now in the main Package.swift ’import KernelInfo’ can be used, which provides the required information. Of course, the declarative Package.swift file should provide enough flexibility to be able to express build decisions based upon the obtained value.

If the KernelInfo functionality is common enough, it can be moved to its own repository and just creating BuildSupport/Package.swift with a dependency to the KernelInfo package will be sufficient.

This is just a simple example of importing specific values into the Package.swift file, but plugins could actually add functionality to SPM. A plugin could for example enable support for a protobuf build stage that can be configured in Package.swift. How the SPM hooks for plugins are defined needs to be figured out, I just wanted to provide a suggestion for how to set-up a project with plugins.

Using a BuildSupport package provides a standard way to write either a simple local extension or depend on a large common plugin from the community. If you want a simple local extension you’ll have to do a little bit of extra work compared to just adding regular Swift code into Package.swift, but I think this separation could help to keep the intentions of the system clear and steer people into the right direction.

I’ve been pondering this a bit, along with the question of how to do build configuration settings in spm.

It feels to me that the goal of keeping spm simple and sticking to the DSL as much as possible is a worthy one, so I was wondering how much could be achieved with a tiny extra bit of layering (another level of indirection :grin:).

Along the lines of some of the suggestions above could we add a new meta-build command which would build, then execute, a target (or product?) with a standard name, defined in the Package.swift file along with the real targets and products (maybe called prebuild or configure or… I’m not quite sure).

This target would be just another executable, written in Swift, and depending on whatever it needed to, including any tool packages, which would be fetched and built as part of the normal process of building the executable.

The executable would then be executed, and expected to output the information needed to configure and build everything else in some stable format such as JSON (swon?). This might include:

  • settings to apply to subsequent swift build calls via -Xswiftc etc
  • tools to build & run to process the source before building (protobuf, mogenerator etc…)
  • the product (and configuration?) to actually build
  • tools to build & run to process & package the build output

I can think of some potential negatives:

  • it’s another thing to build & run
  • the configuration information is not declaratively available in the package, and hence cannot itself be manipulated directly by tools
  • perhaps a bit of recursive configuration problem for the tools and the prebuild target itself?
  • arguably we’re just moving some of the complexity elsewhere, by slight-of-hand

I think that these problems could be overcome however.

An advantage is that it adds no complexity to the package file, and the bootstrap process to get it working is pretty simple:

  • run the existing swift build command, then the build executable
  • parse a dictionary of results
  • invoke swift build and other built executables

Just to clarify something about what I wrote above.

The products, dependencies and targets would remain entirely static and defined by the manifest.

The dynamic information returned by the “prebuild” executable would be the tools to run, and the settings to apply when running them.

This could perhaps be seen as edging towards trying to separate the “package manager”, and “build tool” aspects of spm. Possibly. If you squint and don’t look too closely at the details… ;)

I hacked together a small proof-of-concept: https://github.com/elegantchaos/Builder

Unless I'm misunderstanding something, that seems to move a lot of the configuration out of the Package.swift file. I think we could leverage the power of the Package.swift manifest, as the proposal that started this thread suggested, while still limiting the syntax to a machine editable subset of Swift and without using other configuration files (for most common cases).

I hacked something together as well as an example to what I suggested earlier:

Sort of, yes.

I think the scope of my prototype drifted slightly, and is perhaps now more aimed at addressing ways to provide build settings and custom build phases.

It does feel quite clean, to me, to separate out what to build, which I think is what the Package.swift file is primarily for, from how to build it, which is arguably what the settings/phases are for. I started to see this as an actual advantage, but that may be just because my understanding of the reasons behind the current design isn’t good enough.

I do see what you mean, and the conditional dependency solution in your prototype isn’t directly possible with mine - which is perhaps what the original point of this thread was - sorry! Though I think you might be able to achieve something similar in a different way.

The Configure tool that mine builds returns a list of products to build, so I think you’d have to specify multiple alternate products with the different dependencies and then have it choose one. That could suffer from some sort of combinatorial explosion I guess, but on the other hand the Package.swift file would then list all the potential products.

I guess we can go in many directions with this. The suggestion I did was what came to my mind when I was reading this thread. It seems to me a straightforward, clean, and very powerful solution, but I agree that similar functionality could be accomplished in a different way.

The important question is probably, where do we want to go? And it might be good to know what roadblocks people are hitting at the moment. To be honest I’m not someone that uses Swift daily, so I might not be the best person to say something about this.

1 Like

I have left this on the back burner for way too long but I made a stab at trying to fill in the blank spaces of the proposal that was already in progress: https://gist.github.com/bppr/bdbb9bb4c73a50cf9f52e6c15595c1a2

edit: we may have to adjust some of the nomenclature to avoid conflicting with the SwiftPM proposal for workspaces. I wrote this before that, but… I yield courteously

1 Like

Hi folks,

This is a great discussion; thank you everyone who’s been thinking about (and prototyping!) these problems.

In seems like there are a few issues being wrapped up together here:

  • “What to build”, i.e. changing how the package model is defined
  • How to make behavior conditional on some condition
  • “How to build it”, i.e. defining custom logic for running things SwiftPM can’t natively do
  • How to make non-native tools available on your system

With whatever solution we provide to any of these problems, I think it’s very important to make sure that SwiftPM maintains a full and precise dependency specification. This allows us to build features like smart caching and precise incremental builds, which will be very important for build performance (as well as incremental build correctness!).

As much as possible, I think it’s desirable to keep the core specification of what to build in the Package.swift manifest, without needing additional “levels” to the build. We should have a strong build settings model (though that deserves its own proposal), and we can have conditional build settings there as needed. And machine-editable manifests will eventually make it much easier to maintain that “how to build” without needing package-specific magic to allow maintainers to specify it in their own preferred way.

Telling SwiftPM how to build things that it doesn’t natively know about is, I think, the point of Daniel’s original (skeletal) plug-in proposal. While having native knowledge of common tools is great, there are always going to be things SwiftPM doesn’t know how to do, like running less-common tools and bits of build logic that we don’t want specific native support for. We should add support for this sort of custom logic, but when we do, we should make sure that we’re still able to understand what that logic is doing, to maintain that precise dependency specification. That means that we need to know what the inputs and products are from any custom step, and ideally would be able to carefully control its environment to only expose it to declared inputs (e.g. using sandboxing), though that last step is an optional enhancement.

For how to get tools on your system, I think it’s desirable to vend them via packages that are just part of your normal dependency graph, and subject to the normal constraints around how dependencies get resolved. Ideally, most common custom logic can be distributed as its own package that any client can use, and the precise configuration needed to use it can then be specified using the normal modeling in Package.swift.

Per-package scripts that you don’t want to distribute as their own package should probably be possible at some point, but I suspect that they’re a lot more error-prone (in terms of properly declaring inputs/outputs and not depending on state in the package manager that we may not want to commit to, like the current build directory layout). Unless this is really blocking people from doing what they need to do, I’d suggest we defer freeform scripts if possible until we’re able to better control what they can do and make sure that we won’t break them as SwiftPM evolves.

Ankit and Daniel have been working on fleshing out a proposal in this space, and while I haven’t read it in detail yet, I think it hits all of these criteria. I’d suggest that he start a new thread for that discussion, since this thread has veered pretty far afield from the original topic.

4 Likes

Thanks Rick!

Here is the proposal: Package Manager Extensible Build Tools

1 Like