[Pre-pitch] SwiftPM Manifest based on Result Builders

I really like the @main idea, because it loosens the ordering constraints on the global scope.

Right now if we want to reuse something at several points, or to write dynamic logic, we have two options:

  1. Put it before the main init, cluttering the top of the file with functions and variables, and pushing the most relevant information down so you need to scroll to find it. (For an example, see SwiftPM’s manifest.)
  2. Mutate the package after initializing it, which often requires the extra effort of writing a search algorithm to locate the particular target that needs adjusting. (For an example, see SwiftSyntax’s manifest.)

But if you could define a function or variable at the bottom of the file, and then call it in the main portion at the top, then you could avoid the downsides of both current options.

I really like this. I can never get the syntax right in the existing format and heavily rely on copy-paste to make any significant changes. This would help a lot and I suspect also make it easier for people new to the language.

If the target dependencies could be computed automatically without too much or a performance hit that would be great. The only time I actually remember to also use a new dependency in a target is when I've just failed to do it for another one :sweat_smile:

As to the open questions:

  1. Should we use the @main or the script-like variant? @main is more structures and allows for passing context but more verbose / complicated?

I personally find the script-like variant more readable and I suspect it's going to be more approachable. Are there any benefits to using the @main approach over the script-like one?

One thing that I find quite weird about the @main-variant is that you need to come up with some name for the struct, MyPackage in the example. I always struggle naming things when I don't know how it's going to impact things down the line. It seems like there's not as good a default available here as in the script-like variant, where you can write let package = Package. Using struct Package: PackgeProtocol or something similar would be awkward.

  1. If choosing the script variant over the @main one, could we get rid of the need to let package = ... in fully declarative use cases? It is needed for the imperative use case but feels unnecessary in the fully declarative version.

That would be great and another argument in favour of the script-like variant.

  1. Is “modules” a good term to replace targets and products? Alternatively we can group them by type, e.g. “libraries”, "executables", "tests", "plugins", etc

I've come to think of "modules" as the products really so I would welcome this change. "target" is quite a "build-system-y" kind of term and "product" works best together with "build product", I feel. So "module" would be a good new term if it fits here (which it seems it does).

  1. Is “include” a good term for expressing module <> module relationship?

This one feels a bit weird. It sounds like it's including it instead of using or referencing it. Perhaps .dependsOn? Or .uses?

    Executable("MyExecutable", public: true)
      .dependsOn {
        Internal("MyDataModel")
        Internal("MyUtilities")
        External("SomeModule", from: "some-package") 
      }
  1. Is test module relationship with the module it is testing expressed as “for” and additional dependencies as “include” good terminology? should they be collapsed into one "include" section instead (more similar to the current manliest)?

Seems fine but I wonder if this could be attached to the module instead? I.e.

    Executable("MyExecutable", public: true)
      .include {
        Internal("MyDataModel")
        Internal("MyUtilities")
        External("SomeModule", from: "some-package") 
      }
      .tests {
        Test("MyExecutableTests")
      }
  1. Is “public: true” on Internal dependency the right way to express the dependency is to be exposed publicly? (this replaces product → target relationship for multi-target products)

I feel it would be better if it's more obvious this is an externally consumable module, perhaps via the module type, i.e.

    PublicExecutable(...)
      .include { ... }

although I don't love that it'd be a different type. It's not going to be possible, but public Library(...) would be a great way to declare that.

One thing that puzzles me about the example is the public: true on the nested Internal module:

    Library("MyLibrary", public: true)
      .include {
        Internal("MyDataModel", public: true)
        //                      ^^^^^^^^^^^^
        External("SomeOtherModule", from: "some-other-package") 
      }

Is that a typo or would this allow declaring a public module "MyDataModel" from within the .include clause? I think using PublicLibrary and disallowing it inside .include would be better.

  1. Should we continue to use a tools-version comment at the top of the file? Is there a better technique or name we should consider to express the manifest's schema/api version? For example, would schemaVersion or schemaAPIVersion be a more accurate name?

It's nice that the tools-version tracks the swift version and you don't need to keep track of which version of Swift brought which tools-version but the naming always felt a bit weird. Renaming that to schemaVersion or schemaAPIVersion would be clearer.

Excited to see where this is going! :tada:

4 Likes

tbh I don't think the new syntax, especially with @ main struct etc will be any easier to write by hand from scratch. Especially that IDE (and we're talking about Xcode) is not much better today with autocompletion (as we know from SwiftUI).

Given the manifest is still compiled to JSON, can we consider Package.swift.json as a supported format as well (akin to CocoaPods)? That would simplify even further generating manifests from third-party tools.

7 Likes

So it looks like there are a few distinct changes here:

  1. Unify products and targets using the "module" abstraction
  2. Use result-builders in place of arrays
  3. Use "builder style" APIs

I'll address each one separately

Unify products and targets using the "module" abstraction

This seems like a good simplification. I've already written code that mapped a single array of strings into the products/targets I wanted.

Use result builders in place of arrays:

As I see it, this is a trade-off. The main benefit is not needed to add commas and never having the compiler think you are writing .target(…).target(…). The downside is that there isn't an easy way to say "show me what values I can use here", which we can currently do by typing . and seeing what shows up in autocomplete. This is particularly noticeable in the providers example where there is no way (aside from consulting documentation) to know what other options might exist other than Apt, Yum and Brew.
Personally, I don't see this as compelling enough to make the switch. Result builders for tree-like DSLs like SwiftUI have a clearer value proposition since they get you out of bracket-hell, but for SPM I just don't see enough benefit to make the change.

Builder-style APIs

I'm not really sure I see a clear win here over just having initializers, perhaps the ability to specify initialization arguments in any order you want or split them into meaningful groupings like the following example?

.modules {
  /// Modules related to Foo
}
.dependencies {
  /// Dependencies related to Foo
}
.modules {
  /// Module related to Bar
}
.dependencies {
  /// Dependencies related to Bar
}
4 Likes

I love this (and, for the same reason, the “script-like” approach over the @main approach). This seems to me the same kind of thinking that deliberately sought to ensure that “Hello World” in Swift is the one-liner print("Hello World")—not an accident!

Build tools are complex enough, and making the manifest approachable at first glance is, in my view, no small thing. Eliminating the “target” versus “product” definitions and all other things not essential for user intent are huge.


Along the same lines, I don’t understand at first glance why executables need to be declared public: true—this seems to be needless copying of Swift’s access control model that for good reason requires public APIs to be declared as such, but who by default would assume that their package is going to publish private executables?

I’m also unclear on the distinction between an “internal public” include and an “external” (“external private”?) include as shown in the example by OP; in access control terminology, internal and public are mutually contradictory, while external visibility is what public does—so I would strongly suggest different terms to be used here. Do you mean instead “local” versus “remote”?

14 Likes

Thank you so much @tomerd for writing this pre-pitch. I can definitely feel the key issues you mentioned every time I work with manifest files. As I care about this topic as well, I have been thinking about same problems for some time now, and I came up with a couple of ideas.

I would like to propose few changes to your proposal that I think could make the package manifest even simpler. While doing about this, I have put focus on the following areas:

  • Eliminating internal string references. I think this is a very important topic, and I don't believe that we should have a major manifest redesign without addressing this issue.
  • Following the principles of progressive disclosure. A simple package should be easy to declare, while more advance features should be disclosed when needed.
  • Trying to make changes a sugar on top of the existing implementation. I didn't stick strongly to this one, but it might be close.

Here it goes.

Firstly, I think the Package type initializer should be a result builder where we define public package products. Secondly, I follow your idea of merging products and targets into a single entity.

With that, the simplest package could look like this:

Package {
    Library("MyLibrary")
}

Next, products often have dependencies. The Library type gets an initializer to declare dependencies through a result builder. Additionally, we allow declaring products within declarations of other products. Those products are dependencies of the parent product and are private by default, but can be made public, package, products if needed.

Package {
    Library("MyLibrary") {
        Library("MyUtilities")
        Library("MyDataModels", public: true)
    }
}

We also allow declaring external dependencies within declarations of other products. External packages get an API to reference their products.

Package {
    Library("MyLibrary") {
        Library("MyUtilities")
        Library("MyDataModels")
        ExternalPackage(at: "https://git-service.com/foo/some-package", upToNextMajor: "1.0.0")
            .library("SomeLibrary")
    }
}

When external packages are dependencies of multiple products, then they should be pulled out into a variable.

Package {

    let someLibrary = ExternalPackage(at: "https://git-service.com/foo/some-package", upToNextMajor: "1.0.0")
        .library("SomeLibrary")

    Library("MyLibrary") {
        Library("MyUtilities")
        Library("MyDataModels") {
            someLibrary
        }
        someLibrary
    }
}

I believe this is very important as it eliminates the need for inventing external packages name that can be used as string references, which is confusing for users and I've noticed it made a mess in the SwiftPM implementation.

The same should be done for package products that are dependencies of multiple other products.

Package {

    let myLowLevelLibrary = Library("MyLowLevelLibrary")

    Library("MyLibrary") {
        Library("MyUtilities") {
            myLowLevelLibrary
        }
        Library("MyDataModels") {
            myLowLevelLibrary
        }
    }
}

Saving an instance into a variable, instead of using string references, is a natural thing for a developer to do when passing the same thing to multiple functions. It eliminates the possibility of making a typo in one place and allows the compiler to statically evaluate correctness.

Finally, we can declare test products within products they are supposed to test, through a second result builder closure, eliminating the need for referencing the target product, which becomes an implicit dependency. Loving the application of SE-0279 here!

Package {
    Library("MyLibrary") {
        Library("MyUtilities")
        Library("MyDataModels")
    } tests {
        Tests("MyLibraryTests")
    }
}

Of course, we allow multiple test targets, as well as test targets having their own dependencies.

Package {
    Library("MyLibrary") {
        Library("MyUtilities")
        Library("MyDataModels")
    } tests {
        Tests("MyLibraryTests")
        Tests("MyLibraryOtherTests") {
            Library("MyTestUtilities")
        }
    }
}

Your full example:

import PackageDescription

let someLibrary = ExternalPackage(at: "https://git-service.com/foo/some-package", upToNextMajor: "1.0.0")
    .library("SomeModule")
let someOtherLibrary = ExternalPackage(at: "https://git-service.com/foo/some-other-package", upToNextMajor: "1.0.0")
    .library("SomeOtherModule")
let myDataModelLibrary = Library("MyDataModel", path: "custom/path", swiftSettings: "swiftSettings")

Package {
    Executable("MyExecutable") {
        myDataModelLibrary
        someLibrary
        Library("MyUtilities")
    } tests {
        Tests("MyExecutableTests") {
            Library("MyTestUtilities")
        }
    }
    Library("MyLibrary") {
        myDataModelLibrary
        someOtherLibrary
    } tests {
        Tests("MyLibrary")
    }
}
2 Likes

The "giant initializer" form is also officially recommended during the WWDC 2018 Getting to Know Swift Package Manager session (around the 13:40 mark).

I think something like .depends(on: [Module]) might read better.

A shebang-like directive is often full of hidden traps. Because it relies on a relatively inflexible structure and position not communicated to the user, just having a wrong whitespace or starting on the wrong line can render the entire manifest file uninterpretable. And because it's outside of the manifest/program's logic and the language's syntax rules, the user is not likely to understand the rules of the directive. Even if the user knows the rules, unless the error messages are able to point to the exact problem, it's not often a quick fix, because of the general subtlety and inconspicuousness of the directive's errors.

As someone who spent 2 months to make the tools-version comment line more flexible, I would like to see it deprecated for something more integrated with the manifest's logic.

Maybe we can expose ToolsVersion to the user, and let Package and related type's initializers (and static functions that work as initializers) take it as a parameter. For example, something like this:

Package(minimumSwiftVersion: v5_3) {
    .modules {
        Library("MyConcurrencyLibrary", minimumSwiftVersion: v5_5)
        Library("MyOtherLibrary")
    }
}

And if we add language feature flags to ToolsVersion, it could allow us to write something like this:

Package(minimumSwiftVersion: v5_3) {
    .modules {
        Library(
            "MyDistributedActorLibrary", 
            .when(currentToolsVersion.supportsDistributedActor)
        )
        Library(
            "MyLinuxCppWrapperLibrary",
            .when([
                .languageFeatures([.cppInterop])
                .platforms([.linux])
            ])
        )
        Library("MyOtherLibrary")
    }
}

Something like this where the tools-version requirement is part of the Package initialization can also allow all version-specific manifest files into a single one, which should have better code reuse and be easier to maintain.

4 Likes

That's fair. If I imagine how this could be improved, the obvious choice would be to use some sort of enum for products/targets/dependencies. But then we have to declare a bunch of enums and have separate code to hook them all up, and the whole thing ends up being a lot more verbose.

What we'd ideally want is for some kind of new language feature (perhaps an extension to the result builder transform), where elements of a result builder can define cases in an enum.

Other parts of the project would likely benefit from this, too. For example, the recent regex proposals mention that the cases of an Alternation/OneOf pattern should ideally be available as some kind of enum on the match result, and AFAIK that applies to both regex literals and the Pattern result-builder DSL. So that's nice. Having overlapping requirements makes it easier to scope the desired language feature and maybe implement it one day.

In the mean time, there's a much simpler option - we could just give code completion (which is part of the compiler) some specific knowledge of the Package.swift format, allowing it to auto-complete some common string values (e.g. target names). It's a hack, for sure, but it only affects this one file to give you a better editing experience.

4 Likes

My suggestion would actually be to adopt some sort of TOML format and use the various existing tools or methods to enable a simple autocomplete environment for most IDEs. Since you're already operating at a string level, suggesting string completions isn't too hard. Plus it will be much more readable than the Swift manifest, and much easier to write to for those without full IDE support.

Thinking about it more, a new builder API would be an improvement, if only to get rid of the parameter ordering restrictions I already mentioned. Given how hard it is to find the docs for those APIs outside of Xcode in package mode, any improvement to remove the need to do so would be appreciated.

3 Likes

Would this change allow you to use some kind of "typed libraries" too? I would like to prevent copy/pasting the names, especially with big package manifests.

let myDataModel = Library("MyDataModel")
Library("MyLibrary", public: true) {
    include {
        Internal(myDataModel, public: true)
        External("SomeOtherModule", from: "some-other-package") 
    } 
}

I've got mixed feeling with this, but I certainly see good usecases.

First of all, I would strongly suggest ignoring the 'added complexity' of writing @main in the package definition. If we get more features for basically free, do it. 99.9% of the people will use a premade/generated project and edit things from there. Those who don't will know what they're doing anyways.

I think this move is absolutely great, because it allows us to expand upon the Package manifest extremely easily and in a tonne of directions. The flexibility this would add to the manifest is awesome, especially considering the use of compiler directives such as #if.

My main issue would be how verbose it is for sure, but the verbosity makes this extremely easy to reason about. Adding to that the possibility of using conditions and a (future?) context in which we can derive the target platform(s) and environment we can do a lot of stuff with this.

As for the tools-version, couldn't we make multiple PackageXYZ protocols that expand upon their previous versions? Ignoring the protocol naming: protocol Swift5_7Package: Swift5_6Package { .. }

If I define a package in Swift 5.6+, I'll use Swift5_6Package, which'll be available from that Package manager forwards. If I want to support older Swift versions, I can reply upon the corresponding protocol. This can also be more easily checked for generating package indexes, since we don't use a comment as a workaround.

2 Likes

#if will remain as ill‐advised as it is now...

...because the manifest is compiled for the development device, not the target device. For example, #if os(Linux) in macOS/Linux package breaks cross‐compilation in both directions.

If you need a workaround for some condition absent from PackageDescription, it is much better to do adjustments based on the environment (e.g. if ProcessInfo.processInfo.environment["TARGETING_LINUX"] == "true"). But even then it should only be used as a last resort.

Yeah, I'm aware. I was thinking more along the lines of passing -D CONCURRENCY to opt into certain features early, which would require a dependency if adopted.

My experience was the opposite. When learning SwiftPM I remember relying heavily on long initialisers and the type system (eg enum cases) since they helped me discover the APIs available and figure out what I needed to do.

Given the current state of Swift’s tooling, Result Builders would make this “learning by discovery” incredibly challenging, encouraging copying and pasting etc. I would not know what types are available to me that I can or should use because, while code completion is getting better, it’s still extremely unreliable as an API discovery tool for anything but trivial cases like filling in the parameters of a function call or creating an enumerated value. Last I noticed, I don’t think it can yet suggest to me types I can or should use in a result builder any more than some random, exotic C function.

I love the idea of Result Builders for defining the contents of a package but the tooling isn’t there yet to help anyone but the experienced SwiftPM user or the hardcore StackOverflow/Ray Wunderlich pasteboard user.

9 Likes

Negative on this since it will introduce a lot of churn in the third-party package ecosystem (documentation, tutorials, ...), for seemingly little benefit. The Apple Silicon transition is just panning out, and it feels like there needs to be a period of relative stability, in terms of tooling.

New features will be added exclusively to the new format.

It sounds like this will force, or at least encourage, packages to adopt the new format - even if they don't strictly need the new features.

Breaking down the manifest example above shed lights on several key issues:

Are any of these issues blocking implementing any new features, or blocking fixing bugs due to an ambiguity in the current package definition? Or are they purely usability issues?
Are there examples of new, planned, features that are difficult to express cleanly with the existing giant-initializer format?

By definition, any and all features that would add a parameter to said "giant-initializer"s would be difficult to add without further increasing the complexity of the existing APIs.

To your point about forcing the community to adapt: that's just not true. Any package marking its manifest with an older version should keep working, just like they do now. Until you need the new feature you don't need to change anything, as we saw with SPM's resources feature, to name one.

2 Likes

I posted something similar in an idea above, but it was marked as a spam for 3 days. It would be interesting to get some feedback on that approach.

1 Like

Right, but the current package format has been coping with this for years now. Rather than just re-format what's already there, it would be nice to see the re-design happen at the same time as a new major feature has to be incorporated into the format. That would give more input into the design process, and also be easier to sell.

Yes, but in terms of documentation and tutorials, there's going to be a lot of churn. Take for example the swift-algorithms package README section "Adding Swift Algorithms as a Dependency". Many projects have something like this. It's now going to have to list two different ways to add the package.

2 Likes

First off, I think its very nice this is being tackled - the giant initialiser approach can definitely be improved upon. I think the discussion with static formats with escape hatches quite nicely displays why a fully dynamic format like this makes sense and I don't see a strong argument for supporting a simpler static format for a subset of use cases personally.

I'm also happy to see the conceptual merging of targets and products, which always have seemed a bit weird to me personally.

I'm a bit torn on this one, but the script-like variant is probably the more pragmatic route.

I think grouping them by type might be nicer for larger projects and more readable.

As others already mentioned, something more akin to .uses would probably be more understandable, include will always point my head back to C-header land...

I also prefer schemaVersion (or manifestSchemaVersion to make it even clearer perhaps) - rather avoid magic comments.

One could envision a world where SPM (or some other tool) could generate the appropriate markdown for that as a convenience.

2 Likes

There is a big advantage to using result builders, which is that it encodes the structure of the package into the type system at compile time, rather than building it at runtime. @Xi_Ge alludes to what this might mean:

That is – for simple packages made up only of types and constant values, we would no longer need to "run" the manifest in a sandbox. Instead, we could merely compile it and then extract all the information at build time.

More complex use cases will always need the power of running aribitrary code – but chances are this is not needed by many packages, especially many packages people tend to pull down and depend on. When a package needed the ability to run, not just compile, you could then be asked whether you trust it enough to give it this privilege, something you would probably happily grant your own code but not a copy of leftpad you just downloaded.

18 Likes