[Pre-pitch] SwiftPM Manifest based on Result Builders

Development Package Manager
1

Background and motivation

Swift Package Manager (here after SwiftPM) was released when Swift was made open source in 2016. SwiftPM uses a file named Package.swift with which users describe the package's source structure, the build artifacts such as any executables or libraries the build produces, and any dependencies on other packages.

SwiftPM’s manifest is a Swift program (a script of sorts) which SwiftPM builds and executes as a separate process in a security sandbox to generate a static data model representing the desired package configuration. Currently, the static representation is based on JSON and the exact format of that JSON is an internal implementation detail. The JSON model is later deserialized and loaded into the parent process memory space, driving SwiftPM’s workflows such as the dependency resolution, build, test , etc.

SwiftPM’s manifest is a simple Swift script, with several requirements and constraints::

  1. The manifest must import the PackageDescription module to gain access to the relevant APIs required to construct the package model.
  2. The manifest must construct a package instance of type Package which is defined by the PackageDescription module.
  3. The manifest has access to Swift’s standard library, Dispatch and Foundation but cannot use any other packages or libraries.
  4. The security sandbox limits access to network and file system.

Consider the following example of a manifest file:

import PackageDescription

let package = Package(
    name: "MyPackage",
    products: [
        .executable(
            name: "MyExecutable", 
            target: ["MyExecutable"]
        ),
       .library(
            name: "MyLibrary", 
            targets: ["MyLibrary", "MyDataModel"]
        ),       
    ],
    dependencies: [
        .package(url: "https://git-service.com/foo/some-package", from: "1.0.0"),
        .package(url: "https://git-service.com/foo/some-other-package", from: "1.0.0"),
    ],
    targets: [
        .executableTarget(
            name: "MyExecutable",
            dependencies: [
                "MyDataModel",
                "MyUtilities",
                .product(name: "SomeModule", package: "some-package")                
            ]
        ),
        .target(
            name: "MyLibrary",
            dependencies: [
                "MyDataModel",                
                .product(name: "SomeOtherModule", package: "some-other-package")
            ]
        ),
        .target(
            name: "MyDataModel",
            dependencies: []
        ),
        .target(
            name: "MyUtilities",
            dependencies: []
        ),        
        .target(
            name: "MyTestUtilities",
            dependencies: []
        ),
        testTarget(
            name: "MyExecutableTests",
            dependencies: ["MyExecutable", "MyTestUtilities"]
        ),        
        testTarget(
            name: "MyLibraryTests",
            dependencies: ["MyLibrary"]
        )
    ]
)

Breaking down the manifest example above shed lights on several key issues:

  1. The manifest defines 2 publicly available build artifacts, an executable named MyExecutable and a library named MyLibrary. To define these artifacts, the manifest requires definition of two abstractions: a Target and a Product, having the Target representing a source module and the Product representing a public build artifact. Worth noting that in many cases, each Product points to exactly one Target.
  2. The manifest also defines 3 internal targets (modules) that are not exposed outside the package: MyDataModel, MyUtilities and MyTestUtilities. The MyLibrary product (public build artifact) exports both the MyLibrary and MyDataModel targets (modules).
  3. Finally the manifest defines 2 test targets that are not exposed outside the package: MyExecutableTests and MyLibraryTests, designed to test the MyExecutable and MyLibrary targets respectively. The MyExecutableTests target also depends on the MyTestUtilities target (an internal module).
  4. The manifest defines dependencies on two external packages: https://git-service.com/foo/some-package and https://git-service.com/foo/some-other-package. some-package is used by the MyExecutable target, while some-other-package is used by the MyLibrary target.
  5. The manifest is a “giant initializer expression" for constructing a Package instance. The impact of this design is that it is very hard to get the manifest right when typing it from scratch as mistakes are syntax errors that can be hard to decipher. For the same reason, auto-complete tools struggle with such expression and IDEs offer little help to that end. In practice, the most effective way to write a manifest is to copy/paste a working one and then change it, which is indicative of the usability problem.
  6. Worth mentioning that the Package instance can be constructed empty, and then built up line by line imperatively which helps mitigate the usability problem mentioned above. That said, the “giant initializer” is the more common form of the manifest, arguably because it looks to be declarative and as such easier to read at a glance.
  7. The “giant initializer” design also prevents the use of conditionals such as #if which can be useful in some cases, for example multi-platform support (some caveats apply about this usefulness).
  8. The “giant initializer” API makes the manifest fairly verbose.

Proposed solution

At the time the manifest was designed Swift’s Result Builders were not part of the language, leading to the “giant initializer” design mentioned above. With Result Builders in place, we can potentially use this technique to solve some of the problems mentioned above.

Consider the following example of a manifest file, replicating the example above with a Result Builders based API:

Alternative 1, script based:

import PackageManifest

let package = Package()
  .modules {
    Executable("MyExecutable", public: true)
      .include {
        Internal("MyDataModel")
        Internal("MyUtilities")
        External("SomeModule", from: "some-package") 
      }
    Library("MyLibrary", public: true)
      .include {
        Internal("MyDataModel", public: true)
        External("SomeOtherModule", from: "some-other-package") 
      }
    Library("MyDataModel")
    Library("MyUtilities")
    Library("MyTestUtilities")
    Test("MyExecutableTests", for: "MyExecutable")
      .include("MyTestUtilities")
    Test("MyLibraryTests", for: "MyLibrary")
  }
  .dependencies {
    SourceControl(at: "https://git-service.com/foo/some-package", upToNextMajor: "1.0.0")
    SourceControl(at: "https://git-service.com/foo/some-other-package", upToNextMajor: "1.0.0")    
  }

Breaking down the manifest example above shed lights on several changes:

  1. The Package instance is constructed gradually using Result Builders, making it significantly easier to type a manifest from scratch and allowing IDEs to produce useful auto-complete hints.
  2. Target and module are collapsed into a single “module” entity. Modules that need to be exposed publicly (as build artifacts) use a “public” attribute. Dependencies between modules (internal or external) is expressed as “include”.
  3. Test module relationship with the module it is testing is expressed as “for”, and any additional dependencies is expressed as “include”.
  4. Use of #if conditional is possible.
  5. The manifest is more concise.

Using @main

A slight variation on this could be to define a Package protocol similar to SwiftUI’s App protocol and use a @main entry-point to define the package:

import PackageManifest

@main
struct MyPackage: Package {
  func init(context: Context) {
    self.modules = some Modules {
      ... (same as above)  
    }
    self.dependencies = some Dependencies {
      ... (same as above)    
    }
  }
}

public protocol Package {
  @ModulesBuilder var modules: Modules { get }
  @DependenciesBuilder var dependencies: Dependencies { get }

  init(context: Context)
}

Additional modifiers

The above example focused on key constructs of package: dependencies and modules. Real-world modules tend to be more complex than the example above and include large amount of settings that are applicable to specific module types. Using Result Builders based API allows us to define module specific type modifiers, for example:

let package = Package()
    .modules {
        Executable("MyProgram", public: true)
            .customPath("custom/path")
            .swiftSettings("swiftSettings")
            .include {
                Internal("CSQLite3")
            }
        System("CSQLite3")
            .providers {
                Apt("libsqlite3-dev")
                Yum("sqlite-devel")
                Brew("sqlite3")
            }
    }

This example demonstrates the use of Result Builders modifiers to apply settings. The modifiers are module type specific, for example an Executable module has different set of modifiers (e.g. customPath, swiftSettings) than a System module (e.g. providers). The full list of modifiers matches the existing capabilities in SwiftPM’s current manifest configuration options.

Impact on existing packages

The new format would be supported side-by-side to the existing one for a long transition period. Users would be encouraged (with warnings) to use the new format when updating a manifest to a tools-version greater than the one in which the new format was introduced. New features will be added exclusively to the new format.

Open questions

  1. Should we use the @main or the script-like variant? @main is more structures and allows for passing context but more verbose / complicated?
  2. If choosing the script variant over the @main one, could we get rid of the need to let package = ... in fully declarative use cases? It is needed for the imperative use case but feels unnecessary in the fully declarative version.
  3. Is “modules” a good term to replace targets and products? Alternatively we can group them by type, e.g. “libraries”, "executables", "tests", "plugins", etc
  4. Is “include” a good term for expressing module <> module relationship?
  5. Is test module relationship with the module it is testing expressed as “for” and additional dependencies as “include” good terminology? should they be collapsed into one "include" section instead (more similar to the current manliest)?
  6. Is “public: true” on Internal dependency the right way to express the dependency is to be exposed publicly? (this replaces product → target relationship for multi-target products)
  7. Should we continue to use a tools-version comment at the top of the file? Is there a better technique or name we should consider to express the manifest's schema/api version? For example, would schemaVersion or schemaAPIVersion be a more accurate name?

Reference implementation

Diff view: Comparing apple:main...tomerd:feature/manifest2-poc · apple/swift-package-manager · GitHub

Future directions

Orthogonal to this proposal, an interesting usability improvement could be to remove the need to explicitly declare dependencies on targets. Target dependencies are used to compute the minimal dependency closure and avoid pulling non-required dependencies (see SE-0226). Declaring them explicitly as done today makes the computation easier, but puts the burden on the user, and leads to a verbose and non-esthetic manifest syntax. Since SwiftPM has access to the package and dependencies sources it could potentially deduce this information from the code itself by parsing the import statements.

Alternative considered

The most obvious alternative is to use a static declarative format such as JSON, YAML or TOML to describe packages. The main advantage of using static declarative format is that it is faster and safer to load - Swift based manifest must be compiled and run to extract the underlying information, while a static file can be parsed which is faster and safer. On the flip side, a sufficiently advanced package needs to go beyond what is possible in a static deceleration and often tap into the programatic capabilities offered by the manifest API. This is true not only in the Swift ecosystem but also in other ecosystems, where some ecosystems use a fully static declarative format with escape hatches for things like running scripts, while others use a programatic API / DSL similar to SwiftPM’s manifest API, for example:

From a broad security and performance perspective, escape hatches such as custom scripts and plugins are equivalent to a manifest based on a programatic API or DSL, since the system need to compile and run code in either case. As such, we do not feel that at this time transitioning to a static declarative format with escape hatches would give a significant benefit.

One could argue "why not both?": Static declarative format for simple packages and programatic API for advanced ones. The benefit of such approach is that at least some packages can benefit from better performance and simplified security profile, and the implications are that users would need to learn 2 manifest formats (and languages) and a maintenance burden that can lead to asymmetry in features and quality.

45 Likes
2

Love the idea of more declarative and builder-oriented APIs! Because the proposed APIs resemble static declarative format such as JSON, probably we could extract the data model from the Swift source code without running the code to be more secure? at least for simple package structures?

5 Likes
3

This is great, very excited about -- the manifests are really due for a refresh :slight_smile: I'll dig deeper into it soon.

Quick question (or nitpick?) while we're early here...

I think it is a bit weird that the let = Package() is not a builder, i.e. we can make it also the same style:

let package = Package {
  Modules {  
    Library(...)
  }
}

rather than


let package = Package()
  .modules {

That's doable AFAIR as I was doing some result builders of my own. It keeps the same style which is nice IMHO.

32 Likes
4

Maybe there's InternalModules { ... } and PublicModules { ... } (or exported modules?) to address that by declaring them in a diff block.

In practice this could just do:

var modules = <get modules from the result builder>
// inside PublicModules we'd do:
modules = modules.map { $0.public = true }
5

Interesting. I agree that Package should also be a builder, rather than using SwiftUI-style modifiers for each structural group.

Also, since you can only declare one package per manifest, would we be able to have a hidden global (within the PackageManifest module) which is assigned to in buildFinalResult?

// Within PackageManifest module

public var package: Package?

@resultBuilder
struct Package {

  @discardableResult
  static func buildFinalResult(_ p: Package) -> Package {
    package = p
  }
}

That would eliminate the need the declare a variable named "package", and allow a more declarative format:

import PackageManifest

Package {
  ...
}

It's a small thing but I think it's more visually pleasing.

19 Likes
6

I am in favor of dropping the tools-version comment for an explicitly set variable, much like we were with Package

In terms of which route to go, @main vs. just Result Builders, the question I have is which leaves the door open to potentially importing data from dependency package manifests, such as custom configurations?

Example of what I'm invisioning

// Dependency/Package.swift
import PackageDescription

let schemaVersion = 5.6

enum Configuration: PackageConfiguration { // protocol PackageConfiguration: CaseIterable
  case super
  // details left to another proposal
}

Package(availableConfigurations: Configuration.allCases) {
  // ...
}

// Root Package.swift
import PackageDescription
import DependencyDescription // bikeshed

let schemaVersion = // ...

Package {
  Dependencies {
    External(...)
      .configuration(DependencyDescription.Configuration.super, when: .platform(.linux))
  }
}
3 Likes
7

While I appreciate the “behind the scenes” explanation, I don’t fully understand its relevance to the pitch, does this pitch do something about this static representation that I missed?

2 Likes
8

Personally, I don't find the proposed API compelling, it's just the same thing spelled slightly different. Perhaps its only advantage is the removal of the ordered requirement we currently see with the parameterized version. Better tooling could help with that pain point, but the rest of the issues with the current manifest remain. Perhaps my biggest complaint is that building every type of thing in the manifest necessarily requires interacting with String values which cannot be validated, autocompleted, or otherwise tied together. If I'm going to write Swift for the manifest there should be some advantage to doing so and there isn't right now.

18 Likes
9

I really like the @main idea, because it loosens the ordering constraints on the global scope.

Right now if we want to reuse something at several points, or to write dynamic logic, we have two options:

  1. Put it before the main init, cluttering the top of the file with functions and variables, and pushing the most relevant information down so you need to scroll to find it. (For an example, see SwiftPM’s manifest.)
  2. Mutate the package after initializing it, which often requires the extra effort of writing a search algorithm to locate the particular target that needs adjusting. (For an example, see SwiftSyntax’s manifest.)

But if you could define a function or variable at the bottom of the file, and then call it in the main portion at the top, then you could avoid the downsides of both current options.

10

I really like this. I can never get the syntax right in the existing format and heavily rely on copy-paste to make any significant changes. This would help a lot and I suspect also make it easier for people new to the language.

If the target dependencies could be computed automatically without too much or a performance hit that would be great. The only time I actually remember to also use a new dependency in a target is when I've just failed to do it for another one :sweat_smile:

As to the open questions:

  1. Should we use the @main or the script-like variant? @main is more structures and allows for passing context but more verbose / complicated?

I personally find the script-like variant more readable and I suspect it's going to be more approachable. Are there any benefits to using the @main approach over the script-like one?

One thing that I find quite weird about the @main-variant is that you need to come up with some name for the struct, MyPackage in the example. I always struggle naming things when I don't know how it's going to impact things down the line. It seems like there's not as good a default available here as in the script-like variant, where you can write let package = Package. Using struct Package: PackgeProtocol or something similar would be awkward.

  1. If choosing the script variant over the @main one, could we get rid of the need to let package = ... in fully declarative use cases? It is needed for the imperative use case but feels unnecessary in the fully declarative version.

That would be great and another argument in favour of the script-like variant.

  1. Is “modules” a good term to replace targets and products? Alternatively we can group them by type, e.g. “libraries”, "executables", "tests", "plugins", etc

I've come to think of "modules" as the products really so I would welcome this change. "target" is quite a "build-system-y" kind of term and "product" works best together with "build product", I feel. So "module" would be a good new term if it fits here (which it seems it does).

  1. Is “include” a good term for expressing module <> module relationship?

This one feels a bit weird. It sounds like it's including it instead of using or referencing it. Perhaps .dependsOn? Or .uses?

    Executable("MyExecutable", public: true)
      .dependsOn {
        Internal("MyDataModel")
        Internal("MyUtilities")
        External("SomeModule", from: "some-package") 
      }
  1. Is test module relationship with the module it is testing expressed as “for” and additional dependencies as “include” good terminology? should they be collapsed into one "include" section instead (more similar to the current manliest)?

Seems fine but I wonder if this could be attached to the module instead? I.e.

    Executable("MyExecutable", public: true)
      .include {
        Internal("MyDataModel")
        Internal("MyUtilities")
        External("SomeModule", from: "some-package") 
      }
      .tests {
        Test("MyExecutableTests")
      }
  1. Is “public: true” on Internal dependency the right way to express the dependency is to be exposed publicly? (this replaces product → target relationship for multi-target products)

I feel it would be better if it's more obvious this is an externally consumable module, perhaps via the module type, i.e.

    PublicExecutable(...)
      .include { ... }

although I don't love that it'd be a different type. It's not going to be possible, but public Library(...) would be a great way to declare that.

One thing that puzzles me about the example is the public: true on the nested Internal module:

    Library("MyLibrary", public: true)
      .include {
        Internal("MyDataModel", public: true)
        //                      ^^^^^^^^^^^^
        External("SomeOtherModule", from: "some-other-package") 
      }

Is that a typo or would this allow declaring a public module "MyDataModel" from within the .include clause? I think using PublicLibrary and disallowing it inside .include would be better.

  1. Should we continue to use a tools-version comment at the top of the file? Is there a better technique or name we should consider to express the manifest's schema/api version? For example, would schemaVersion or schemaAPIVersion be a more accurate name?

It's nice that the tools-version tracks the swift version and you don't need to keep track of which version of Swift brought which tools-version but the naming always felt a bit weird. Renaming that to schemaVersion or schemaAPIVersion would be clearer.

Excited to see where this is going! :tada:

4 Likes
11

tbh I don't think the new syntax, especially with @ main struct etc will be any easier to write by hand from scratch. Especially that IDE (and we're talking about Xcode) is not much better today with autocompletion (as we know from SwiftUI).

Given the manifest is still compiled to JSON, can we consider Package.swift.json as a supported format as well (akin to CocoaPods)? That would simplify even further generating manifests from third-party tools.

9 Likes
12

So it looks like there are a few distinct changes here:

  1. Unify products and targets using the "module" abstraction
  2. Use result-builders in place of arrays
  3. Use "builder style" APIs

I'll address each one separately

Unify products and targets using the "module" abstraction

This seems like a good simplification. I've already written code that mapped a single array of strings into the products/targets I wanted.

Use result builders in place of arrays:

As I see it, this is a trade-off. The main benefit is not needed to add commas and never having the compiler think you are writing .target(…).target(…). The downside is that there isn't an easy way to say "show me what values I can use here", which we can currently do by typing . and seeing what shows up in autocomplete. This is particularly noticeable in the providers example where there is no way (aside from consulting documentation) to know what other options might exist other than Apt, Yum and Brew.
Personally, I don't see this as compelling enough to make the switch. Result builders for tree-like DSLs like SwiftUI have a clearer value proposition since they get you out of bracket-hell, but for SPM I just don't see enough benefit to make the change.

Builder-style APIs

I'm not really sure I see a clear win here over just having initializers, perhaps the ability to specify initialization arguments in any order you want or split them into meaningful groupings like the following example?

.modules {
  /// Modules related to Foo
}
.dependencies {
  /// Dependencies related to Foo
}
.modules {
  /// Module related to Bar
}
.dependencies {
  /// Dependencies related to Bar
}
4 Likes
13

I love this (and, for the same reason, the “script-like” approach over the @main approach). This seems to me the same kind of thinking that deliberately sought to ensure that “Hello World” in Swift is the one-liner print("Hello World")—not an accident!

Build tools are complex enough, and making the manifest approachable at first glance is, in my view, no small thing. Eliminating the “target” versus “product” definitions and all other things not essential for user intent are huge.

Along the same lines, I don’t understand at first glance why executables need to be declared public: true—this seems to be needless copying of Swift’s access control model that for good reason requires public APIs to be declared as such, but who by default would assume that their package is going to publish private executables?

I’m also unclear on the distinction between an “internal public” include and an “external” (“external private”?) include as shown in the example by OP; in access control terminology, internal and public are mutually contradictory, while external visibility is what public does—so I would strongly suggest different terms to be used here. Do you mean instead “local” versus “remote”?

16 Likes
14

Thank you so much @tomerd for writing this pre-pitch. I can definitely feel the key issues you mentioned every time I work with manifest files. As I care about this topic as well, I have been thinking about same problems for some time now, and I came up with a couple of ideas.

I would like to propose few changes to your proposal that I think could make the package manifest even simpler. While doing about this, I have put focus on the following areas:

  • Eliminating internal string references. I think this is a very important topic, and I don't believe that we should have a major manifest redesign without addressing this issue.
  • Following the principles of progressive disclosure. A simple package should be easy to declare, while more advance features should be disclosed when needed.
  • Trying to make changes a sugar on top of the existing implementation. I didn't stick strongly to this one, but it might be close.

Here it goes.

Firstly, I think the Package type initializer should be a result builder where we define public package products. Secondly, I follow your idea of merging products and targets into a single entity.

With that, the simplest package could look like this:

Package {
    Library("MyLibrary")
}

Next, products often have dependencies. The Library type gets an initializer to declare dependencies through a result builder. Additionally, we allow declaring products within declarations of other products. Those products are dependencies of the parent product and are private by default, but can be made public, package, products if needed.

Package {
    Library("MyLibrary") {
        Library("MyUtilities")
        Library("MyDataModels", public: true)
    }
}

We also allow declaring external dependencies within declarations of other products. External packages get an API to reference their products.

Package {
    Library("MyLibrary") {
        Library("MyUtilities")
        Library("MyDataModels")
        ExternalPackage(at: "https://git-service.com/foo/some-package", upToNextMajor: "1.0.0")
            .library("SomeLibrary")
    }
}

When external packages are dependencies of multiple products, then they should be pulled out into a variable.

Package {

    let someLibrary = ExternalPackage(at: "https://git-service.com/foo/some-package", upToNextMajor: "1.0.0")
        .library("SomeLibrary")

    Library("MyLibrary") {
        Library("MyUtilities")
        Library("MyDataModels") {
            someLibrary
        }
        someLibrary
    }
}

I believe this is very important as it eliminates the need for inventing external packages name that can be used as string references, which is confusing for users and I've noticed it made a mess in the SwiftPM implementation.

The same should be done for package products that are dependencies of multiple other products.

Package {

    let myLowLevelLibrary = Library("MyLowLevelLibrary")

    Library("MyLibrary") {
        Library("MyUtilities") {
            myLowLevelLibrary
        }
        Library("MyDataModels") {
            myLowLevelLibrary
        }
    }
}

Saving an instance into a variable, instead of using string references, is a natural thing for a developer to do when passing the same thing to multiple functions. It eliminates the possibility of making a typo in one place and allows the compiler to statically evaluate correctness.

Finally, we can declare test products within products they are supposed to test, through a second result builder closure, eliminating the need for referencing the target product, which becomes an implicit dependency. Loving the application of SE-0279 here!

Package {
    Library("MyLibrary") {
        Library("MyUtilities")
        Library("MyDataModels")
    } tests {
        Tests("MyLibraryTests")
    }
}

Of course, we allow multiple test targets, as well as test targets having their own dependencies.

Package {
    Library("MyLibrary") {
        Library("MyUtilities")
        Library("MyDataModels")
    } tests {
        Tests("MyLibraryTests")
        Tests("MyLibraryOtherTests") {
            Library("MyTestUtilities")
        }
    }
}

Your full example:

import PackageDescription

let someLibrary = ExternalPackage(at: "https://git-service.com/foo/some-package", upToNextMajor: "1.0.0")
    .library("SomeModule")
let someOtherLibrary = ExternalPackage(at: "https://git-service.com/foo/some-other-package", upToNextMajor: "1.0.0")
    .library("SomeOtherModule")
let myDataModelLibrary = Library("MyDataModel", path: "custom/path", swiftSettings: "swiftSettings")

Package {
    Executable("MyExecutable") {
        myDataModelLibrary
        someLibrary
        Library("MyUtilities")
    } tests {
        Tests("MyExecutableTests") {
            Library("MyTestUtilities")
        }
    }
    Library("MyLibrary") {
        myDataModelLibrary
        someOtherLibrary
    } tests {
        Tests("MyLibrary")
    }
}
2 Likes
15

The "giant initializer" form is also officially recommended during the WWDC 2018 Getting to Know Swift Package Manager session (around the 13:40 mark).

I think something like .depends(on: [Module]) might read better.

A shebang-like directive is often full of hidden traps. Because it relies on a relatively inflexible structure and position not communicated to the user, just having a wrong whitespace or starting on the wrong line can render the entire manifest file uninterpretable. And because it's outside of the manifest/program's logic and the language's syntax rules, the user is not likely to understand the rules of the directive. Even if the user knows the rules, unless the error messages are able to point to the exact problem, it's not often a quick fix, because of the general subtlety and inconspicuousness of the directive's errors.

As someone who spent 2 months to make the tools-version comment line more flexible, I would like to see it deprecated for something more integrated with the manifest's logic.

Maybe we can expose ToolsVersion to the user, and let Package and related type's initializers (and static functions that work as initializers) take it as a parameter. For example, something like this:

Package(minimumSwiftVersion: v5_3) {
    .modules {
        Library("MyConcurrencyLibrary", minimumSwiftVersion: v5_5)
        Library("MyOtherLibrary")
    }
}

And if we add language feature flags to ToolsVersion, it could allow us to write something like this:

Package(minimumSwiftVersion: v5_3) {
    .modules {
        Library(
            "MyDistributedActorLibrary", 
            .when(currentToolsVersion.supportsDistributedActor)
        )
        Library(
            "MyLinuxCppWrapperLibrary",
            .when([
                .languageFeatures([.cppInterop])
                .platforms([.linux])
            ])
        )
        Library("MyOtherLibrary")
    }
}

Something like this where the tools-version requirement is part of the Package initialization can also allow all version-specific manifest files into a single one, which should have better code reuse and be easier to maintain.

5 Likes
16

That's fair. If I imagine how this could be improved, the obvious choice would be to use some sort of enum for products/targets/dependencies. But then we have to declare a bunch of enums and have separate code to hook them all up, and the whole thing ends up being a lot more verbose.

What we'd ideally want is for some kind of new language feature (perhaps an extension to the result builder transform), where elements of a result builder can define cases in an enum.

Other parts of the project would likely benefit from this, too. For example, the recent regex proposals mention that the cases of an Alternation/OneOf pattern should ideally be available as some kind of enum on the match result, and AFAIK that applies to both regex literals and the Pattern result-builder DSL. So that's nice. Having overlapping requirements makes it easier to scope the desired language feature and maybe implement it one day.

In the mean time, there's a much simpler option - we could just give code completion (which is part of the compiler) some specific knowledge of the Package.swift format, allowing it to auto-complete some common string values (e.g. target names). It's a hack, for sure, but it only affects this one file to give you a better editing experience.

4 Likes
17

My suggestion would actually be to adopt some sort of TOML format and use the various existing tools or methods to enable a simple autocomplete environment for most IDEs. Since you're already operating at a string level, suggesting string completions isn't too hard. Plus it will be much more readable than the Swift manifest, and much easier to write to for those without full IDE support.

Thinking about it more, a new builder API would be an improvement, if only to get rid of the parameter ordering restrictions I already mentioned. Given how hard it is to find the docs for those APIs outside of Xcode in package mode, any improvement to remove the need to do so would be appreciated.

3 Likes
18

Would this change allow you to use some kind of "typed libraries" too? I would like to prevent copy/pasting the names, especially with big package manifests.

let myDataModel = Library("MyDataModel")
Library("MyLibrary", public: true) {
    include {
        Internal(myDataModel, public: true)
        External("SomeOtherModule", from: "some-other-package") 
    } 
}
19

I've got mixed feeling with this, but I certainly see good usecases.

First of all, I would strongly suggest ignoring the 'added complexity' of writing @main in the package definition. If we get more features for basically free, do it. 99.9% of the people will use a premade/generated project and edit things from there. Those who don't will know what they're doing anyways.

I think this move is absolutely great, because it allows us to expand upon the Package manifest extremely easily and in a tonne of directions. The flexibility this would add to the manifest is awesome, especially considering the use of compiler directives such as #if.

My main issue would be how verbose it is for sure, but the verbosity makes this extremely easy to reason about. Adding to that the possibility of using conditions and a (future?) context in which we can derive the target platform(s) and environment we can do a lot of stuff with this.

As for the tools-version, couldn't we make multiple PackageXYZ protocols that expand upon their previous versions? Ignoring the protocol naming: protocol Swift5_7Package: Swift5_6Package { .. }

If I define a package in Swift 5.6+, I'll use Swift5_6Package, which'll be available from that Package manager forwards. If I want to support older Swift versions, I can reply upon the corresponding protocol. This can also be more easily checked for generating package indexes, since we don't use a comment as a workaround.

2 Likes
20

#if will remain as ill‐advised as it is now...

...because the manifest is compiled for the development device, not the target device. For example, #if os(Linux) in macOS/Linux package breaks cross‐compilation in both directions.

If you need a workaround for some condition absent from PackageDescription, it is much better to do adjustments based on the environment (e.g. if ProcessInfo.processInfo.environment["TARGETING_LINUX"] == "true"). But even then it should only be used as a last resort.

1 Like