Convention-based configuration: a possible way to reduce overhead and repetition, and so encourage declarative Package.swifts?

Currently, there's a non-trivial amount of boilerplate and repetition for many simple and complex cases with SwiftPM, and it seems like we could make the experience slicker by taking inspiration from other package managers that have more conventions to facilitate low-configuration packages.

SwiftPM now

The manifest of https://github.com/apple/example-package-deckofplayingcards currently looks like:

// swift-tools-version:4.0
/*
 This source file is part of the Swift.org open source project
 Copyright 2015 Apple Inc. and the Swift project authors
 Licensed under Apache License v2.0 with Runtime Library Exception
 See http://swift.org/LICENSE.txt for license information
 See http://swift.org/CONTRIBUTORS.txt for Swift project authors
*/

import PackageDescription

let package = Package(
    name: "DeckOfPlayingCards",
    products: [
        .library(name: "DeckOfPlayingCards", targets: ["DeckOfPlayingCards"]),
    ],
    dependencies: [
        .package(url: "https://github.com/apple/example-package-fisheryates.git", from: "2.0.0"),
        .package(url: "https://github.com/apple/example-package-playingcard.git", from: "3.0.0"),
    ],
    targets: [
        .target(
            name: "DeckOfPlayingCards",
            dependencies: ["FisherYates", "PlayingCard"]),
        .testTarget(
            name: "DeckOfPlayingCardsTests",
            dependencies: ["DeckOfPlayingCards"]),
    ]
)

This is a fairly common case: a package that provides a library and has tests, and yet it requires saying the name DeckOfPlayingCards... 6 times, along with fairly tautological things like specifying that the tests depend on the main code!

Cargo's approach

The equivalent with, say, Rust's cargo would be a Cargo.toml manifest like the following, along with the main library code in src/lib.rs and tests as arbitrary files in tests/*.rs:

# ... copyright header, I guess ...

[package]
name = "DeckOfPlayingCards"
version = "0.1.0"
authors = ["Swift Project <you@example.com>"]

[dependencies]
# If the dependencies are published in a registry:
FisherYates = "2.0.0" 
PlayingCard = "3.0.0"
# or,
# FisherYates = { git = "https://github.com/apple/example-package-fisheryates.git" }
# PlayingCard = { git = "https://github.com/apple/example-package-playingcard.git" }

This mentions the name just once, but if customisation of location etc. is required, a [lib] section can control the library product and, similarly, [[test]] and [[bin]] (executables) and [[example]] ("test" executables). There's a few details that mean this works particularly well with Cargo:

  • a package corresponds to products of zero or one libraries (i.e. things that one can import), and any number of binaries/tests/examples, etc.
  • the latter implicitly/automatically have a dependency on the library if it exists
  • all of the targets (both library and the others) automatically have access to all the dependencies, plus tests and examples depend on an extra set of "development dependencies" (Swift Package Manager Evolution Ideas)

The first means that depending on a package is unambiguous: it is a dependency on the single library product of that package, and the second two mean that there's no need to explicitly list out what each target depends on.

This definitely has some downsides:

  • a project that consists of many internal libraries needs quite a few Cargo.toml and coordinating dependency versions across them requires using a workspace
  • the exact dependencies that a target/product has aren't listed (e.g. imagine a package that depends on external libraries A and B and has two products X and Y: it might be that X only depends on A and Y only B)
  • it is a little less clear exactly what a package provides from just the manifest (this isn't so bad with Cargo's one-library-per-package rule: it is clear what library a package can provide)

But it also has a variety of upsides:

  • publishing/creating a package is step that "weighs" less, mentally, as things Just Work and feel slick

  • there's much less repetition, meaning people aren't tempted to write non-declarative things like

    let name = "DeckOfPlayingCards"
    let package = Package(
      name: name,
      // ...
        .testTarget(name: name + "Tests", ...)
      // ...
    

SwiftPM in future?

With some conventions similar to cargo's the Package could look more like the following:

let package = Package(
    name: "DeckOfPlayingCards",
    dependencies: [
        .package(url: "https://github.com/apple/example-package-fisheryates.git", from: "2.0.0"),
        .package(url: "https://github.com/apple/example-package-playingcard.git", from: "3.0.0"),
    ]
)

I think it's worth noting that SwiftPM does already have some convention-based configuration: the original configuration (essentially) uses the conventions for the path target property: Sources/FisherYates and Tests/FisherYatesTest. Thus, adding more conventions seems more of a (large) extension of the current behaviour than a fundamental change in philosophy.

Possible conventions

An initial set of conventions might be something like

  1. Sources/X creates a target X that depends on all library products of all the dependencies. There's also some inference of other dependencies and products from the contents of the directory:
    1. if it's not a Swift product (e.g. contains C files), there's no product, but the target is considered an (importable) module and it doesn't depend on anything within the current package
    2. if there's no main.swift, it's a library product .library(name: "X", targets: ["X"]) and (importable) module, and also doesn't depend on anything within the current package
    3. if there is one, it's an executable product .executable(name: "X", targets: ["X"]), and the target depends on the modules (i.e. the previous two points) of the current package
  2. Tests/X creates a test target X that depends on all the dependencies as above, and the modules of the current package.

One can still explicitly specifying things, which would stop the conventions applying to that particular target/product, but it would not stop the conventions applying to other things that match the patterns above, e.g. if one wants to tighten the dependencies of a single target, or have modules that depend on another modules within the same package.

Real-world example: NIO libraries

For swift-nio-http2, I suspect the rules above mean it reduces to:

let package = Package(
    name: "swift-nio-http2",
    dependencies: [
        .package(url: "https://github.com/apple/swift-nio.git", from: "1.7.0"),
        .package(url: "https://github.com/apple/swift-nio-nghttp2-support.git", from: "1.0.0"),
    ]
)

The NIOHTTP2Server executable and target is inferred from Sources/NIOHTTP2Server existing and having main.swift, and the NIOHTTP2 library and target is inferred from Sources/NIOHTTP2 existing without a main.swift, etc. The only dependencies this changes is NIOTLS becomes a direct dependency of NIOHTTP2Server (not a transitive one) and NIOHTTP2Tests, and CNIONghttp2 depends on the two external dependencies too.

Similarly, swift-nio-ssl could likely become:

let package = Package(
    name: "swift-nio-ssl",
    dependencies: [
        .package(url: "https://github.com/apple/swift-nio.git", from: "1.0.0"),
        .package(url: "https://github.com/apple/swift-nio-ssl-support.git", from: "1.0.0"),
    ]
)

On the other hand, the current set up of swift-nio wouldn't completely disappear, but it would shrink dramatically (the target array would have 6 elements, instead of 25). One could still remove all the executable products and their targets, the tests ones, as well as the dependency-less importable ones, however there's dependencies between the library targets/products that mean they'd have to be explicitly specified:

import PackageDescription

let package = Package(
    name: "swift-nio",
    products: [
        .library(name: "NIO", targets: ["NIO"]),
        .library(name: "NIOTLS", targets: ["NIOTLS"]),
        .library(name: "NIOHTTP1", targets: ["NIOHTTP1"]),
        .library(name: "NIOConcurrencyHelpers", targets: ["NIOConcurrencyHelpers"]),
        .library(name: "NIOFoundationCompat", targets: ["NIOFoundationCompat"]),
        .library(name: "NIOWebSocket", targets: ["NIOWebSocket"]),
    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-nio-zlib-support.git", from: "1.0.0"),
    ],
    targets: [
        .target(name: "NIO",
                dependencies: ["CNIOLinux",
                               "CNIODarwin",
                               "NIOConcurrencyHelpers",
                               "CNIOAtomics",
                               "NIOPriorityQueue",
                               "CNIOSHA1"]),
        .target(name: "NIOFoundationCompat", dependencies: ["NIO"]),
        .target(name: "NIOConcurrencyHelpers",
                dependencies: ["CNIOAtomics"]),
        .target(name: "NIOHTTP1",
                dependencies: ["NIO", "NIOConcurrencyHelpers", "CNIOHTTPParser", "CNIOZlib"]),
        .target(name: "NIOTLS", dependencies: ["NIO"]),
        .target(name: "NIOWebSocket",
                dependencies: ["NIO", "NIOHTTP1", "CNIOSHA1"]),
    ]
)

(Benefit: the targets array is now small enough that it's more reasonable to have it directly in the package, rather than using a (slightly) non-declarative separate var.)

Arguably, removing some but not all of the importable targets like this it hard to understand, because some of the importable targets in this package are only deduced from the filesystem, which differs to cargo: due to the one-library-per-package rule, any "module" dependencies would appear as explicit external dependencies in the [dependencies] section of the manifest. The conventions I proposed above could easily instead require that dependencies within explicit .targets can only refer to non-inferred targets, and similarly for targets within an explicit product. With this rule, the above would have to also write .target(name: "CNIOLinux") and .target(name: "CNIODarwin") etc.

An alternative that avoids the confusion above, and satisfies that stricter rule, would be to move things into other packages, e.g. all the C* packages and lower-level NIOConcurrencyHelpers and NIOPriorityQueue:

import PackageDescription

let package = Package(
    name: "swift-nio",
    products: [
        .library(name: "NIO", targets: ["NIO"]),
        .library(name: "NIOTLS", targets: ["NIOTLS"]),
        .library(name: "NIOHTTP1", targets: ["NIOHTTP1"]),
        .library(name: "NIOFoundationCompat", targets: ["NIOFoundationCompat"]),
        .library(name: "NIOWebSocket", targets: ["NIOWebSocket"]),
    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-nio-zlib-support.git", from: "1.0.0"),
        // hypothetical URLs:
        .package(url: "https://github.com/apple/swift-nio-c-support-libraries.git", from: "1.0.0"),
        .package(url: "https://github.com/apple/swift-nio-concurrency-helpers.git", from: "1.0.0"),
        .package(url: "https://github.com/apple/swift-nio-priority-queue", from: "1.0.0"),
    ],
    targets: [
        .target(name: "NIO"),
        .target(name: "NIOFoundationCompat", dependencies: ["NIO"]),
        .target(name: "NIOHTTP1",
                 dependencies: ["NIO", "NIOConcurrencyHelpers", "CNIOHTTPParser", "CNIOZlib"]),
        .target(name: "NIOTLS", dependencies: ["NIO"]),
        .target(name: "NIOWebSocket",
                dependencies: ["NIO", "NIOHTTP1", "CNIOSHA1"]),
    ]
)

Thoughts?

(To be clear, while I work on Swift, I do not work on the package manager. I'm purely speaking from my previous experience using Cargo here.)

3 Likes

Thanks for the post, @huon. It is definitely difficult to find the right balance between convention and configuration. SwiftPM was highly convention based when it was initially introduced. For e.g., this is how the manifest looked like for DeckOfPlayingCards.

This caused a lot of confusion for the users as it made the model very difficult to understand. For e.g., you needed to memorize all the rules and look at the disk structure to figure out the project model. We tried fixing all those issues with SE-0158. I think we can extend additional conventions as long as a new convention is very clear, beneficial and doesn't require memorizing something.

1 Like

Ah, I hadn't realised the history. From SE-0158 it also looks declarative manifests are a non-goal with the points like "Make all properties of Package and Target mutable". With all that context, this makes my proposal much less reasonable. Thanks for the pointers!

No, non-declaration manifest is actually a non-goal. We made everything mutable because we don’t currently provide APIs to allow conditionalization in a declarative manner :/

Oh, I see; that's a temporary concession pending a more structured way to conditionalize things? That makes sense too!

1 Like