[Pitch] Package Editor Commands

Evolution Pitches
,
1

Hi all,

I wanted to share a quick pitch on adding some swift package subcommands for making mechanical edits to the Package.swift manifest. I've got this maybe 70% implemented over at https://github.com/apple/swift-package-manager/pull/3034 building on some earlier work by @Aciid, and I think it's at the point now where it makes sense to start discussing the CLI interface and people's use cases. As always, feedback and questions are welcome! Without further ado...

Package Editor Commands

Introduction

Because Swift package manifests are written in Swift using the PackageDescription API, it is difficult to automate common tasks like adding a new product, target, or dependency. This proposal introduces new swift package subcommands to perform some common editing tasks which can streamline users' workflows and enable new higher-level tools.

Motivation

There are a number of reasons someone might want to edit a package using a CLI or library interface instead of editing a manifest by hand:

  • In some situations, it's less error-prone than editing the manifest manually. Package authors could provide a one line command in their README to easily integrate the latest version.
  • Because more of the process is automated, users would no longer need to remember details of the package layout convention, like which folders they need to create when adding a new library target.
  • Using libSwiftPM, IDEs could offer to update the manifest automatically when the user tries to import a missing dependency or create a new target.
  • Future features like package collections and package registries could make dependency integration easier for users browsing for packages.

Additionally, many other package managers offer similar features:

  • npm's npm install command for adding dependencies to its package.json
  • Tools like cargo-edit for editing the cargo.toml format used by Rust
  • Elm's elm install command for adding dependencies to elm.json

Proposed solution

This proposal introduces three new swift package subcommands: add-product, add-target, and add-dependency, which edit the manifest of the current package. Together, these encompass many of the most common editing operations performed by users when working on a package.

Detailed design

New Commands

The following subcommands will be added to swift package:

swift package add-product <name> [--type <type>] [--targets <targets>]

name: The name of the new product.
type: executable, library, static-library, or dynamic-library. If unspecified, this will default to library.
targets: A comma separated list of target names to to add to the new product.

swift package add-target <name> [--type <type>] [--no-test-target] [--dependencies <dependencies>]
name: The name of the new target.
type: library, executable or test. If unspecified, this will default to library. The distinction between library and executable targets doesn't impact the manifest currently, but will determine whether or not a main.swift file is created.
--no-test-target: By default, a test target is added for each regular target unless this flag is present.
dependencies: A comma separated list of target dependency names.

In addition to editing the manifest, the add-target command will create the appropriate Sources and Tests subdirectories for new targets.

swift package add-dependency <url> [--exact <version>] [--revision <revision>] [--branch <branch>] [--from <version>] [--up-to-next-minor-from <version>]
url: The URL of the new package dependency. This may also be the path to a local package.
Only one of the following options may appear to specify a package dependency requirement:
--exact : Specifies a .exact(<version>) requirement in the manifest.
--revision : Specifies a .revision(<revision>) requirement in the manifest.
--branch : Specifies a .branch(<branch>) requirement in the manifest.
--from : Specifies a .upToNextMajor(<version>) requirement in the manifest.
--up-to-next-minor-from : Specifies a .upToNextMinor(<version>) requirement in the manifest.

If no requirement is specified, the command will default to a .upToNextMajor requirement on the latest version of the package.

Note: These new commands will be restricted to only operate on package manifests having a swift-tools-version of 5.2 or later. This decision was made to reduce the complexity of the feature, as there were a number of major changes to the PackageDescription module in Swift 5.2. It is expected that in the future, support for editing manifests with older tools versions will be maintained on a best-effort basis.

Examples

Given an initially empty package:

// swift-tools-version:5.3
import PackageDescription

let package = Package(
    name: "MyPackage"
)

The following commands can be used to add dependencies, targets, and products:

swift package add-dependency https://github.com/apple/swift-argument-parser
swift package add-target MyLibrary
swift package add-target MyExecutable --no-test-target --dependencies MyLibrary,ArgumentParser
swift package add-product MyLibrary --targets MyLibrary

Resulting in this manifest:

// swift-tools-version:5.3
import PackageDescription

let package = Package(
    name: "MyPackage",
    products: [
        .library(
            name: "MyLibrary",
            targets: [
                "MyLibrary",
            ]),
    ],
    dependencies: [
        .package(name: "swift-argument-parser", url: "https://github.com/apple/swift-argument-parser", .upToNextMajor(from: "0.3.1")),
    ],
    targets: [
        .target(
            name: "MyLibrary",
            dependencies: []),
        .testTarget(
            name: "MyLibraryTests",
            dependencies: [
                "MyLibrary",
            ]),
        .target(
            name: "MyExecutable",
            dependencies: [
                "MyLibrary",
                .product(name: "ArgumentParser", package: "https://github.com/apple/swift-argument-parser")
            ]),
    ]
)

Security

This proposal has minimal impact on the security of the package manager. Packages added using the add-dependency subcommand will be fetched and their manifests will be loaded, but this is no different than if the user manually edited the manifest to include them.

Impact on Existing Packages

Since this proposal only includes new editing commands, it has no impact on the semantics of existing packages.

Alternatives considered

The only alternative seriously considered was not including this functionality at all. It's worth noting that maintaining this functionality over time and adapting it to changes in the PackageDescription API will require a nontrivial effort. However, the benefits of including the functionality are substantial enought that it seems like a worthwhile tradeoff.

Future directions

Support for Deleting Products/Targets/Dependencies and Renaming Products/Targets

This functionality was considered, but ultimately removed from the scope of the initial proposal. Most of these editing operations appear to be fairly uncommon, so it seems better to wait and see how the new commands are used in practice before rolling out more.

Add a swift package upgrade Command

A hypothetical swift package upgrade command could automatically update the version specifiers of package dependencies to newer versions, similar to npm upgrade. This is another manifest editing operation very commonly provided by other package managers.

50 Likes
[Accepted with modification] SE-0301: Package Editor Commands
[Pitch] New command for package creation, and package templates
2

+100

This is awesome! Thanks for driving this :clap:

2 Likes
3

Great pitch. Thanks for this!

Could we also add support for binary targets at this or later stage? Thanks.

4

I think it would be reasonable to support something like swift package add-target BinaryTarget —type binary —url <url> —checksum <checksum> as part of this proposal, maybe with some kind of warning if you try to do it on non-Darwin platforms. System library targets are also left out of the proposal, mostly because I couldn’t come up with a good CLI for them.

1 Like
5

Awesome; I was just looking for these commands yesterday!

2 Likes
6

Would this also work with Xcode-based package management?
As far as I can tell, there is no Package.swift file when adding SPM packages through Xcode's UI...

7

Xcode projects which have package dependencies but no Package.swift manifest aren’t packages themselves, so these new commands (and the rest of the already existing swift package commands) don’t apply to them.

1 Like
8

What should I expect if I run a command like that on a complex package file containing executable code? (example)

Is something like that completely unsupported and can break my package? I would like for it to show an error message and don't do anything instead.

1 Like
9

The new tools will make a best-effort to edit manifests that aren’t fully declarative, and report an error if they can’t. I don’t think it makes sense to specify the exact behavior in the proposal because it can likely be improved over time, but currently it’s:

  • if adding a new item and the corresponding argument doesn’t exist in the package initializer at all, insert it and the new item
  • if the corresponding argument is an array literal expression, insert the new item
  • if the corresponding argument is a concatenation which contains at least one array literal expression, insert the new item into that
  • otherwise, report an error

In practice, I expect this will handle nearly all non-declarative manifests, with maybe a few exceptions here and there. You might notice that the logic above isn’t completely foolproof - some clever operator overloading could break it - but it’s so unlikely someone would do that in a manifest that I don’t think it’s a concern.

3 Likes
10

Firstly, a question: what does this do about conditional-compilation blocks (#if os(linux), etc), imperative code (package.dependencies += [...]), and comments? Are they all preserved? Can we detect them and fail the operation if it will destroy information?

I think this is a problem any editing tool will face. FWIW, my preference would be to detect those situations and fail, with a flag to force the change.

Secondly, I'd like to propose an alternative - I agree that this is really important and useful functionality, but I think we could implement it better. You already mentioned some of the difficulties with this approach:

I'd like to add a couple more:

  • Limited functionality. It is always likely to remain very limited due to the aforementioned maintenance burden, and the limitations of the command-line interface. It doesn't scale to give every operation on every type it's own, individual CLI entry-point.

  • Difficult to remember. You mention as one of the motivations:

    Instead of remembering package layout conventions, they will now have to remember commands with very long names, each with unique sets of flags, and each flag itself also with a relatively long name (e.g. --up-to-next-minor-from). Your 4-line example has a lot of repetition (some of that is inherent to CLI interfaces, but it's still ugly and typo-prone).

    Simply put, I don't see the CLI ever being anybody's preferred way of sitting down and making a change to a Package.swift file; IMO the focus of this feature should be automation.

So here's the thing: what if we didn't do all that? Why are we even creating a parallel interface? For scripting? We already have a great Package interface, written in Swift, and Swift supports scripting. So why can't I write a script to mutate a Package instance?

Well, actually, we can! It turns out that loads of projects have secret little scripts hidden inside their Package.swift and are doing this exact thing already! SwiftPM itself does it, swift-driver has something similar, so does sourcekit-lsp, etc. Other projects may require custom build scripts because environment-specific options and paths cannot be written using one Package.swift file, and they lack the tools to automatically edit them.

On the one hand, these little scripts at the end of Package.swift files are for configuration, not for the specific use-cases presented in this pitch (integrating a package you found on the web). On the other hand, they are an example of machine-editing Package.swift files, using the Package API, which is extremely widespread today.

So the alternative that I'd like to pitch is to use Swift's existing script functionality to allow for focussed scripts which edit a Package object.

swift package update-script <script name>

We'd create a new module, EditablePackage (?), with a single global declaration. All package update scripts would implicitly import EditablePackage if they don't do it explicitly.

// module EditablePackage
import PackageDescription

var package: Package

We'd run the existing Package.swift file, resulting in us constructing a Package object named package, which we'd assign to EditablePackage's global. Then we'd run the script file in a special context where EditablePackage is implicitly imported. Once the script is finished, we'd validate and serialise the updated object found in EditablePackage.package.

Pros:

  • We get the full flexibility of the Package API, with no additional maintenance burden translating them in to CLI switches. This would be immensely more powerful, enabling use-cases from integrating packages to configuration scripts (such as those used by SwiftPM, sourcekit-lsp, etc). The possibilities are endless.

  • It's just Swift. There is one thing to remember, and you already know how it works. It's easy to read and verify that the script does what you expect.

Cons:

  • The command accepts a file, rather than a string. That isn't very convenient, since you often want to run some commands without writing a file first.

    However, there are ways to use stdin as your input file. The convention is to use a sole - as the file name. The swift compiler already supports that (clang also supports it, as do other tools like VSCode - it's a super useful feature). For example, if you run:

    xcrun swift -

    You can just start typing code, with newlines and quotes and everything, and finish off with EOF/CMD+D and the compiler will run it.

    For copy/pasting commands, you can use a "heredoc". For example, try copying and pasting the following in to a terminal:

    xcrun swift - <<EOF
var a = "hello"
a.append(", world")
print(a)
EOF

    The best thing? Everybody here can read what this code does.

  • Some situations can become a little more verbose. Consider as a CLI:

    swift package add-dependency https://github.com/apple/swift-argument-parser
swift package add-target MyLibrary
swift package add-target MyExecutable --no-test-target --dependencies MyLibrary,ArgumentParser
swift package add-product MyLibrary --targets MyLibrary

    Or as a script:

    swift package update-script - <<EOF
package.dependencies += .package(url: "https://github.com/apple/swift-argument-parser")
package.targets += [
  .target(name: "MyLibrary")
  .executableTarget(name: "MyExecutable", .dependencies: ["MyLibrary", "ArgumentParser"])
]
package.products += .library(name: "MyLibrary", targets: ["MyLibrary"])
EOF

    Which is disappointing. The Swift version has bits of ceremony (like .package(url:...)) which aren't required in the CLI. But the CLI does one operation at a time, so it has a bunch of ceremony of its own (swift package add-) that the script doesn't have.

    And let's not forget that we can add APIs to EditablePackage to make these scripts smaller. For example, we could add a function to add a dependency and add it as a dependency to some list of targets, in one move. It's a lot easier to do these things if we're writing APIs for Swift scripts using the Package type. Maybe that's also a way to expose things like --no-test-target.

3 Likes
SE-0303: Package Manager Extensible Build Tools
11

  1. Conditional compilation blocks really shouldn't appear in manifests since they'll break cross-compilation, but they'll be preserved during editing. If the entire Package initializer was conditionally compiled you might run into failures

  2. Imperative code is left alone. All the editing operations described only rewrite arguments to the Package initializer

  3. Comments are preserved, although they could be moved some cases. For example, if I add a new target to this list:

[
  "one",
  "two", // comment
]

It's ambiguous whether we should associate the comment with the element "two", or the last element after the addition. Comments not directly adjacent to edits will be unaffected.

  1. The manifest will be compiled and loaded before committing any edit, and the changes will be rolled back if it fails for any reason.

The main problem here is that we'd have no way to persist arbitrary edits to the manifest as far as I can tell? Side effects, environment variables, and conditional compilation could all introduce branching, and tracing the execution of the script wouldn't be enough to recover all of that information. I think we also disagree on the motivations somewhat: I expect users will use the CLI somewhat often for simple invocations like swift package add-target MyExecutable --type-executable when they don't remember specific syntax.

2 Likes
12

I see, so this appears to be a source-level edit. That's interesting.

Right. AFAIK, it is discouraged for Package.swift files to contain this sort of code in the first place (they should be declarative), so people would have to remove them, but we'd have a clear transition strategy: consider your current Package.swift a template and use a configuration script to generate the appropriate Package.swift, using environment-specific edits. It's the same Package interface so you should be able to pretty-much cut and paste that code in to a new file and have it work.

Right. When weighed against the needs of automation, how much more flexible the scripting approach is, and the significantly lower overall maintenance burden it has, I don't think that use-case justifies the effort.

Editing a Package.swift file isn't hard, even for the biggest projects. It isn't as convenient as having a tool do it for you, but ultimately it's just a Swift file, containing a big, declarative initialiser and no imperative code :wink:. You can also create shortcuts or command aliases, write canned scripts for common operations, or we can add APIs to the EditablePackage module to make those scripts easier.

None of those things are quite as convenient as having a specific tool entry-point for the thing you want to do, but considering the use-cases it serves better against the use-cases it doesn't serve as well, and most importantly, the maintenance burden and possibility of bugs creeping in, I think it's a compelling alternative.

The biggest issue with scripting is that I don’t think it would be possible to preserve comments, at all. And that may actually be a deal-breaker :slightly_frowning_face:

2 Likes
13

First of all, love this work. Excited to see these commands come to SwiftPM

I wanted to quickly note that although it has been historically discouraged to use conditional compilation blocks in package manifests, we can’t ignore that there are legitimate reasons to add them. It’s more of an “avoid when unnecessary” than a “this is bad, period.” See this very recent recommendation RE static compilation for Linux, for example: Static linking on Linux in Swift 5.3.1 - #3 by drexin

My point being, it’s important we don’t ignore or intentionally avoid the topic in the context of package edit commands. Based on the above comments, this has been thought about and won’t break.

1 Like
14

Out of all of these, the one with the largest value is add-dependency as it fills a very noticeable gap in SwiftPM compared to its contemporaries, and explaining to primarily iOS engineers who don't have any experience with SwiftPM leaves them scratching their heads or groaning when it's time to add a dependency.

The other two are "nice to have", but I'm more inclined to leave that more to the scripting world like @Karl is suggesting, because I think being able to programmatically edit the Package as needed is useful as part of a whole suite of capabilities with Package Build Tools extensions

15

That sounds like a good solution! Thanks!

16

Thanks for the feedback everyone! I've pushed some revisions to the proposal here to add some additional clarification based on the discussion. There's not much in the way of major changes, but I:

  • clarified that add-target will support binary targets,
  • expanded the alternatives considered section a bit based on the discussions here, and
  • included a slightly more complicated example so it's clearer the tools are making syntax-level edits which preserve other comments/formatting/code in the file
17

This pitch reminds me of PlistBuddy, Apple's tool for reading and editing plist files. I use PlistBuddy to script incrementing the build number in Info.plist files as part of an automated build. It does this by reading the current build number, incrementing it and writing it back.

I don't see anything in this proposal about reading values, only writing them. I haven't used spm so I'm not sure if that's needed but it seems like a useful feature for scripting.

18

@phoneyDev The existing swift package describe command is very close to what you're describing, and adding --type json makes its output machine-readable, so most of the important use cases are covered already.

One thing I am interested in is using the new PackageSyntax library which powers this proposal to record more source location information for different parts of the manifest. This could be used to improve the describe output, support inline Package.swift diagnostics, etc. That's all outside the scope of this proposal though.

19

This is a super useful feature for spm to have (I was looking for it almost a year ago) and just wanted to thank you for making it actually happen!

20

This is a great idea, but I'd like to ask for 3 features and one pitch documentation change here:

  1. Support for GitHub repositories with a shortcut on add-dependency in the url section which allows me to write something like swift package add-dependency sunshinejr/SwiftyUserDefaults and will figure out the URL itself, or alternatively adding the option --github to make it more explicit.

  2. Support for adding the GitHub tagline automatically as a comment above each dependency entry to make it easier for people reading the manifest what the dependencies are about, e.g. running swift package add-dependency sunshinejr/SwiftyUserDefaults --github --tagline will automatically result in something like this (note also the empty lines between entries in this mode):

    dependencies: [
    // Modern Swift API for NSUserDefaults
    .package(name: "SwiftyUserDefaults", url: "https://github.com/sunshinejr/SwiftyUserDefaults.git", .upToNextMajor("3.1.5")),

    // Delightful console output for Swift developers.
    .package(name: "Rainbow", url: "https://github.com/onevcat/Rainbow.git", .upToNextMajor("3.1.5")),

    // A powerful framework for developing CLIs in Swift
    .package(name: "SwiftCLI", url: "https://github.com/jakeheis/SwiftCLI.git", .upToNextMajor("6.0.1")),
 ],

  3. Could we also add an option to keep the different array sections sorted by alphabet, like --sorted based on the name or alternatively the last portion before the .git in the URL?

  4. The new commands section currently does a good job in documenting the different options, but I think it would be great for each section to add at least one full example including a real library URL in the case of add-dependency so some things become more clear on first sight.

2 Likes