Awesome; I was just looking for these commands yesterday!
Would this also work with Xcode-based package management?
As far as I can tell, there is no
Package.swift file when adding SPM packages through Xcode's UI...
Xcode projects which have package dependencies but no Package.swift manifest aren’t packages themselves, so these new commands (and the rest of the already existing swift package commands) don’t apply to them.
What should I expect if I run a command like that on a complex package file containing executable code? (example)
Is something like that completely unsupported and can break my package? I would like for it to show an error message and don't do anything instead.
The new tools will make a best-effort to edit manifests that aren’t fully declarative, and report an error if they can’t. I don’t think it makes sense to specify the exact behavior in the proposal because it can likely be improved over time, but currently it’s:
- if adding a new item and the corresponding argument doesn’t exist in the package initializer at all, insert it and the new item
- if the corresponding argument is an array literal expression, insert the new item
- if the corresponding argument is a concatenation which contains at least one array literal expression, insert the new item into that
- otherwise, report an error
In practice, I expect this will handle nearly all non-declarative manifests, with maybe a few exceptions here and there. You might notice that the logic above isn’t completely foolproof - some clever operator overloading could break it - but it’s so unlikely someone would do that in a manifest that I don’t think it’s a concern.
Firstly, a question: what does this do about conditional-compilation blocks (
#if os(linux), etc), imperative code (
package.dependencies += [...]), and comments? Are they all preserved? Can we detect them and fail the operation if it will destroy information?
I think this is a problem any editing tool will face. FWIW, my preference would be to detect those situations and fail, with a flag to force the change.
Secondly, I'd like to propose an alternative - I agree that this is really important and useful functionality, but I think we could implement it better. You already mentioned some of the difficulties with this approach:
I'd like to add a couple more:
Limited functionality. It is always likely to remain very limited due to the aforementioned maintenance burden, and the limitations of the command-line interface. It doesn't scale to give every operation on every type it's own, individual CLI entry-point.
Difficult to remember. You mention as one of the motivations:
Instead of remembering package layout conventions, they will now have to remember commands with very long names, each with unique sets of flags, and each flag itself also with a relatively long name (e.g.
--up-to-next-minor-from). Your 4-line example has a lot of repetition (some of that is inherent to CLI interfaces, but it's still ugly and typo-prone).
Simply put, I don't see the CLI ever being anybody's preferred way of sitting down and making a change to a
Package.swiftfile; IMO the focus of this feature should be automation.
So here's the thing: what if we didn't do all that? Why are we even creating a parallel interface? For scripting? We already have a great
Package interface, written in Swift, and Swift supports scripting. So why can't I write a script to mutate a
Well, actually, we can! It turns out that loads of projects have secret little scripts hidden inside their
Package.swift and are doing this exact thing already! SwiftPM itself does it, swift-driver has something similar, so does sourcekit-lsp, etc. Other projects may require custom build scripts because environment-specific options and paths cannot be written using one
Package.swift file, and they lack the tools to automatically edit them.
On the one hand, these little scripts at the end of
Package.swift files are for configuration, not for the specific use-cases presented in this pitch (integrating a package you found on the web). On the other hand, they are an example of machine-editing Package.swift files, using the
Package API, which is extremely widespread today.
So the alternative that I'd like to pitch is to use Swift's existing script functionality to allow for focussed scripts which edit a
swift package update-script <script name>
We'd create a new module,
EditablePackage (?), with a single global declaration. All package update scripts would implicitly
import EditablePackage if they don't do it explicitly.
// module EditablePackage import PackageDescription var package: Package
We'd run the existing
Package.swift file, resulting in us constructing a
Package object named
package, which we'd assign to
EditablePackage's global. Then we'd run the script file in a special context where
EditablePackage is implicitly imported. Once the script is finished, we'd validate and serialise the updated object found in
We get the full flexibility of the
PackageAPI, with no additional maintenance burden translating them in to CLI switches. This would be immensely more powerful, enabling use-cases from integrating packages to configuration scripts (such as those used by SwiftPM, sourcekit-lsp, etc). The possibilities are endless.
It's just Swift. There is one thing to remember, and you already know how it works. It's easy to read and verify that the script does what you expect.
The command accepts a file, rather than a string. That isn't very convenient, since you often want to run some commands without writing a file first.
However, there are ways to use stdin as your input file. The convention is to use a sole
-as the file name. The swift compiler already supports that (clang also supports it, as do other tools like VSCode - it's a super useful feature). For example, if you run:
xcrun swift -
You can just start typing code, with newlines and quotes and everything, and finish off with EOF/CMD+D and the compiler will run it.
For copy/pasting commands, you can use a "heredoc". For example, try copying and pasting the following in to a terminal:
xcrun swift - <<EOF var a = "hello" a.append(", world") print(a) EOF
The best thing? Everybody here can read what this code does.
Some situations can become a little more verbose. Consider as a CLI:
swift package add-dependency https://github.com/apple/swift-argument-parser swift package add-target MyLibrary swift package add-target MyExecutable --no-test-target --dependencies MyLibrary,ArgumentParser swift package add-product MyLibrary --targets MyLibrary
Or as a script:
swift package update-script - <<EOF package.dependencies += .package(url: "https://github.com/apple/swift-argument-parser") package.targets += [ .target(name: "MyLibrary") .executableTarget(name: "MyExecutable", .dependencies: ["MyLibrary", "ArgumentParser"]) ] package.products += .library(name: "MyLibrary", targets: ["MyLibrary"]) EOF
Which is disappointing. The Swift version has bits of ceremony (like
.package(url:...)) which aren't required in the CLI. But the CLI does one operation at a time, so it has a bunch of ceremony of its own (
swift package add-) that the script doesn't have.
And let's not forget that we can add APIs to
EditablePackageto make these scripts smaller. For example, we could add a function to add a dependency and add it as a dependency to some list of targets, in one move. It's a lot easier to do these things if we're writing APIs for Swift scripts using the
Packagetype. Maybe that's also a way to expose things like
Conditional compilation blocks really shouldn't appear in manifests since they'll break cross-compilation, but they'll be preserved during editing. If the entire
Packageinitializer was conditionally compiled you might run into failures
Imperative code is left alone. All the editing operations described only rewrite arguments to the Package initializer
Comments are preserved, although they could be moved some cases. For example, if I add a new target to this list:
[ "one", "two", // comment ]
It's ambiguous whether we should associate the comment with the element "two", or the last element after the addition. Comments not directly adjacent to edits will be unaffected.
- The manifest will be compiled and loaded before committing any edit, and the changes will be rolled back if it fails for any reason.
The main problem here is that we'd have no way to persist arbitrary edits to the manifest as far as I can tell? Side effects, environment variables, and conditional compilation could all introduce branching, and tracing the execution of the script wouldn't be enough to recover all of that information. I think we also disagree on the motivations somewhat: I expect users will use the CLI somewhat often for simple invocations like
swift package add-target MyExecutable --type-executable when they don't remember specific syntax.
I see, so this appears to be a source-level edit. That's interesting.
Right. AFAIK, it is discouraged for Package.swift files to contain this sort of code in the first place (they should be declarative), so people would have to remove them, but we'd have a clear transition strategy: consider your current Package.swift a template and use a configuration script to generate the appropriate Package.swift, using environment-specific edits. It's the same
Package interface so you should be able to pretty-much cut and paste that code in to a new file and have it work.
Right. When weighed against the needs of automation, how much more flexible the scripting approach is, and the significantly lower overall maintenance burden it has, I don't think that use-case justifies the effort.
Editing a Package.swift file isn't hard, even for the biggest projects. It isn't as convenient as having a tool do it for you, but ultimately it's just a Swift file, containing a big, declarative initialiser and no imperative code . You can also create shortcuts or command aliases, write canned scripts for common operations, or we can add APIs to the
EditablePackage module to make those scripts easier.
None of those things are quite as convenient as having a specific tool entry-point for the thing you want to do, but considering the use-cases it serves better against the use-cases it doesn't serve as well, and most importantly, the maintenance burden and possibility of bugs creeping in, I think it's a compelling alternative.
The biggest issue with scripting is that I don’t think it would be possible to preserve comments, at all. And that may actually be a deal-breaker
First of all, love this work. Excited to see these commands come to SwiftPM
I wanted to quickly note that although it has been historically discouraged to use conditional compilation blocks in package manifests, we can’t ignore that there are legitimate reasons to add them. It’s more of an “avoid when unnecessary” than a “this is bad, period.” See this very recent recommendation RE static compilation for Linux, for example: Static linking on Linux in Swift 5.3.1
My point being, it’s important we don’t ignore or intentionally avoid the topic in the context of package edit commands. Based on the above comments, this has been thought about and won’t break.
Out of all of these, the one with the largest value is
add-dependency as it fills a very noticeable gap in SwiftPM compared to its contemporaries, and explaining to primarily iOS engineers who don't have any experience with SwiftPM leaves them scratching their heads or groaning when it's time to add a dependency.
The other two are "nice to have", but I'm more inclined to leave that more to the scripting world like @Karl is suggesting, because I think being able to programmatically edit the Package as needed is useful as part of a whole suite of capabilities with Package Build Tools extensions
That sounds like a good solution! Thanks!
Thanks for the feedback everyone! I've pushed some revisions to the proposal here to add some additional clarification based on the discussion. There's not much in the way of major changes, but I:
- clarified that add-target will support binary targets,
- expanded the alternatives considered section a bit based on the discussions here, and
- included a slightly more complicated example so it's clearer the tools are making syntax-level edits which preserve other comments/formatting/code in the file
This pitch reminds me of PlistBuddy, Apple's tool for reading and editing plist files. I use PlistBuddy to script incrementing the build number in Info.plist files as part of an automated build. It does this by reading the current build number, incrementing it and writing it back.
I don't see anything in this proposal about reading values, only writing them. I haven't used spm so I'm not sure if that's needed but it seems like a useful feature for scripting.
@phoneyDev The existing
swift package describe command is very close to what you're describing, and adding
--type json makes its output machine-readable, so most of the important use cases are covered already.
One thing I am interested in is using the new
PackageSyntax library which powers this proposal to record more source location information for different parts of the manifest. This could be used to improve the
describe output, support inline
Package.swift diagnostics, etc. That's all outside the scope of this proposal though.
This is a super useful feature for spm to have (I was looking for it almost a year ago) and just wanted to thank you for making it actually happen!
This is a great idea, but I'd like to ask for 3 features and one pitch documentation change here:
Support for GitHub repositories with a shortcut on
urlsection which allows me to write something like
swift package add-dependency sunshinejr/SwiftyUserDefaultsand will figure out the URL itself, or alternatively adding the option
--githubto make it more explicit.
Support for adding the GitHub tagline automatically as a comment above each dependency entry to make it easier for people reading the manifest what the dependencies are about, e.g. running
swift package add-dependency sunshinejr/SwiftyUserDefaults --github --taglinewill automatically result in something like this (note also the empty lines between entries in this mode):
dependencies: [ // Modern Swift API for NSUserDefaults .package(name: "SwiftyUserDefaults", url: "https://github.com/sunshinejr/SwiftyUserDefaults.git", .upToNextMajor("3.1.5")), // Delightful console output for Swift developers. .package(name: "Rainbow", url: "https://github.com/onevcat/Rainbow.git", .upToNextMajor("3.1.5")), // A powerful framework for developing CLIs in Swift .package(name: "SwiftCLI", url: "https://github.com/jakeheis/SwiftCLI.git", .upToNextMajor("6.0.1")), ],
Could we also add an option to keep the different array sections sorted by alphabet, like
--sortedbased on the name or alternatively the last portion before the
.gitin the URL?
The new commands section currently does a good job in documenting the different options, but I think it would be great for each section to add at least one full example including a real library URL in the case of
add-dependencyso some things become more clear on first sight.
I’m hesitant to add support for a specific git host like GitHub at the moment, mostly because registry support is likely to land in the near future and a command like you describe should search the user’s configured registries instead of trying to form a git URL IMO.
Sorting might be a reasonable addition. I’d be interested in hearing from people about whether they do this frequently in their own projects? At a glance, it seems like most of the packages in the source compatibility suite don’t.
IMO sorting is too much work to do manually and enforce within a team, but I'd personally appreciate tool support for it.
Given the simplicity of the CocoaPods declarations, we keep those sorted pretty easily. Sorting SPM packages is hard to do manually so I don't do it now, but being able to do it automatically would be appreciated. But sorting is very useful when reading the package and wanting to find a single dependency. In fact, I think it should be the default.
It’s unclear to me how they should be sorted, is it by package name? By owner?
I tend to sort dependencies by purpose or “importance” so that when reviewing dependencies to see if they make sense to keep that there’s some context behind how it’s used in the project
Here’s an example from a shared iOS package:
let dependencies: [Package.Dependency] = [ /* utilities */ .package(url: "https://github.com/apple/swift-algorithms", .upToNextMinor(from: "0.0.2")), /* networking */ .package(name: "Apollo", url: "https://github.com/apollographql/apollo-ios", .upToNextMinor(from: "0.34.1")), // this is "temporarily" needed to support PP5 .package(name: "SocketIO" , url: "https://github.com/socketio/socket.io-client-swift", .upToNextMinor(from: "15.2.0")), /* logging */ .package(url: "https://github.com/apple/swift-log", .upToNextMajor(from: "1.0.0")), /* architecture / design patterns */ .package(url: "https://github.com/quickbirdstudios/XCoordinator", .upToNextMinor(from: "2.0.7")), .package(name: "Lottie", url: "https://github.com/airbnb/lottie-ios", .upToNextMinor(from: "3.1.9")), ]
And then for RediStack I have it sorted by API packages first, then network:
dependencies: [ .package(url: "https://github.com/apple/swift-log.git", from: "1.0.0"), .package(url: "https://github.com/apple/swift-metrics.git", "1.0.0" ..< "3.0.0"), .package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0") ]