Swift Run scripts

Development Package Manager
1

Hey there everyone, I'm building a medium sized developer tool using SwiftPM (danger-swift) and am starting to collect a lot of single line commands for my project. These range from using my dependencies:

swift run sourcedocs generate --spm-module Danger --output-folder docs/reference

to building and installing the tool

swift build --disable-sandbox -c release --static-swift-stdlib

and then some useful snippets during deployment

get_sha:
	wget https://github.com/danger/$(TOOL_NAME)/archive/$(VERSION).tar.gz -O $(TAR_FILENAME)
	shasum -a 256 $(TAR_FILENAME)
	rm $(TAR_FILENAME)

these all live in a Makefile today, but I spend most of my day inside node and really miss the "scripts" section of the package.json. This is basically a section of composable smaller scripts (and hook-in points for custom commands during the dependency management) which lets you consolidate your CLI commands into a single place.

I was wondering if folks were interested in something like this:


let package = Package(
    name: "danger-swift",
    products: [
        .library(name: "Danger", type: .dynamic, targets: ["Danger"]),
        .executable(name: "danger-swift", targets: ["Runner"])
    ],
    dependencies: [
        .package(url: "https://github.com/JohnSundell/Marathon.git", from: "3.1.0"),
        // Dev dependencies
        .package(url: "https://github.com/eneko/SourceDocs.git", from: "0.5.1"),
        .package(url: "https://github.com/realm/SwiftLint.git", from: "0.28.1"),
        .package(url: "https://github.com/nicklockwood/SwiftFormat.git", from: "0.35.7"),
    ],
    scripts: [
      .command(name:"docs", "sourcedocs generate --spm-module Danger --output-folder Documentation/reference"),
      .command(name: "format", "swiftformat"),
      .command(name: "lint", "swiftlint"),
      .command(name: "ci", ["lint", "format"]),
      .command(name: "deploy", "node --require ts-node/register scripts/deploy.ts")
   ]
 })

Which would allow:

// (already exists)
// compile the app, and run it
swift run danger-swift
// compile swiftlint, and run it
swift run swiftlint 

// (new)
// effectively running "swift run swiftlint"
swift run lint 
// runs both swift run lint, and then swift run format
swift run ci 
// runs "swift run  sourcedocs generate --spm-module Danger --output-folder docs/reference"
swift run docs
// Runs the command
swift run deploy

Are folks interested?

32 Likes
2

This is something that the JavaScript folks have been doing for a while now. I think it's a worry addition to SwiftPM if it's not already there.

3

This is definitely interesting. I guess the advantage over makefile is that we can keep everything contained in SwiftPM? I do wonder if these type of custom command belong outside the manifest file because they're not really required to configure a package.

1 Like
4

I think it would be useful to allow specifying scripts in the manifest. I would hate to see the SPM keep itself limited to just describing dependencies and how something is built. A lot of times it's useful for big projects to define a multitude of different house-keeping jobs, ways of compiling, running, etc. And keeping everything in the manifest makes it easier for devs to see everything about what they're working with.

5

Should users be able to specify the commands composed with the double pipe or double ampersand?

6

I would say that without that, the feature would be extremely limiting as to be almost unusable for even basic things. A lot of the time a "script" is actually just a combination of other "scripts".

7

I don't think @Aciid meant to not include the feature per se, but rather that it might just not be specified inside the Package.swift file. I can imagine that there could be a Commands.swift file (just some strawman name) or similar where those commands could be placed.

2 Likes
8

BTW, a similar topic has come up before:

Even though this is more about running scripts after certain events, a solution could combine all those features.

1 Like
9

Yes my comment wasn't very clear. I was arguing against that kind of pattern. For one it seems odd that scripts/commands would be its own file but not anything else related to SPM manifests. If you did have a Commands.swift, why not allow a Dependencies.swift? Another reason I would argue against separate files is that Package.swift no longer describes all about a particular package. Whereas I believe that a proper manifest file should encompass everything about a specific package, not only how it's built and its products.

On the flip side I could kind of see why you would want seperate files for these kinds of things. Whereas npm just uses a plan .json file for its manifest, we have a .swift one that runs in an interpreter. I'll try not to rekindle the json/yaml/etc vs Swift manifest debate, but using a .swift files does mean we have to be more careful around adding too many things to the package constructor, less we make it even harder for people to remember the correct order of the init.

I wonder if we could compromise and extend Package.swift to allow defining a Commands type that SPM can look for?

2 Likes
10

When thinking about a simple example like a currency library, I can imagine there being a .gyb file which would produce a .swift file with all the currencies of the world as structs. Now processing this gyb file would be a classic example of a script that should be included in such a library. As such it would make sense to me to include it in the Package.swift file in a separate section.

In an application there could be a script to update the version numbers for all targets or similar (super-made-up dummy example). Come to think of it, there could actually be user-specific scripts outside the current Swift project but I digress…

I am not overly familiar how npm scripts are used, but it seems to me that there are two kind of scripts.
In a library, the scripts would be more or less part of the package, while in an application the scripts are somewhat specific shortcuts to commonly used commands pertaining to this specific app. Is this something that is expressed in npm or even worth distinguishing?

11

While that's a good point, I think the convenience of having it inside the Package file trumps it. I'm very happy to use the scripts section in node packages.

1 Like
12

I don't think this is really something worth distinguishing. "scripts" are best thought of as "anything you would need to do in a terminal, but automated". So you could have libraries that define scripts to publish a new version of the lib that runs the tests, does any extra work to setup before publishing, and then publish.

13

Personally, I would much rather like a Swift equivalent to Ruby's Rake (or plain Makefiles). I'm somewhat not convinced of the value of coupling a task runner to the package manager.

1 Like
14

The interesting thing here is that the Package.swift is both lib reference (Xcode target/Podspec/Gemspec) and project reference (Xcodeproj/Gemfile/Podfile), which means there might always be a bit of a tension between "is this useful when having the package.swift for building as a lib" vs "is this useful when it's the root of a project and needs to define project metadata"

This is really only useful for the latter case

In NPM, no this doesn't exist. How some of those types of projects handle it is by allowing a lot of the configuration to live inside the project definition file - I built something like this with PackageConfig - then you just need to run the main command for whatever tool via the nom run command.

This is quite an interesting idea, the two domains could be completely separated:

let package = Package(
    name: "danger-swift",
    products: [
        .library(name: "Danger", type: .dynamic, targets: ["Danger"]),
        .executable(name: "danger-swift", targets: ["Runner"])
    ],
    dependencies: [
        .package(url: "https://github.com/JohnSundell/Marathon.git", from: "3.1.0"),
        // Dev dependencies
        .package(url: "https://github.com/eneko/SourceDocs.git", from: "0.5.1"),
        .package(url: "https://github.com/realm/SwiftLint.git", from: "0.28.1"),
        .package(url: "https://github.com/nicklockwood/SwiftFormat.git", from: "0.35.7"),
    ],
 })

let project = Project(
    scripts: [
      .command(name:"docs", "sourcedocs generate --spm-module Danger --output-folder Documentation/reference"),a
      .command(name: "format", "swiftformat"),
      .command(name: "lint", "swiftlint"),
      .command(name: "ci", ["lint", "format"]),
      .command(name: "deploy", "node --require ts-node/register scripts/deploy.ts")
   ], 
    config: [
      .setting(name: "swiftlint", ["ignore":["MyModule"]]) 
   ]
)

and maybe that's more extensible and lean into the separation completely?

1 Like
15

Oh, I missed this example before. I think as soon as we start composing these commands, this feature starts looking more like a task runner. A task runner can become a big project of its own but maybe that is the right direction for such custom commands? We already have a way of writing powerful build systems using llbuild and I think it'll be great to write a Swift DSL on top of it for running custom commands. I am not sure if such a task runner should be part of SwiftPM or not though.

1 Like
16

Having a task runner bundled with swift would be really nice.
It could be a side project to SPM, even though the latter would need to be aware of it so that we can bundle our tasks along our dependencies.
In fact SPM may even use it internally to define its own task (resolve, edit, dump, ...).

Didn't test it but Beak approach to define tasks/commands seems interesting. Plus side IMO:

  • It's not text based
  • We can execute shell commands or swift pure code :)
  • A lot of possibilites (arguments, optional arguments, ...)
1 Like
17

I would love this functionality in SPM. I'm playing around in the node ecosystem right now and am finding a task runner to be a big boon to productivity. As far as I'm concerned it's a big +1 from me.

1 Like
18

I think it is dubious to have SPM to act as a task manager. Unless it will grow to systems like Gradle, where you can basically build the whole pipeline using their plugin APIs. It is not clear to me whether this is a development vector for SPM.

1 Like
19

In general I'm not sure how I feel about this, but with specific regard to what Rust calls build scripts I would be very much in favor. It is very helpful for metaprogramming (Make, CMake, and Autotools also easily support this).

To not have build scripts in SPM seems like a huge oversight to me.

20

I think we have to distinguish two features here:

  • running custom code as part of the build process. We have a draft proposal for this here: Package Manager Extensible Build Tools
  • specifying tasks which can be run manually outside of the build process, this is what @orta is proposing here.

These might seem similar, but I think we'd intentionally want to separate them in SwiftPM. We want to be much more strict about what happens during the build process due to the reasons outlined in the extensible build tools draft proposal, but tasks which are run manually do not need to meet the same high bar.