[Pitch] SwiftPM support for Swift scripts (revision)

Pitch: SwiftPM support for Swift scripts

Introduction

Swift is a general-purpose language that aims to expand its availability and impact on various domains and platforms. We believe great scripting support and experience is an important part of extending Swift’s usage and improving the impact of Swift as a language. Swift already includes basic support for scripting via the driver. This proposal aims to enable Swift package support in Swift scripts, making scripting support far more powerful than before.

Swift-evolution threads:

Motivation

The Swift standard library maintains a "batteries-not-included" approach and relies on external packages to provide more functionality. However, this introduces friction in writing Swift scripts as there's no easy way to reference external packages from a single file. For example, there's no way to easily include basic scripting utilities like a command-line argument parsing library.

The Swift community has also recognized and tried to provide better scripting support through projects like Marathon and swift-sh. These projects leverage the Swift Package Manager to fetch and build external packages referenced in a Swift script. We believe we can take this approach further and provide a much better scripting experience by improving the integration between the Swift language and the Swift Package Manager. This will also allow Swift users to easily share their scripts without requiring to install additional tooling.

This proposal would dramatically improve the developer experience for writing scripts in Swift. Dependencies for common use-cases would be easy to integrate and scripts would be portable across machines without additional tools installed on the host machine.

Proposed solution

@package syntax

We propose to add a new attribute @package to import statements for referencing Swift packages. This attribute would be only valid when directly running from Swift (via swift MyScript.swift) and will cause an error in general compilations through swiftc and in Swift packages. SwiftPM will resolve the dependency graph, and then fetch, build and expose the correct target to the script. For example,

@package(url: "https://github.com/swift-server/async-http-client.git", from: "1.0.0")
import AsyncHTTPClient

The attribute above indicates the following dependencies:

// Package dependency
.package(url: "https://github.com/swift-server/async-http-client.git", from: "1.0.0")

// Target dependency
.product(name: "AsyncHTTPClient", package: "async-http-client")

The @package(...) syntax will resemble SwiftPM's Package Dependency Description which should feel natural to developers familiar with Swift Package Manager manifest syntax.

swift-script tool in SwiftPM

To speed up the execution of Swift scripts, user-wide cache will be enabled for Swift scripts, including cache for dependencies and binaries. The cache will separate form SwiftPM’s global cache.

We propose to add a swift-script tool for managing script caches. This tool will gain most functions from swift-package, including:

swift script clean <script> <options>
swift script reset <script> <options>
swift script update <script> <options>
swift script resolve <script> <options>
swift script show-dependencies <script> <options>
...

A script will be built using release configuration if it is called by swift MyScript.swift. swift-script provides SwiftPM-like subcommands to allow users switching between build modes for convenience of debugging and other usages.

swift script build <script> <options>
swift script run <script> <options>

Detailed design

@package declaration

In most common cases, users are expected to import a dependency where the product name is equivalent to the imported module name. @package declarations accept exactly the same arguments as Package.Dependency methods.

@package(url: "https://github.com/swift-server/async-http-client.git", from: "1.0.0")
import AsyncHTTPClient

When the product and target names don’t match, the product name must be explicitly specified with an extra products argument which accepts an array of product names.

@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1", products: ["SwiftToolsSupport"])
import TSCBasic
import TSCUtility

By default, Swift will try to apply the package declaration to all the following imports. Although all the modules from imported products are available throughout the whole script, we propose to emit a warning if the import statement isn’t following the @package declaration.

// ✅ warning-free
@package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
import NIO
import NIOHTTP1
import Foundation
// ⚠️ warning: module import from 'swift-tools-support-core' should follow the @package declaration immediately
@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1", products: ["SwiftToolsSupport"])
import TSCBasic
import Foundation
import TSCUtility
// ❌ error: no such module 'NIOHTTP1'
@package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
import NIO
import Foundation
import NIOHTTP1

Currently, redeclaring a package for multiple times is allowed only with exactly the same package description.

// ✅ warning-free
@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1", products: ["SwiftToolsSupport"])
import TSCBasic
import Foundation

@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1", products: ["SwiftToolsSupport"])
import TSCUtility
// ✅ warning-free
@package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
import NIO
import Foundation

@package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
import NIOHTTP1
// ❌ error: declaration of package "swift-tools-support-core" conflicts
@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1", products: ["SwiftToolsSupport"])
import TSCBasic
import Foundation

@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.2", products: ["SwiftToolsSupport"])
import TSCUtility

If a script file containing @package attributes is compiled by swiftc, or inside a Swift package through swift-build and swift-run, the driver will emit an error.

error: @package declaration is not allowed in non-scripting mode

Alternatively, you can use the -ignore-package-declarations flag to ignore them if you want to manually import the libraries.

Building and running a script

The process of building and running a script is very similar to building a package. On swift script build MyScript.swift, SwiftPM will first ask the frontend to parse package declarations, and the frontend will respond with a JSON output. The output is composed of package inferences from the script file:

#!/usr/bin/swift script run

@package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
import NIO
import NIOHTTP1
import Foundation

@package(url: "https://github.com/apple/swift-tools-support-core.git", .upToNextMinor("0.2.0"), products: ["SwiftToolsSupport"])
import TSCBasic
import TSCUtility
{
  "modules": [
    {
      "name": "NIO",
      "line": 4
    },
    {
      "name": "NIOHTTP1",
      "line": 5
    },
    {
      "name": "Foundation",
      "line": 6
    },
    {
      "name": "TSCBasic",
      "line": 9
    },
    {
      "name": "TSCUtility",
      "line": 10
    }
  ],
  "packages": [
    {
      "package": "swift-nio",
      "line": 3,
      "repositoryURL": "https://github.com/apple/swift-nio.git",
      "state": {
        "branch": null,
        "revision": null,
        "version": {
          "version": "2.0.0",
          "resolvingStrategy": "upToNextMajor"
        }
      },
      "declaredDependencies": [
        {
          "product": "NIO",
          "target": "NIO"
        }
      ],
      "possibleDependencies": [
        {
          "product": "NIOHTTP1",
          "target": "NIOHTTP1"
        },
        {
          "product": "Foundation",
          "target": "Foundation"
        }
      ]
    },
    {
      "package": "swift-tools-support-core",
      "line": 8,
      "repositoryURL": "https://github.com/apple/swift-tools-support-core.git",
      "state": {
        "branch": null,
        "revision": null,
        "version": {
          "version": "0.2.0",
          "resolvingStrategy": "upToNextMinor"
        }
      },
      "declaredDependencies": [
        {
          "product": "SwiftToolsSupport",
          "target": "TSCBasic"
        }
      ],
      "possibleDependencies": [
        {
          "product": "SwiftToolsSupport",
          "target": "TSCUtility"
        }
      ]
    }
  ]
}

SwiftPM will resolve package dependencies from the output, and compose a “Package.resolved” file in its unique cache directory. If any module importation is misplaced, SwiftPM will emit a warning.

Then SwiftPM will fetch and build the dependencies, and build the script with -ignore-package-declarations. By default, a script will be built using release configuration, and the -c <configuration> option is available. The build cache will also be placed in the cache directory, to enable fast startup of prebuilt scripts.

swift script run MyScript.swift will execute the script after a successful build, and it prefers the prebuilt cache. You can use swift script clean MyScript.swift to clear the build cache or use swift script reset MyScript.swift to clear all its cache. resolve and update are also available. You can also pass an -update flag to swift script run to update the dependencies before every run (only recommended for background tasks which has a minimal requirement of startup time).

The swift-script tool

The swift-script tool manages build and dependency caches of Swift scripts. The supported subcommands include build, run, clean, reset, resolve and update, which are introduced in the last section. There are some more new subcommands proposed for general management of scripts.

swift script list will dump all the known and cached scripts’ paths, and we propose to calculate an ID for every script which can be a user-wide alternative and short name of its path. swift script cleanup will remove all the caches where the original script no longer exists.

To deal with special cases, we plan to add an -all flag to some of the subcommands. If we pass -all instead of the script path, build, clean, reset, resolve and update will apply its functionality to all the cached scripts. We also propose a -filter <filter> flag to allow narrowing the scope with a filter.

swift shortcut

Currently, swift <path-to-swift-file> is used to interpret a Swift file. With SwiftPM support for scripting, we think it’s reasonable to allow Swift scripts to be invoked directly with the same command. This will add a new step to the driver’s behavior.

Under the script mode (formerly interpret mode), the driver will first ask the frontend to dump package declarations from the input if there is only one. If the result is empty, the driver will keep its original behavior; otherwise, the driver will invoke swift-script run with the result to build and run it as a script.

We also propose to add a -no-script flag to turn down the check and use interpret mode as before.

Creating a package from the script

swift-package will get the ability of creating a package from a script file by --from-script argument. This functionality doesn’t require an actual build. If the script relies on a package with a relative path, or the script is contained within the package directory, the command will fail.

swift package init --from-script ../MyScript.swift

By default, the package created will share the same name with the directory, with the target named after the script file.

// swift-tools-version:5.4

import PackageDescription

let package = Package(
    name: "my-tool",
    products: [
        .executable(
            name: "MyScript",
            targets: ["MyScript"]),
    ],
    dependencies: [
        .package(url: "https://github.com/swift-server/async-http-client.git", from: "1.0.0"),
        .package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1")
    ],
    targets: [
        .target(
            name: "MyScript",
            dependencies: [
                .product(name: "AsyncHTTPClient", package: "async-http-client"),
                .product(name: "SwiftToolsSupport", package: "swift-tools-support-core")
			]
		)
    ]
)

The script file, with all @package declarations removed, will be saved to Sources/MyScript/main.swift.

Source compatibility

This is an additive change with no impact on source compatibility.

Effect on ABI stability and API resilience

The new attribute is only applicable to application code, so there is no effect on ABI stability or API resilience.

Future Directions

Executable from Package

While scripting becomes a new way of exposing a package’s ability to users, there are a large number of existing packages which comes along with an executable target.

If we can call these executables through a command-line tool or inside a script, it will make distribution and installation of Swift-based tools far more convenient.

Alternatives considered

Use #package instead of @package

#package syntax indicates that package dependency is processed at compile time, which is another acceptable approach.

We prefer @package to #package because using an attribute will better declare the relationship between the package and the imported target. Furthermore, @package is more flexible for future performance optimizations.

Reuse current SwiftPM tools for scripts

SwiftPM has already got a full set of tools like swift-package, swift-build and swift-run. It seems reasonable to build scripting support on top of them, so we don’t need any new commands.

Existing tools of SwiftPM is designed specifically for Swift packages, which has lots of difference with scripts. We think extending such tools for scripts will add to unnecessary complexity and cause confusion for users.

Add install subcommand to swift-script

From long ago there are calls for adding installation ability to SwiftPM. With the introduction of scripting support, Swift codes are never so close to being directly invoked from the system shell.

However, installation of scripts includes various factors like permission, directory and relative path changes, which is difficult for SwiftPM to handle. Users are suggested to manually install or link the script (e.g., my-tool) and add the following line to the beginning:

#!/usr/bin/env swift script run

Then it should be able to call the tool simply by my-tool if the parent directory has been added to PATH.

Remove the -all flag

We can remove the -all flag in the swift-script tool and, instead, apply its logic when no input is specified.

We believe that the -all logic is not designed for general usages and should be used with extra care. Therefore, making it the default behavior without any extra flag is improper and may cause serious consequences.

21 Likes

The draft proposal is part of my pre-GSoC work on this idea. All discussions and suggestions are warmly welcomed!

2 Likes

It's great to see some more work in this area! I just have one question for now:

Do you have an implementation strategy in mind for this behavior? If we rely on the fast dependency scanner and explicit modules to find annotated imports and defer to SwiftPM for resolution and building, annotating each individual import will probably be necessary to later construct the right command line for the immediate mode frontend invocation used to execute the script. I think it's important to carefully consider the various options here, because the semantics could have a pretty big impact on the performance of the pre-scan stage for large scripts.

1 Like

This looks good! But there is quite a lot here.

IIUC, GSoC is only 6 weeks (?), and achieving all of this within that time seems ambitious. The shared cache (certainly user-local rather than system-wide), the swift script commands, package creation, etc - these are all useful additions, but I don't think any of them are necessary for an MVP.

We've had a lot of very exciting GSoC projects over the years by talented students, but from what I've observed, many of those projects were ultimately never merged. Sometimes it's better to start small.

The football team I support have a kind of slogan. It's a quote from a former manager which sums up the club's philosophy quite well:

“It is better to fail aiming high than to succeed aiming low. And we of Spurs have set our sights very high, so high in fact that even failure will have in it an echo of glory.”

It's rubbish. Don't be like Spurs. It's better to start with something contained, achievable, and get it accepted/merged so that you've at least advanced the state-of-the-art by a little bit. Then build from that.

8 Likes

As for GSoC, the goal is to demonstrate a workable model. Things like swift-script, IMO, are for post-GSoC period because these parts are a lot easier to implement and largely depend on APIs instead of models.

I wish to discover the most suitable working logic with the whole community before GSoC officially starts. You’re right that GSoC works are likely to be abandoned, and I think picking a proper model before everything starts is the key to preventing this project from such ending. The original proposal didn’t touch on that.

1 Like

I’d expect feedbacks from this thread that can help improve both the core logic and the design details, so I can finalize the proposal and complete the implementation based on my GSoC project after GSoC ends.

Frankly speaking, I didn’t look very deep into the performance. The proposed design is theoretically workable, and I initially plan to rely on caching to speed up startups (which doesn’t need parsing).

However, I’ll be very glad if there is a better general model. The proposed syntax is very new and there’re very few existing implementations for reference here.

P.S. I’m looking forward to working out better semantics (I haven’t started coding yet, the current one is only conceptual) with the mentors during GSoC. That would be a meaningful experience I believe.

Thanks for putting this together @stevapple

is this something that could be done with SwiftSyntax instead of the frontend, and would that be a simpler architecture? #3230 by @owenv is looking to add a dependency on SwiftSyntax anyways

this feels like we are casting a rather large net. would it be useful to limit the scope of the proposal to the the key command(s)? maybe even only "run"? we can always add more commands in subsequent proposal if we think they are needed.

A SwiftSyntax based approach would probably work, but I think integrating with the frontend's dependency scanner has at least a couple advantages:

  • It can take advantage of delayed function body parsing for longer scripts, and when libSwiftScan is available it can load the library directly to avoid JSON serialization and deserialization, similar to SwiftSyntax
  • The new driver already has support for generating explicit module build jobs based on the scanner's output before execing an immediate mode script. Adding a new dependency type for packages here would keep the driver in control of the build process because it wouldn't need to defer to SwiftPM when running a script without dependencies.
2 Likes

I agree with @owenv’s opinions. Initially I decide to put it in the frontend because of SourceKit integration, and because this would allow the driver to interpret a script without package dependencies individually.

Since caching is critical to allowing the prebuilt script to be immediately called as a tool, I ended up finding a full set of cache management tools is also vital. If there’s not, users would not have an easy way to handle the caches, which are very likely to become wastes someday.

SwiftSyntax may also be useful because the “Create package from script” functionality needs to remove all the @package attributes. This is only needed in SwiftPM, which is very possible to use SwiftSyntax for. (In spite that this isn’t the core and has little to do with GSoC)

IMO, this would be an acceptable limitation for an MVP. You don’t often need to import multiple modules from a package, and if you do, duplicating the package line a few times isn’t such an enormous burden.

I think it’s better to have the thing functional, then try to optimise the syntax burden later. It would also be reasonable to limit package imports to the start of the file, if that makes it easier.

1 Like

on the surface, the prebuilt scripts cache managment aspect feels like crossing a line into tools management - something that could potentially be achieved using hombrew or similar.

fwiw, I think tools management is an interesting future direction for SwiftPM - for both tools available as scripts or full-fledged packages, but it feels like a separate proposal.

would it be beneficial to list out the tradeoffs of prebuilt scripts vs. ones that are built on-demand? performance is obviously is a key aspect, but would be interesting to weight that against complexity and alternative solutions for the tools management aspect.

1 Like

Swift packages (especially “heavy” ones) are built relatively slow. If there’s no cache for a script which depends on other packages, the packages will need to be built every time we want to execute the script. I’m afraid this is unacceptable for debugging and testing, and will also harm its usability as a tool.

Caching, on the other side, needs a set of compatible management tools and occupies extra disk space, but it’s definitely worth it. I don’t think it will add much to complexity.

I think the scripting proposal actually means to focus on this area. I’ve put the package side to future directions.

I've been playing with a project where this would be incredibly beneficial and would love for something like this to be available!

I like the proposed API and think it's a great starting point for something that should evolve over time. Just having access to things like ArgumentParser will make it much more likely for me to use Swift scripts :slight_smile:

That is a great pitch.

There are some occasions where a Swift script would make more sense, but I use a full blown Xcode project for a <100 lines of code script just because I want to use a small Package to parse a DSL, setup a n automated process or just get some constants. It would be beneficial for micro libraries to be used in Swift scripts (instead of copying the code or creating a Xcode Command Line Project).

Thanks for putting this together, it would be an awesome addition to the ecosystem.

A note on wording : it feels weird to me calling @package an ‘attribute on import’ since it can apply on several imports. I would call it a package (dependency) statement.

The MVP, for me, would be the following:

  • @package and import at the top of the file
  • swift script run <script.swift> (and possibly the new behavior of swift <script.swift>)
  • swift script cleanup <script.swift>

The rest is cherry on top.

The basic goal will be allowing @package and imports at anywhere, but @package may not fall through to subsequent imports. This would be a higher goal that I’ll try my best to work it out.

Did you mean swift script reset in the proposal?

I’m a bit hesitant about the spellings of clean, reset and cleanup. This draft uses clean and reset as-is in Swift packages, but the case seems very different with scripts. If you have any new idea, welcome to bring it up for discussion.

My bad. I do mean swift script reset in the proposal. Also swift script clean, supposing it’s not much more work.

I think clean and reset are not bad. To me it doesn’t seem so different from packages.

Overall I'm a massive +1 for pitch. I use Swift for almost all of my scripting these days mainly due to code reuse and it's a language I actually know and am comfortable in.

There are some significant pain points which this pitch should solve. The first is obviously pulling in code from anything other than Foundation. I end up duplicating the same lines of code across every script I write. So whilst I would love to see improvements in the way Swift handles interacting with external processes I'm happy to just wrap that in a library for now.

One outstanding question I have is about build artefacts and package resolution. SwiftPM relies on Package.resolved to work out what versions need to be used so are scripts now going to have an accompanying resolved file?

Additionally, in terms of build artefacts for scripts - will we have a .build directory for scripts that contain built modules? And how will that work when we have multiple scripts in the same directory, or in the same directory as a real package? (That question also applies to Package.resolved files as well). I'm hoping that the coming 5.4 release for a shared SwiftPM cache on the system should improve resolution times but that doesn't apply to build artefacts (yet) so that needs to be considered.

1 Like
Terms of Service

Privacy Policy

Cookie Policy