Pitch: SwiftPM support for Swift scripts
- Proposal: SE-NNNN
- Authors: Rahul Malik, Ankit Aggarwal, David Hart, YR Chen
- Review Manager: TBD
- Status: Awaiting implementation
Introduction
Swift is a general-purpose language that aims to expand its availability and impact on various domains and platforms. We believe great scripting support and experience is an important part of extending Swift’s usage and improving the impact of Swift as a language. Swift already includes basic support for scripting via the driver. This proposal aims to enable Swift package support in Swift scripts, making scripting support far more powerful than before.
Swift-evolution threads:
Motivation
The Swift standard library maintains a "batteries-not-included" approach and relies on external packages to provide more functionality. However, this introduces friction in writing Swift scripts as there's no easy way to reference external packages from a single file. For example, there's no way to easily include basic scripting utilities like a command-line argument parsing library.
The Swift community has also recognized and tried to provide better scripting support through projects like Marathon and swift-sh. These projects leverage the Swift Package Manager to fetch and build external packages referenced in a Swift script. We believe we can take this approach further and provide a much better scripting experience by improving the integration between the Swift language and the Swift Package Manager. This will also allow Swift users to easily share their scripts without requiring to install additional tooling.
This proposal would dramatically improve the developer experience for writing scripts in Swift. Dependencies for common use-cases would be easy to integrate and scripts would be portable across machines without additional tools installed on the host machine.
Proposed solution
@package syntax
We propose to add a new attribute @package to import statements for referencing Swift packages. This attribute would be only valid when directly running from Swift (via swift MyScript.swift) and will cause an error in general compilations through swiftc and in Swift packages. SwiftPM will resolve the dependency graph, and then fetch, build and expose the correct target to the script. For example,
@package(url: "https://github.com/swift-server/async-http-client.git", from: "1.0.0")
import AsyncHTTPClient
The attribute above indicates the following dependencies:
// Package dependency
.package(url: "https://github.com/swift-server/async-http-client.git", from: "1.0.0")
// Target dependency
.product(name: "AsyncHTTPClient", package: "async-http-client")
The @package(...) syntax will resemble SwiftPM's Package Dependency Description which should feel natural to developers familiar with Swift Package Manager manifest syntax.
swift-script tool in SwiftPM
To speed up the execution of Swift scripts, user-wide cache will be enabled for Swift scripts, including cache for dependencies and binaries. The cache will separate form SwiftPM’s global cache.
We propose to add a swift-script tool for managing script caches. This tool will gain most functions from swift-package, including:
swift script clean <script> <options>
swift script reset <script> <options>
swift script update <script> <options>
swift script resolve <script> <options>
swift script show-dependencies <script> <options>
...
A script will be built using release configuration if it is called by swift MyScript.swift. swift-script provides SwiftPM-like subcommands to allow users switching between build modes for convenience of debugging and other usages.
swift script build <script> <options>
swift script run <script> <options>
Detailed design
@package declaration
In most common cases, users are expected to import a dependency where the product name is equivalent to the imported module name. @package declarations accept exactly the same arguments as Package.Dependency methods.
@package(url: "https://github.com/swift-server/async-http-client.git", from: "1.0.0")
import AsyncHTTPClient
When the product and target names don’t match, the product name must be explicitly specified with an extra products argument which accepts an array of product names.
@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1", products: ["SwiftToolsSupport"])
import TSCBasic
import TSCUtility
By default, Swift will try to apply the package declaration to all the following imports. Although all the modules from imported products are available throughout the whole script, we propose to emit a warning if the import statement isn’t following the @package declaration.
// ✅ warning-free
@package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
import NIO
import NIOHTTP1
import Foundation
// ⚠️ warning: module import from 'swift-tools-support-core' should follow the @package declaration immediately
@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1", products: ["SwiftToolsSupport"])
import TSCBasic
import Foundation
import TSCUtility
// ❌ error: no such module 'NIOHTTP1'
@package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
import NIO
import Foundation
import NIOHTTP1
Currently, redeclaring a package for multiple times is allowed only with exactly the same package description.
// ✅ warning-free
@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1", products: ["SwiftToolsSupport"])
import TSCBasic
import Foundation
@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1", products: ["SwiftToolsSupport"])
import TSCUtility
// ✅ warning-free
@package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
import NIO
import Foundation
@package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
import NIOHTTP1
// ❌ error: declaration of package "swift-tools-support-core" conflicts
@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1", products: ["SwiftToolsSupport"])
import TSCBasic
import Foundation
@package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.2", products: ["SwiftToolsSupport"])
import TSCUtility
If a script file containing @package attributes is compiled by swiftc, or inside a Swift package through swift-build and swift-run, the driver will emit an error.
error: @package declaration is not allowed in non-scripting mode
Alternatively, you can use the -ignore-package-declarations flag to ignore them if you want to manually import the libraries.
Building and running a script
The process of building and running a script is very similar to building a package. On swift script build MyScript.swift, SwiftPM will first ask the frontend to parse package declarations, and the frontend will respond with a JSON output. The output is composed of package inferences from the script file:
#!/usr/bin/swift script run
@package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
import NIO
import NIOHTTP1
import Foundation
@package(url: "https://github.com/apple/swift-tools-support-core.git", .upToNextMinor("0.2.0"), products: ["SwiftToolsSupport"])
import TSCBasic
import TSCUtility
{
"modules": [
{
"name": "NIO",
"line": 4
},
{
"name": "NIOHTTP1",
"line": 5
},
{
"name": "Foundation",
"line": 6
},
{
"name": "TSCBasic",
"line": 9
},
{
"name": "TSCUtility",
"line": 10
}
],
"packages": [
{
"package": "swift-nio",
"line": 3,
"repositoryURL": "https://github.com/apple/swift-nio.git",
"state": {
"branch": null,
"revision": null,
"version": {
"version": "2.0.0",
"resolvingStrategy": "upToNextMajor"
}
},
"declaredDependencies": [
{
"product": "NIO",
"target": "NIO"
}
],
"possibleDependencies": [
{
"product": "NIOHTTP1",
"target": "NIOHTTP1"
},
{
"product": "Foundation",
"target": "Foundation"
}
]
},
{
"package": "swift-tools-support-core",
"line": 8,
"repositoryURL": "https://github.com/apple/swift-tools-support-core.git",
"state": {
"branch": null,
"revision": null,
"version": {
"version": "0.2.0",
"resolvingStrategy": "upToNextMinor"
}
},
"declaredDependencies": [
{
"product": "SwiftToolsSupport",
"target": "TSCBasic"
}
],
"possibleDependencies": [
{
"product": "SwiftToolsSupport",
"target": "TSCUtility"
}
]
}
]
}
SwiftPM will resolve package dependencies from the output, and compose a “Package.resolved” file in its unique cache directory. If any module importation is misplaced, SwiftPM will emit a warning.
Then SwiftPM will fetch and build the dependencies, and build the script with -ignore-package-declarations. By default, a script will be built using release configuration, and the -c <configuration> option is available. The build cache will also be placed in the cache directory, to enable fast startup of prebuilt scripts.
swift script run MyScript.swift will execute the script after a successful build, and it prefers the prebuilt cache. You can use swift script clean MyScript.swift to clear the build cache or use swift script reset MyScript.swift to clear all its cache. resolve and update are also available. You can also pass an -update flag to swift script run to update the dependencies before every run (only recommended for background tasks which has a minimal requirement of startup time).
The swift-script tool
The swift-script tool manages build and dependency caches of Swift scripts. The supported subcommands include build, run, clean, reset, resolve and update, which are introduced in the last section. There are some more new subcommands proposed for general management of scripts.
swift script list will dump all the known and cached scripts’ paths, and we propose to calculate an ID for every script which can be a user-wide alternative and short name of its path. swift script cleanup will remove all the caches where the original script no longer exists.
To deal with special cases, we plan to add an -all flag to some of the subcommands. If we pass -all instead of the script path, build, clean, reset, resolve and update will apply its functionality to all the cached scripts. We also propose a -filter <filter> flag to allow narrowing the scope with a filter.
swift shortcut
Currently, swift <path-to-swift-file> is used to interpret a Swift file. With SwiftPM support for scripting, we think it’s reasonable to allow Swift scripts to be invoked directly with the same command. This will add a new step to the driver’s behavior.
Under the script mode (formerly interpret mode), the driver will first ask the frontend to dump package declarations from the input if there is only one. If the result is empty, the driver will keep its original behavior; otherwise, the driver will invoke swift-script run with the result to build and run it as a script.
We also propose to add a -no-script flag to turn down the check and use interpret mode as before.
Creating a package from the script
swift-package will get the ability of creating a package from a script file by --from-script argument. This functionality doesn’t require an actual build. If the script relies on a package with a relative path, or the script is contained within the package directory, the command will fail.
swift package init --from-script ../MyScript.swift
By default, the package created will share the same name with the directory, with the target named after the script file.
// swift-tools-version:5.4
import PackageDescription
let package = Package(
name: "my-tool",
products: [
.executable(
name: "MyScript",
targets: ["MyScript"]),
],
dependencies: [
.package(url: "https://github.com/swift-server/async-http-client.git", from: "1.0.0"),
.package(url: "https://github.com/apple/swift-tools-support-core.git", from: "0.0.1")
],
targets: [
.target(
name: "MyScript",
dependencies: [
.product(name: "AsyncHTTPClient", package: "async-http-client"),
.product(name: "SwiftToolsSupport", package: "swift-tools-support-core")
]
)
]
)
The script file, with all @package declarations removed, will be saved to Sources/MyScript/main.swift.
Source compatibility
This is an additive change with no impact on source compatibility.
Effect on ABI stability and API resilience
The new attribute is only applicable to application code, so there is no effect on ABI stability or API resilience.
Future Directions
Executable from Package
While scripting becomes a new way of exposing a package’s ability to users, there are a large number of existing packages which comes along with an executable target.
If we can call these executables through a command-line tool or inside a script, it will make distribution and installation of Swift-based tools far more convenient.
Alternatives considered
Use #package instead of @package
#package syntax indicates that package dependency is processed at compile time, which is another acceptable approach.
We prefer @package to #package because using an attribute will better declare the relationship between the package and the imported target. Furthermore, @package is more flexible for future performance optimizations.
Reuse current SwiftPM tools for scripts
SwiftPM has already got a full set of tools like swift-package, swift-build and swift-run. It seems reasonable to build scripting support on top of them, so we don’t need any new commands.
Existing tools of SwiftPM is designed specifically for Swift packages, which has lots of difference with scripts. We think extending such tools for scripts will add to unnecessary complexity and cause confusion for users.
Add install subcommand to swift-script
From long ago there are calls for adding installation ability to SwiftPM. With the introduction of scripting support, Swift codes are never so close to being directly invoked from the system shell.
However, installation of scripts includes various factors like permission, directory and relative path changes, which is difficult for SwiftPM to handle. Users are suggested to manually install or link the script (e.g., my-tool) and add the following line to the beginning:
#!/usr/bin/env swift script run
Then it should be able to call the tool simply by my-tool if the parent directory has been added to PATH.
Remove the -all flag
We can remove the -all flag in the swift-script tool and, instead, apply its logic when no input is specified.
We believe that the -all logic is not designed for general usages and should be used with extra care. Therefore, making it the default behavior without any extra flag is improper and may cause serious consequences.
