SwiftPM support for Swift scripts

Evolution Pitches
1

SwiftPM support for Swift scripts

Introduction

Swift is a general-purpose language that aims expand it's availability and impact on various domains and platforms. We believe great scripting support and experience is an important part of improving the impact of Swift as a language. Swift already includes basic support for scripting via the Swift command-line tool. This is a proposal for greatly improving the script support by providing a deeper integration with the Swift Package Manager.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation

The Swift standard library maintains a "batteries-not-included" approach and relies on external packages to provide more functionality. However, this introduces friction in writing Swift scripts as there's no easy way to reference external packages from a single file. For example, there's no way to easily include basic scripting utilities like a command-line argument parsing library.

The Swift community has also recognized and tried to provide better scripting support through projects like Marathon and swift-sh. These projects leverage the Swift Package Manager to fetch and build external packages referenced in a Swift script. We believe we can take this approach further and provide a much better scripting experience by improving the integration between the Swift language and the Swift Package Manager. This will also allow Swift users to easily share their scripts without requiring to install additional tooling.

This proposal would dramatically improve the developer experience for writing scripts in Swift. Dependencies for common use-cases would be easy to integrate and scripts would be portable across machines without additional tools installed on the host machine.

Proposed solution

We propose to add a new attribute @package to import statements for referencing Swift packages. This attribute would be only valid when running in the Swift interpreter (via swift MyScript.swift or swift run MyScript.swift) and will be rejected if used during compilation of a regular Swift module or the Swift REPL. Swift Package Manager will then fetch and build the declared packages and make them available to the script when executing it.

@package(url: "https://github.com/jpsim/Yams.git", from: "2.0.0")
import Yams

This script can be executable by either swift MyScript.swift or swift run MyScript.swift which will utilize SwiftPM's dependency fetching and resolution logic.

The @package(...)syntax will allow the same parameters as SwiftPM's Package Dependency Description which should feel natural to developers familiar with Swift Package Manager manifest syntax.

Detailed design

Basic usage

The most common case is expected to be importing a dependency where the product name is equivalent to the imported module name.

@package(url: "https://github.com/jpsim/Yams.git", from: "2.0.0")
import Yams

When a dependency product and target names do not match, the product must be explicitly specified.

For an example, lets look at the Package description below for swift-tools-support-core.

let package = Package(
    name: "swift-tools-support-core",
    products: [
        .library(
            name: "SwiftToolsSupport",
            type: .dynamic,
            targets: ["TSCBasic", "TSCUtility"]),
    ],
    // Rest of Package.swift truncated for brevity .....
}

With the above package description, the SwiftToolsSupport product does not match a target within the package description so it must be specified explicitly.

@package(url: "https://github.com/apple/swift-tools-support-core.git", .exact("0.0.1"), products: ["SwiftToolsSupport"])
import TSCBasic 
import TSCUtility

Dependency version requirements

The @package(..) annotation would also allow a developer to specify requirements to ensure your scripts are reproducible. The syntax will contain the same level of control and specificity as PackageDependencyDescription.Requirement which should feel like a natural to developers familiar with Swift Package Manager manifest syntax.

Revision

@package(url: "https://github.com/jpsim/Yams.git", .revision("5fa313eae1ca127ad3c706e14c564399989cb1b1")) 
import Yams

Branch

@package(url: "https://github.com/jpsim/Yams.git", .branch("releases/1.0")) 
import Yams

Exact Version

@package(url: "https://github.com/jpsim/Yams.git", .exact("2.0.0")) 
import Yams

Range of version

@package(url: "https://github.com/jpsim/Yams.git", from: "1.0.0") 
import Yams

Local dependency
Absolute path

@package(path: "/Users/username/Yams") 
import Yams

Relative path

@package(path: "dependencies/Yams") 
import Yams

Integration with Swift Compiler

The @package() syntax will need to be added to the AST definition and compiler front-end will require changes to parse this syntax.

Integration with SwiftPM

Execution

Scripts that utilize the @package() attribute can be executed by swift run and managed by SwiftPM.

SwiftPM will utilize the changes to the compiler front-end to access packages referenced using the @package attribute.

Package resolution

Once package dependencies are known, SwiftPM will use it's existing logic for resolving those dependencies and outputting a ".resolved" file to ensure future invocations are reproducible.

The name of the resolved file will be script_name.resolved.

Built products

Products built from scripts will be located in common per-user location ~/.swiftpm/scripts/...

Alternatives considered

Global package dependencies

One alternative could be adding a feature to globally install packages that can be referenced by any script. This is usually a bad idea because it negatively affects the portability of our scripts.

87 Likes
2

I love it :heart_eyes:

I do think the syntax could be a little shorter, though. Is there really a need for explicit url/path parameter labels?

If there's any way that we could shorten the URLs themselves (github://user/repo?), that would also be great. Personally, I feel swift-sh goes a little far towards abbreviating the syntax, but it's the right idea.

2 Likes
3

Would one need to specify the @package attribute multiple times if they wanted to import several modules defined in the same package?

To be more concrete, in your example with swift-tools-support-core, if I wanted to import both TSCBasic and TSCUtility, would I have to do this?


@package(url: "https://github.com/apple/swift-tools-support-core.git", .exact("0.0.1"), products: ["SwiftToolsSupport"])
import TSCBasic

@package(url: "https://github.com/apple/swift-tools-support-core.git", .exact("0.0.1"), products: ["SwiftToolsSupport"])
import TSCUtility
1 Like
4

I like that this proposal suggests using the same syntax as in the package manifest — that means a suggestion like yours could be pitched against SwiftPM as a whole, not specifically for this feature.

9 Likes
5

I'll update the proposal but the short answer is you only need one declaration. With the product name specified you have access to the underlying declared targets

1 Like
6

Really like the direction this pitch is going in!

1 Like
7

That's a good point - we could probably do without them (the url:/path: labels) in regular Package.swift files, too. You don't write or edit them very often, so I guess it wasn't worth anybody's time before now; but this opens up new use-cases. I guess the Github package repo support that was announced at WWDC will also open up more concise syntax options.

In any case, it's a source-compatible change we could make in a follow-up proposal.

8

It's great to see improvements in this area! My one concern is that if this is an attribute on import, the compiler might need to scan all of the top level declarations in the script before deciding if it can interpret it directly or needs to hand things off to SwiftPM. I suppose it might be possible to modify the fast dependency scanner to do this quickly though, especially since most scripts aren't that long.

Also, is the attribute intended to be supported in the REPL? I would assume not but it might be nice to clarify that in the proposal.

9

It is not intended to be used within the REPL, I'll add this to the proposal

1 Like
10

This feel more like a # than an attribute.

#package(url: "https://github.com/apple/swift-tools-support-core.git", .exact("0.0.1"), products: ["SwiftToolsSupport"])

import TSCUtility
import TSCBasic

Can you also clarify the role of stand alone files using the following? #!/usr/bin/swift sh

#!/usr/bin/swift sh
@package(url: "https://github.com/apple/swift-tools-support-core.git", .exact("0.0.1"), products: ["SwiftToolsSupport"])

import TSCUtility
import TSCBasic
5 Likes
11

Can you elaborate on a bit more on why?

12

I prefer attaching the attribute to the import statement because we can optimize for the common case where the product name is same as the module name and SwiftPM can automatically do the right thing and only resolve the products that are actually used in the script (see SE-0226).

3 Likes
13

Attributes are usually attached to something but I don't see the need for a package import to be attached to the import statement. Maybe if we did something like the below

#!/usr/bin/swift sh
@package(url: "https://github.com/apple/swift-tools-support-core.git", .exact("0.0.1"), products: ["SwiftToolsSupport"])
import TSCUtility, TSCBasic

Otherwise if we use the # it signals better to me that this is a "run time" behavior.

1 Like
14

+1 for thinking more about scripting. Do we want to ensure we can share the same source files between scripting and larger-scale projects?

15

I'm a big +1 on this.

I have been using swift-sh regularly for the last year and it's great to see SPM evolve this way.
Another alternative you might have forgot is Beak.

Something that happened often is that some part of my script is used in my app/cli/swift module so I have made libraries but It doesn't always feel right.

Maybe it could be great to allow script to compile in regular Swift module, and even provide a define like SPM_SCRIPTING_MODE that will exclude code from being build in the swift module/app but will always work in script. If you could ignore #!/usr/bin/swift too will be awesome.

#!/usr/bin/swift
import Foundation

func findTheTruthOfTheUniverse(_ args : [String]) {
  print("Hello world")
}

#if SPM_SCRIPTING_MODE
findTheTruthOfTheUniverse(CommandLine.arguments)
#end
1 Like
16

RE suggestions that the script also be usable in a full fledged project: I don’t see the need. If something is shared between a script and a full project, can’t it live in a module imported by both? I can see the utility in being able to migrate from a script to a full project (a use-case supported by swift-sh).

17

I do prefer have declaration on the top like you propose because I think the intent is more clear.

Having @package(url:...) above each import clutter the code and if you have multiple import from the same package it's not really understandable for newcomers why you don't have an @package(url:...) above each and every import.

Since it's a scripting feature I do prefer the # approach.

1 Like
18

Most of the times when I need to share code, I did share my code through library in other package but most of the final script have a big function in them that could be re-used elsewhere.

And this will make script testable with XCTest.

1 Like
19

This is great, thanks for this pitch.

I think we should consider making the version specification for @package optional. Scripts are often short-lived, so developers should have an easy way to just choose the latest version.

I think we should also treat the resolved file a little differently. One reason is also the short-lived case where the extra file would be an extra nuisance. The other reason would be thinking of scripts which are installed as part of the system, e.g. /usr/bin or in some other location where the current user doesn't have write access. I think we should not emit the resolved at all by default, but let people opt in via a CLI option and also allow customising the path via another one. Script authors would then be able to persist this via the #! line.

10 Likes
20

This whole file is not really a “swift file,” though, it is a script file that instructs the shell executing it to run it via the swift executable. The swift parser should not need to deal with the hash-bang line at the beginning IMO just to support pulling the whole file into Xcode or writing tests against it. The second I need testability, I’m going to move whatever needs unit testing into a module controlled by its own package file because the extra overhead of a package file is much less if you are maintaining test files anyway.

1 Like