SwiftScript: Using Swift for scripting with third party packages support

[Serika_PHB (Haobo Peng)]

Swift has already supported running a single source file directly in the CLI. However, it can be hard to use it together with third party packages. I think the simplicity of Swift syntax make it one of the nice choices for scripting and filling this gap of working with packages might be useful. So, I tried to implement SwiftScript, which is a CLI tools that help using Swift for scripting while allowing third party packages.

GitHub - Star-Lord-PHB/swift-script: A command line tools for executing single swift...

There are already some similar work available, one of them is swift-sh. It is a great project, but personally, I prefer managing packages with styles similar to tools like pip3 instead of declaring the packages in the script using comments.

About SwiftScript

The installation guidance and usage instructions can be found in the README in the repository, here I'll just give a brief introduction.

SwiftScript is implemented as an SPM wrapper that will build your script together with all the packages with a SPM project.

The subcommands and their abstracts are listed in the screenshot above.

Run a script

swiftscript /path/to/script <additional_args>

Search for a package

SwiftScript will try to find the package identity in Swift Package Index and print out some useful information

swiftscript search <package_identity>

Install a package

Packages to install can be specified with identity or url

Version of the package to install can be specified using Version Requirement, which is the same as the one used in Package.swift in SPM project.

• --exact \<version\>
• --branch \<branch\>
• --from \<version\> (up to next major)
• --up-to-next-minor-from \<version\>
• --to \<version\> (upper bound version)

swiftscript install <package_identity> <version_requirement>
swiftscript install <package_url> <version_requirement>

Edit a script

Editing a Swift source file can be done in any editor, but auto-completion will not work for packages installed through SwiftScript. For such case, use the edit subcommand.

swiftscript edit /path/to/script

It will create a temporary SPM project with all the dependencies already configured. By default it will try to find VSCode in the environment (i.e.: the code command), but the editor used can be configured using swiftscript config set editor command.

Any editor should be OK as long as:

• It is able to open a folder
• It can be configured to NOT return before the editor window is closed

Known Limitations

• No directly installable binary available yet
• Testing fail to run on ARM Windows
• Cannot build in Release Mode on ARM Windows for some reason
• Does not support Module Alias yet
• Script building might be slow, so it can take sometime before starting to execute the script.
• Maybe more ...

Finally

I'm not sure whether people are interested in using Swift as script since there aren't many relative discussions. Please feel free to discuss whether such tools are really useful.

[wes1 (Wes)]

Scripting with Swift is much more pleasant than e.g., bash, but dependencies and packaging are a pain point.

Swift-sh and SwiftScript seem to go down the road of creating a mini-language for configuring dependencies that replicates some of what SPM is doing in Package.swift. Both also create and delegate to temporary SPM projects.

An alternative approach is to treat a script file as a "peer" copy of a single-file executable in some SPM package which has a library (with dependencies). A launcher is responsible for finding the associated package, creating, updating and building the executable if needed, and then invoking the binary.

That's the approach of https://github.com/swift-nest/clutch.

The (Swift bird) "nest" is the package with a "clutch" of related scripts (eggs) sharing a common library and its dependencies (e.g., one nest package for build scripts, another for video processing, etc.).

The user can edit and check in the nest package (and library and executables) as usual. The scripts can be anywhere on the filesystem, and the tool also supports finding nests, listing peers in a nest, emitting peer (templates) to edit, etc.

Otherwise, there have also been efforts to better support scripting in SPM, e.g., https://forums.swift.org/t/swiftpm-support-for-swift-scripts/33126 and https://forums.swift.org/t/pitch-swiftpm-support-for-swift-scripts-revision/46717

Also https://github.com/GeorgeLyon/Shwift helps with actually writing scripts, esp. async support, and Foundation seems to be working on more ergonomic and correct subprocess: https://forums.swift.org/t/review-2nd-sf-0007-subprocess/76547

[Serika_PHB (Haobo Peng)]

I have heard about Shwift and the new Subprocess in foundation, but the clutch and the @package syntax is new to me. I'll spend some more time reading through them. Thanks for sharing!

One thing to clarify is that SwiftScript does not create temporary SPM projects every time when running a script. In fact, there is only one SPM project for building all the scripts. This SPM project will be created in ~/.swift-script/runner when SwiftScript is installed and every time a new package is installed using the install subcommand, corresponding dependencies will be appended into Package.swift, which will stay there until being removed using the uninstall subcommand. Temporary SPM projects will only be created when searching for a package from identity (used for running swift package describe to gather information of that package).

For clutch, I struggle understanding the differences between its idea versus delegating tasks to SPM project. It seems that clutch is also using SPM?

[wes1 (Wes)]

@Serika_PHB

It seems that clutch is also using SPM?

Yes, clutch uses SPM for any builds, dependency specifications, etc.

@Serika_PHB

there is only one SPM project for building all the scripts
...
every time a new package is installed using the install subcommand, corresponding dependencies will be appended into Package.swift,

Ah, I like the simplicity of that, having packages "installed" globally. And the integration with swift package index is great, so users quickly get the package coordinates.

clutch doesn't support the convenient install/uninstall command or automatic runner package or finding packages or their coordinates. It requires the user to create any nest (including the default at ~/git/Nest) and add library dependencies in the Package.swift. clutch doesn't try to do any magic behind the scenes; it's just automating what a user would have to do if they had scattered script files with dependencies. But because scripts end up as regular executable targets, they can be debugged, tested, managed in git, etc. as usual.

But anything like this - a launcher invoking builds then the script executable - can end up conflating multiple layers of errors: is it a clutch/launcher bug? a build problem? a problem launching the script executable? A problem with the script? Can the user ignore errors from layer or another? Is the exit code correct? If the build is failing, can the user replicate it or get verbose output?

So another "cost" for the user is the work of detangling these layers; if users find it initially easy to make scripts but later difficult to fix them, then the tool becomes a kind of trap where they invest time into a bunch of scripts, and then get stuck, unable to fix them or blocked by launcher bugs.

In terms of workflow and clarity, I like the swift-sh approach of specifying the package coordinates after the import statement because you don't get the spooky action-at-a-distance effect of changing some dependency elsewhere and then breaking your scripts.

Similarly, there's the issue with shared libraries/packages of failures in one part propagating to others (so e.g., one script is blocked by another's compiler error). Fortunately SPM allows you to specify the target when building to side-step other executable target failures, but any time a common dependency is updated, it could break all the client scripts. So it's nice to have some way to check that dependency upgrades don't break any clients.

And then there are questions about paths, async behaviors, filesystems/pipes, etc. all of which have platform-specific issues even granting that Foundation mostly has API's for all of them. Since launchers launch other processes, wait for them to end, and convey results, they're a single point of failure highly exposed to these issues.

For that reason, users might prefer not to install a launcher using a system package manager, particularly if the launcher is newish. If they check out the package source and build/install it themselves, they have some confidence in the binary, and they can directly debug or work around any problems in the launcher itself.

[dschaefer2 (Doug Schaefer)]

Cool. I've been thinking of how we could extend Swift's current REPL with package dependencies with the help of SwiftPM. It would be nice to have that built in, kinda like python with pip.

[Serika_PHB (Haobo Peng)]

@wes1

So another "cost" for the user is the work of detangling these layers; if users find it initially easy to make scripts but later difficult to fix them, then the tool becomes a kind of trap where they invest time into a bunch of scripts, and then get stuck, unable to fix them or blocked by launcher bugs.

Indeed, debugging and fixing a single swift file is quite challenging, especially when packages are included. SwiftScript does provide a few things to help in such situation.

1. When a package is installed, SwiftScript will try to build the runner SPM project. Although the main motivation of doing this is to speed up subsequence build, it's can also be used to check whether the installed package itself can actually be built. If this building process fail, the Package.swift file will be reverted back to the original state (i.e.: the package will not be installed).
2. When building a script, users can pass in -v or --verbose flag to see the build logs from SPM, which should help identifying some bugs. The screenshot below is an example, where the script tries to import SystemPackage while swift-system is not installed:
   
   
3. The edit subcommand will create a separated temporary SPM project with all the dependencies already configured and open it with an configured editor (VSCode by default), where users can build and debug with the script manually.