SwiftPM support for Swift scripts

mackoj · 2020-03-19T02:54:38.950Z

Hi @Rahul_Malik since a lot of people have positive feedback on this idea should this move to the next phase ?

uliwitness · 2020-03-20T08:37:27.570Z

Why exclude the REPL? I could see Playgrounds benefiting greatly from being able to also pull in dependencies. Especially in teaching Swift, being able to pull in an IntArray so you don't have to explain generics too early or the like would be very useful.

tkremenek · 2020-03-20T14:58:15.545Z

mackoj:

Hi @Rahul_Malik since a lot of people have positive feedback on this idea should this move to the next phase ?

I am really sorry I have not had a chance to properly engage on this thread (most recently because of circumstances created by COVID-19) but I wanted to say that I'm inspired by both this proposal and work that various people are doing in the community to explore this problem space. Having a first class story for scripting that ties into the package manager is really important for Swift.

I hesitate to even make these remarks right now, given that I haven't had chance to set aside proper time to engage with this immediate discussion, but a few high-level thoughts come to mind:

If we had a package index, how would that impact this proposed idea? Obviously that depends on how a possible package index would work.
The proposal excludes the REPL. Is that a tactical concern to keep the implementation scope limited at first, or a fundamental design decision? In other words, there can be a broader arc of where such an enhancement fits in, but the feature itself is supported in an increasing number of contexts over time.

I agree that the REPL and other interactive environments could benefit from having something like this.

uliwitness · 2020-03-29T10:42:03.782Z

I don't think dotfiles or library are an either/or decision. If you look at what comes with macOS X, you can see that Apple left most standard Unix tools with their default settings. E.g. Apache, which isn't owned by Apple Most stuff is in /etc and the like, but most of the user-serviceable configurable paths like the web page directory were redirected to Apple-style directories like ~/Sites and /Library/WebServer/Documents.

So there is no reason why Swift scripts could not use different directories on other platforms to be a good citizen, like Apache does.

Regarding .CFUserTextEncoding, my guess is that's not a user default because UserDefaults probably needed that to do its work. And .DS_Store doesn't really fit the discussion, as it is used to store Finder view settings, like folder background images, on a per-folder basis, so is really just a per-folder override of Finder's default settings. It's the easiest to deal with if kept with its folder (also, Finder doesn't have to clean it up then).

And no, .DS_Store is not just for compatibility with other platforms and filesystems. Even HFS+ disk images use it to store their background picture.

mackoj · 2020-04-18T22:46:08.585Z

I’m really eager to see this feature in Swift I use scripting everyday. Do you think it might be in Swift 5.3 ? @Rahul_Malik
That would help making 5.3 a big release for SPM(since it already have a few other great features plan’ed for 5.3).

alejoaa · 2020-04-23T00:17:32.290Z

It would be amazing to have this feature in Swift! I wanted to push Swift usage at work, but we do a lot of scripting and unfortunately the lack of an easy way to import packages is hampering such objective. This needs to be built-in, thanks for your proposal!

adam-fowler · 2020-04-30T07:20:20.595Z

Up until this point I have been relying on swift-sh to provide SPM support in swift scripts. But since swift 5.2 it has been broken due to Package.swift changes. I have rolled my own version of swift-sh to get around some of the issues but that isn't a permanent solution.

To have swift SPM support in scripts being treated as a first class citizen would be great. Relying on a tool that doesn't get updated lockstep with the rest of swift is not ideal.

mackoj · 2020-05-16T19:38:59.874Z

This don’t seem to move. @Aciid what could we do as a community to make this go foward ?

stevapple · 2020-07-27T14:06:46.556Z

I think this would be a great feature for Swift 6...But it seems to stuck for some time. What can we do to push it further except for discussing it here? @Rahul_Malik

mackoj · 2020-09-19T00:18:42.585Z

I don’t think the original version authors have time or will to make this go forward so it will be up to us to make this move.

What could be the next step ?

eneko · 2021-02-08T04:29:34.364Z

I love this proposal, hoping it gets implemented.

If anything, I'd like to suggest the addition of a command to convert a Swift script into an executable package. Maybe something like this:

swift package init --from myscript.swift

The reasoning is that scripting is great for tinkering, prototyping, and creating quick tools. However, once these scripts grow both in complexity and over time, becoming long-lived scripts, it would be great to have an easy way to refactor them into a Package without having to do the manual work.

This addition might also help easing any discussions between short and long-lived scripts.

thecoolwinter · 2021-03-10T15:12:53.677Z

I love this proposal so far, one thing I'd add that's been mentioned before is updating packages with implied verisons, a build flag --update could be used that checks each package for an update against it's source.

Then the @package could still be used normally to imply the latest version, and the swift build --update could be used every once in a while to check for new versions.

stevapple · 2021-03-10T15:40:04.983Z

Looks great! Here’s my suggestion on syntax:

# Update on every run
# Unrecommended because updating and rebuilding takes time
swift script.swift --update 

# Update and rebuild immediately
# I propose to add a `swift-script` tool which is similar to `swift-package`
swift script update script.swift

thecoolwinter · 2021-03-13T19:16:52.434Z

# I propose to add a `swift-script` tool which is similar to `swift-package`

+1 for the swift script tool, but we could add it as a tool in SwiftPM instead of it's own thing. I'd say this kind of syntax feels the most fitting and these could be some sub-commands to manage the cache, run scripts etc.

swift script [filename] [options, eg: -update, -clean] // Runs a script and installs packages
swift script list // Lists all packages in ~/Library/Developer/SwiftPM/Cache
swift script clean // Cleans cache
swift script remove [repository] // Removes repository from cache

stevapple · 2021-03-19T10:44:22.689Z

Hello to all the Swift fellows!

As a GSoC applicant who decided to pick up this idea, I’ve drafted a revised proposal which adds lots of details to the original one. Here are a few highlights for the new parts:

The new swift-script tool for managing scripts and caches;
Design details on how a script is built and run;
Some alternative designs.

Welcome to check it out!

[Pitch] SwiftPM support for Swift scripts (revision) Pitches

Pitch: SwiftPM support for Swift scripts Proposal: SE-NNNN Authors: Rahul Malik, Ankit Aggarwal, David Hart, YR Chen Review Manager: TBD Status: Awaiting implementation Introduction Swift is a general-purpose language that aims to expand its availability and impact on various domains and platforms. We believe great scripting support and experience is an important part of extending Swift’s usage and improving the impact of Swift as a language. Swift already includes basic support for script…

wes1 · 2021-04-13T18:25:15.855Z

Hi Folks,

This is a promising effort and great discussion. As a distant observer my urge is to simplify and ship!

Why not just have a .swift file declare a PackageDependencies.Script object with a bootstrap static package member and a main entry point for SwiftPM tooling to build/run?

Easy scripting could really help expand the Swift ecosystem. This could have HUGE benefits to the community.

But proliferating swift syntax and alternate build systems can really complicate the ecosystem. Scripting should add near-zero complexity or confusion.

I prefer a single source of truth for configuration, and would not want to see @package url's in Doit.swift code conflicting with Package.swift statements. I also don't see package-imports as the only problem now with scripts (hash-bang's? shared build locations? reusing scripts?), so I don't see language tweaks converging in a summer on something viable (assuming we prefer shipping over blogging :)

So, perhaps reduce requirements to minimum-viable and strengthen constraints...

Goal is to write the smallest tool extension to build+run a smallish single-file script with only package dependencies.

So, instead of

  MyPackage/
    Sources/
      Source.swift
      main.swift
    Package.swift

We have

  SomeScripts/
     Dothis.swift
     Dothat.buildrun.swift
     ...

Some constraints:

No syntax changes in the language. sourcekit-lsp should handle scripts without modification, and scripts should compile like any other .swift file (if moved to an appropriate package).
Scripts use exactly the same build SwiftPM caching and build logic as swift packages. No new ./built-here locations (but see below).
(Scripts should not be run automatically during builds. I shouldn't be running untrusted code just by building a package I import.)

As for implementation, I would add type Script to PackageDependencies; Script would declare the package and main entry point. Then new buildrun tool mode (i.e., your SwiftPM script) could bootstrap:

If the swift file only declares a single Script object, swift buildrun .../Doit.swift will interpret the Script static member package: Package declaration to bootstrap the build for the script object, and optionally run the declared main entry point.

Then it's just a matter of API design and coding conventions for a Script implementation to reduce the package declaration and CLI using reasonable defaults/conventions, and some tooling shims (or script wrappers) to implement build-run.

Notice in v1 that scripts are leaves of a dependency tree:

Nothing depends on script code.
Script code only depends on packages, not on other swift files.

This I think reduces scripts to small CLI controllers assembling functionality from normal packages; anything larger should just be in a regular package (or deferred to v-1+). (Thus the dev cycle: hack a script, if anything reusable evolves, throw it into a package, iterate...)

Swift tooling would need to support this bootstrapping, and the .build directory layout updated to handle multiple nested scripts, e.g., if scripts are in SomeScripts/, then SomeScripts/.build/scripts/Doit/{normal package layout} (or an option for a user-wide build directory). swift buildrun might support a base directory for local packages (preferably relative to the jailed script directory tree in v1).

Simple IDE launch support might depend on using an extended suffix, e.g., Doit.buildrun.swift. For v1 CLI, users would deploy shell mechanisms (e.g. aliases/functions) to avoid a hash-bang at the top of the file, perhaps based on the same Doit.buildrun.swift convention.

As for script boundaries:

Script package names are not visible outside the script and might not be consistent across builds.
There is no sharing between script build environments (at least in v1), resulting in significant duplication of commonly-imported packages.
When migrating the script code into a package, the excess package declaration is ignored. I.e., it is an error to include the script file in a regular package, unless the regular package dependencies are sufficient. In this case, no attempt is made to detect deltas between the script-internal package declaration and the external package declaration.

If this could be implemented in a summer, it would be a great first step!

Wes

danramteke · 2021-05-27T03:27:13.089Z

Rahul_Malik:

Proposal: SE-NNNN

what is the Proposal number? That link is broken

wowbagger · 2021-05-27T03:58:16.116Z

"NNNN" is normally used as a placeholder. This proposal has not been officially proposed yet, so it doesn't have a number.

danramteke · 2021-05-27T04:00:39.165Z

Ah thanks for clarifying.

I'm very interested in this proposal. How can I help officially propose it?

wowbagger · 2021-05-27T04:08:38.794Z

@stevapple (along with this pitch's original authors) started a new pitch as a project for this year's Google Summer of Code. I think you can probably contact them to see how or what you can help with.