[Pitch] SwiftPM support for Swift scripts (revision)

[stevapple (YR Chen)]

For scripts, I propose to use a unified directory for caching. A cache location for my-script.swift, for example, may be ~/.swiftpm/scripts/my-script-abc123, which will contain the .build and Package.resolved cache. abc123 is a hash value computed against the real path of the script.

Such design can prevent scripts from producing scattered wastes, and enable centralized control over build caches. This will allow scripts to be moved easily throughout the system. swift script cleanup can remove the cache for scripts that no longer exist.

[stevapple (YR Chen)]

@0xTim

The first is obviously pulling in code from anything other than Foundation. I end up duplicating the same lines of code across every script I write. So whilst I would love to see improvements in the way Swift handles interacting with external processes I'm happy to just wrap that in a library for now.

Not going to be done in this pitch in order to keep it simple and focused on scripting support. With this pitch you may use a library package with absolute path to eliminate the effort.

A more elegant way to interact with external programs and scripts will be mentioned in future directions. I believe that’s another important improvement worthy of further discussions.

[Cavelle_Benjamin]

swift-sh does this I believe. One of the problems with this is that you potentially run into problems when you have the script named the same across different folders. Maybe a hash of the folder name and the script name for managing things centrally, or simply apply the folder name then the script as part of the path so it's recognizable.

[stevapple (YR Chen)]

I think the answer has already been there👀

@stevapple

A cache location for my-script.swift , for example, may be ~/.swiftpm/scripts/my-script-abc123 , which will contain the .build and Package.resolved cache. abc123 is a hash value computed against the real path of the script.

You’re right that projects like swift-sh already does the same thing, but I believe the community would prefer # or @ annotations with familiar SwiftPM-styled declaration, instead of other workarounds like comments.

In other words, we’re now going to make scripting officially supported by SwiftPM itself.

[Cavelle_Benjamin]

Yep. I missed it.

[danramteke (Daniel R)]

I'm excited for this functionality.

For my use case, I would like to use relative paths for import statements as well. Relative paths are already supported by Swift Package Manager, so hopefully there's no issue.

@package(path: "../my/local/swift/package")
import MyLocalSwiftPackage

[stevapple (YR Chen)]

Hello to every one again — Now as a GSoC 2021 student! I'm really looking forward to the special experience with the Swift community.

Before the project officially starts, I drawed the following flow charts to show how swift and swiftc are going to fit in with this pitch. There will be some additional steps (marked in red), but the behavior will remain the same for existing codes.

swift

swiftc

---

Here's also a more complex one demonstrating some key steps of building and running a script. The shared steps are marked in red:

If you have any question or suggestion, feel free to discuss it here:)

[owenv (Owen Voorhees)]

Neat! I think the proposed approach will work well. A couple comments:

• If -emit-package-declarations is going to be separate from -scan-dependencies, swiftc should avoid creating a standalone job whose only purpose is to diagnose disallowed uses of the @package attribute. This would add quite a bit of overhead from all of the extra parsing that would be required in multi-file compiles. I think what I'd recommend is something like the following:
  • by default, swift-frontend always emits an error when it sees the attribute. This can probably be added to one of the AST walkers in MiscDiagnostics.cpp so that it can be done as part of a normal compilation
  • The -ignore-package-declarations frontend flag is used only when compiling a script with interactive mode swift, and disables the diagnostic
  • -emit-package-declarations implies -ignore-package-declarations and also ignores the diagnostic
• It would be interesting to see how the performance compares between building the script and its dependencies as an executable versus building the dependencies into a shared library (similar to swift run --repl) and running the script in the JIT. The approach here seems totally reasonable though

BTW, the flow charts do a great job of communicating the proposed design - once they're finalized they'd be great additions to the documentation folder in the swift repository IMO

[rintaro (Rintaro Ishizaki)]

@stevapple

Then SwiftPM will fetch and build the dependencies, and build the script with -ignore-package-declarations . By default, a script will be built using release configuration, and the -c <configuration> option is available. The build cache will also be placed in the cache directory, to enable fast startup of prebuilt scripts.

If I understand correctly this builds the script as an executable, and execute it. Am I right?

Have you considered executing the script with frontend immediate mode (i.e. swift-frontend -frontend -interpret <script>) with appropriate -I/-L/-l options after building the packages as dylibs? If this proposal intend to replace swift <script> mode, I feel like JIT execution fits better.

Something like:

1. swift-frontend -emit-package-declarations <script> → package declarations
2. If there are external package dependencies
   1. SwiftPM to build swiftmodues and dylibs → Package.swiftmodule and libPackage.dylib
   2. build frontend arguments → -I /path/to/modules -L /path/to/lib -lPackage
3. swift-frontend -frontend -interpret -I /path/to/modules -L /path/to/lib -lPackage <script>

What do you think?

[stevapple (YR Chen)]

@rintaro

If I understand correctly this builds the script as an executable, and execute it. Am I right?

Exactly.

@rintaro

Have you considered executing the script with frontend immediate mode (i.e. swift-frontend -frontend -interpret <script> ) with appropriate -I / -L / -l options after building the packages as dylibs? If this proposal intend to replace swift <script> mode, I feel like JIT execution fits better.

I think it’s quite worth keeping the JIT mode with this feature, thanks for your suggestion!

Meanwhile, swift-script is actually not intended to replace the current behavior of swift <script> — the graph already shows that a script without @package definitions will still be executed using the interpreter. The new behavior of swift <script> serves as an add-on, a shortcut to the whole new feature.

The main goal for swift-script is to easily build tools with a single script file, where we want the script to behave just like an executable. Therefore, I believe pre-building an executable provides better performance here. It would be interesting to compare these two models after the implementation is available, and see which one better meets the need of the whole community.

[rintaro (Rintaro Ishizaki)]

@stevapple

Meanwhile, swift-script is actually not intended to replace the current behavior of swift <script> — the graph already shows that a script without @package definitions will still be executed using the interpreter. The new behavior of swift <script> serves as an add-on, a shortcut to the whole new feature.

OK, but I expect swift <script> is executed by JIT regardless of existence of @packages in the script.

@stevapple

The main goal for swift-script is to easily build tools with a single script file, where we want the script to behave just like an executable. Therefore, I believe pre-building an executable provides better performance here. It would be interesting to compare these two models after the implementation is available, and see which one better meets the need of the whole community.

To get better performance or an executable file, I would use swift package init —from-script. For me, swift-script (or swift <script>) will be mainly for quick development before packaging, or running onetime scripts. But this is just a personal preference

Being able to choose between compile/run or JIT would be great, though.

[uliwitness (Not a Kitteh)]

I don't know enough about the Swift compiler's internals to judge what is more work, so just a short question: Would it make sense to just use an intermediates folder right next to the script and leave the cache for later? I think SwiftPM already does this with a .build directory?

Feels like a quick way to get something like this bootstrapped and usable would be to just scan over the script for @package declarations, use them and the script file's name to generate a Package.swift, and then just build and run it using regular SwiftPM.

That way you'd leverage a lot of existing infrastructure, and then you could always go in deeper and add more complex features (like making the Swift compiler parse @package declarations, and then either ignore them in script mode or output an error in regular mode), but at every point you'd already have something working.

Anyway, this might already be obvious to you and I might be imagining the wrong parts of the project being the actual difficulty.

[tevamerlin]

One thing I don’t understand about this (otherwise very useful) proposal is: why forbid the use of swiftc in the presence of @package annotations?

Since it is proposed that swift myscript.swift be a shortcut for swift-script run myscript.swift, why not do the same for swiftc and have swiftc myscript.swift be a shortcut for swift-script build myscript.swift -o myscript?
I would find it very useful to be able to compile little, one-source-file tools using one simple command whether they have dependencies or not.

[Jeehut (Cihat Gündüz)]

What’s the state of this Pitch? Is there any plan to making it an official proposal? Or is there any follow-up pitch I missed that’s being discussed?

I feel it’s about time that Swift gets proper scripting support. It also goes in line with the #1 goal for Swift 6: “Accelerate growth of the Swift software ecosystem“ from On the road to Swift 6.

[JetForMe (Rick M)]

I, too, would love to see this implemented. I think it's a great addition.

[davdroman (David Roman)]

@Jeehut

I feel it’s about time that Swift gets proper scripting support. It also goes in line with the #1 goal for Swift 6: “Accelerate growth of the Swift software ecosystem“ from On the road to Swift 6 .

Yeah I think the same. In fact, the swiftly CLI being developed by the SSWG would be a perfect reason to push this work forward too, as getting started writing Swift scripts on any OS could become as easy as:

$ swiftly install 5.7
$ echo -e '#!/usr/bin/env swift\nprint("Hello world")' > helloworld.swift
$ chmod +x helloworld.swift
$ ./helloworld.swift

[stevapple (YR Chen)]

For who may be interested, the syntax of @package is pre-pitched at Pre-Pitch: `@package` argument syntax for review:)

[stevapple (YR Chen)]

For anyone who’re interested in continuing the work for GSoC 2023, I would like to talk about it a bit more and provide some useful takeaways for you from the GSoC 2021 work.

SwiftPM support for Swift scripts is not a Swift- or SwiftPM-only feature. It requires interoperability between the Swift compiler, compiler driver and SwiftPM — and the trickiest part is calling SwiftPM from the Swift compiler. Many community members, including me, don’t like the idea of integrating SwiftPM definitions directly into the compiler. Therefore, the feature had better be done by some kind of Swift compiler plug-in provided by SwiftPM.

2 years ago, we didn’t have a stable Swift Syntax package to use from SwiftPM, nor did we have Swift macros, so we ended up with an ugly hack in the compiler. The new reference parser implementation for this feature did the parsing in a far more elegant way, but there’re suggestions that it should be redesigned to use macros or compiler extensions (see the discussion in Pre-Pitch: `@package` argument syntax).

There’re some pieces that you may take away from previous works:

1. You can refer to the syntax definition described in @package argument syntax. After SwiftPM 5.6 the PackageDescription APIs are mostly stable, so the final syntax is very likely to resemble that pitch.
2. You can learn how to dynamically check for scripting mode in Swift Driver from this PR. This is used to unblock swift <script.swift> styled calls for SwiftPM-enabled scripts without breaking existing behavior.
3. You may learn from the design of swift-script tool in this PR. The implementation may change dramatically, but some features (like quiet mode and global cache management) are useful.
4. Finally, you may read the GSoC 2021 writeups for this project, and try to get some inspiration:)

Welcome (again) to the Swift community and enjoy your GSoC journey😃