[Pitch] SwiftPM support for Swift scripts (revision)

My bad. I do mean swift script reset in the proposal. Also swift script clean, supposing it’s not much more work.

I think clean and reset are not bad. To me it doesn’t seem so different from packages.

Overall I'm a massive +1 for pitch. I use Swift for almost all of my scripting these days mainly due to code reuse and it's a language I actually know and am comfortable in.

There are some significant pain points which this pitch should solve. The first is obviously pulling in code from anything other than Foundation. I end up duplicating the same lines of code across every script I write. So whilst I would love to see improvements in the way Swift handles interacting with external processes I'm happy to just wrap that in a library for now.

One outstanding question I have is about build artefacts and package resolution. SwiftPM relies on Package.resolved to work out what versions need to be used so are scripts now going to have an accompanying resolved file?

Additionally, in terms of build artefacts for scripts - will we have a .build directory for scripts that contain built modules? And how will that work when we have multiple scripts in the same directory, or in the same directory as a real package? (That question also applies to Package.resolved files as well). I'm hoping that the coming 5.4 release for a shared SwiftPM cache on the system should improve resolution times but that doesn't apply to build artefacts (yet) so that needs to be considered.

3 Likes

For scripts, I propose to use a unified directory for caching. A cache location for my-script.swift, for example, may be ~/.swiftpm/scripts/my-script-abc123, which will contain the .build and Package.resolved cache. abc123 is a hash value computed against the real path of the script.

Such design can prevent scripts from producing scattered wastes, and enable centralized control over build caches. This will allow scripts to be moved easily throughout the system. swift script cleanup can remove the cache for scripts that no longer exist.

2 Likes

Not going to be done in this pitch in order to keep it simple and focused on scripting support. With this pitch you may use a library package with absolute path to eliminate the effort.

A more elegant way to interact with external programs and scripts will be mentioned in future directions. I believe that’s another important improvement worthy of further discussions.

1 Like

swift-sh does this I believe. One of the problems with this is that you potentially run into problems when you have the script named the same across different folders. Maybe a hash of the folder name and the script name for managing things centrally, or simply apply the folder name then the script as part of the path so it's recognizable.

1 Like

I think the answer has already been there👀

You’re right that projects like swift-sh already does the same thing, but I believe the community would prefer # or @ annotations with familiar SwiftPM-styled declaration, instead of other workarounds like comments.

In other words, we’re now going to make scripting officially supported by SwiftPM itself.

1 Like

Yep. I missed it. :raised_hands:t5:

I'm excited for this functionality.

For my use case, I would like to use relative paths for import statements as well. Relative paths are already supported by Swift Package Manager, so hopefully there's no issue.

@package(path: "../my/local/swift/package")
import MyLocalSwiftPackage
2 Likes

Hello to every one again — Now as a GSoC 2021 student! I'm really looking forward to the special experience with the Swift community.

Before the project officially starts, I drawed the following flow charts to show how swift and swiftc are going to fit in with this pitch. There will be some additional steps (marked in red), but the behavior will remain the same for existing codes.

swift

swiftc


Here's also a more complex one demonstrating some key steps of building and running a script. The shared steps are marked in red:

If you have any question or suggestion, feel free to discuss it here:)

6 Likes

Neat! I think the proposed approach will work well. A couple comments:

  • If -emit-package-declarations is going to be separate from -scan-dependencies, swiftc should avoid creating a standalone job whose only purpose is to diagnose disallowed uses of the @package attribute. This would add quite a bit of overhead from all of the extra parsing that would be required in multi-file compiles. I think what I'd recommend is something like the following:
    • by default, swift-frontend always emits an error when it sees the attribute. This can probably be added to one of the AST walkers in MiscDiagnostics.cpp so that it can be done as part of a normal compilation
    • The -ignore-package-declarations frontend flag is used only when compiling a script with interactive mode swift, and disables the diagnostic
    • -emit-package-declarations implies -ignore-package-declarations and also ignores the diagnostic
  • It would be interesting to see how the performance compares between building the script and its dependencies as an executable versus building the dependencies into a shared library (similar to swift run --repl) and running the script in the JIT. The approach here seems totally reasonable though :+1:

BTW, the flow charts do a great job of communicating the proposed design - once they're finalized they'd be great additions to the documentation folder in the swift repository IMO

1 Like

If I understand correctly this builds the script as an executable, and execute it. Am I right?

Have you considered executing the script with frontend immediate mode (i.e. swift-frontend -frontend -interpret <script>) with appropriate -I/-L/-l options after building the packages as dylibs? If this proposal intend to replace swift <script> mode, I feel like JIT execution fits better.

Something like:

  1. swift-frontend -emit-package-declarations <script> → package declarations
  2. If there are external package dependencies
    1. SwiftPM to build swiftmodues and dylibs → Package.swiftmodule and libPackage.dylib
    2. build frontend arguments → -I /path/to/modules -L /path/to/lib -lPackage
  3. swift-frontend -frontend -interpret -I /path/to/modules -L /path/to/lib -lPackage <script>

What do you think?

Exactly.

I think it’s quite worth keeping the JIT mode with this feature, thanks for your suggestion!

Meanwhile, swift-script is actually not intended to replace the current behavior of swift <script> — the graph already shows that a script without @package definitions will still be executed using the interpreter. The new behavior of swift <script> serves as an add-on, a shortcut to the whole new feature.

The main goal for swift-script is to easily build tools with a single script file, where we want the script to behave just like an executable. Therefore, I believe pre-building an executable provides better performance here. It would be interesting to compare these two models after the implementation is available, and see which one better meets the need of the whole community.

1 Like

OK, but I expect swift <script> is executed by JIT regardless of existence of @packages in the script.

To get better performance or an executable file, I would use swift package init —from-script. For me, swift-script (or swift <script>) will be mainly for quick development before packaging, or running onetime scripts. But this is just a personal preference :slight_smile:

Being able to choose between compile/run or JIT would be great, though.

1 Like

I don't know enough about the Swift compiler's internals to judge what is more work, so just a short question: Would it make sense to just use an intermediates folder right next to the script and leave the cache for later? I think SwiftPM already does this with a .build directory?

Feels like a quick way to get something like this bootstrapped and usable would be to just scan over the script for @package declarations, use them and the script file's name to generate a Package.swift, and then just build and run it using regular SwiftPM.

That way you'd leverage a lot of existing infrastructure, and then you could always go in deeper and add more complex features (like making the Swift compiler parse @package declarations, and then either ignore them in script mode or output an error in regular mode), but at every point you'd already have something working.

Anyway, this might already be obvious to you and I might be imagining the wrong parts of the project being the actual difficulty.

One thing I don’t understand about this (otherwise very useful) proposal is: why forbid the use of swiftc in the presence of @package annotations?

Since it is proposed that swift myscript.swift be a shortcut for swift-script run myscript.swift, why not do the same for swiftc and have swiftc myscript.swift be a shortcut for swift-script build myscript.swift -o myscript?
I would find it very useful to be able to compile little, one-source-file tools using one simple command whether they have dependencies or not.

What’s the state of this Pitch? Is there any plan to making it an official proposal? Or is there any follow-up pitch I missed that’s being discussed?

I feel it’s about time that Swift gets proper scripting support. It also goes in line with the #1 goal for Swift 6: “Accelerate growth of the Swift software ecosystem“ from On the road to Swift 6.

6 Likes

I, too, would love to see this implemented. I think it's a great addition.

4 Likes

Yeah I think the same. In fact, the swiftly CLI being developed by the SSWG would be a perfect reason to push this work forward too, as getting started writing Swift scripts on any OS could become as easy as:

$ swiftly install 5.7
$ echo -e '#!/usr/bin/env swift\nprint("Hello world")' > helloworld.swift
$ chmod +x helloworld.swift
$ ./helloworld.swift
6 Likes

For who may be interested, the syntax of @package is pre-pitched at Pre-Pitch: `@package` argument syntax for review:)

5 Likes

For anyone who’re interested in continuing the work for GSoC 2023, I would like to talk about it a bit more and provide some useful takeaways for you from the GSoC 2021 work.

SwiftPM support for Swift scripts is not a Swift- or SwiftPM-only feature. It requires interoperability between the Swift compiler, compiler driver and SwiftPM — and the trickiest part is calling SwiftPM from the Swift compiler. Many community members, including me, don’t like the idea of integrating SwiftPM definitions directly into the compiler. Therefore, the feature had better be done by some kind of Swift compiler plug-in provided by SwiftPM.

2 years ago, we didn’t have a stable Swift Syntax package to use from SwiftPM, nor did we have Swift macros, so we ended up with an ugly hack in the compiler. The new reference parser implementation for this feature did the parsing in a far more elegant way, but there’re suggestions that it should be redesigned to use macros or compiler extensions (see the discussion in Pre-Pitch: `@package` argument syntax).

There’re some pieces that you may take away from previous works:

  1. You can refer to the syntax definition described in @package argument syntax. After SwiftPM 5.6 the PackageDescription APIs are mostly stable, so the final syntax is very likely to resemble that pitch.
  2. You can learn how to dynamically check for scripting mode in Swift Driver from this PR. This is used to unblock swift <script.swift> styled calls for SwiftPM-enabled scripts without breaking existing behavior.
  3. You may learn from the design of swift-script tool in this PR. The implementation may change dramatically, but some features (like quiet mode and global cache management) are useful.
  4. Finally, you may read the GSoC 2021 writeups for this project, and try to get some inspiration:)

Welcome (again) to the Swift community and enjoy your GSoC journey😃

5 Likes