For scripts, I propose to use a unified directory for caching. A cache location for my-script.swift, for example, may be ~/.swiftpm/scripts/my-script-abc123, which will contain the .build and Package.resolved cache. abc123 is a hash value computed against the real path of the script.
Such design can prevent scripts from producing scattered wastes, and enable centralized control over build caches. This will allow scripts to be moved easily throughout the system. swift script cleanup can remove the cache for scripts that no longer exist.
swift-sh does this I believe. One of the problems with this is that you potentially run into problems when you have the script named the same across different folders. Maybe a hash of the folder name and the script name for managing things centrally, or simply apply the folder name then the script as part of the path so it's recognizable.
You’re right that projects like swift-sh already does the same thing, but I believe the community would prefer # or @ annotations with familiar SwiftPM-styled declaration, instead of other workarounds like comments.
In other words, we’re now going to make scripting officially supported by SwiftPM itself.
Hello to every one again — Now as a GSoC 2021 student! I'm really looking forward to the special experience with the Swift community.
Before the project officially starts, I drawed the following flow charts to show how swift and swiftc are going to fit in with this pitch. There will be some additional steps (marked in red), but the behavior will remain the same for existing codes.
Neat! I think the proposed approach will work well. A couple comments:
If -emit-package-declarations is going to be separate from -scan-dependencies, swiftc should avoid creating a standalone job whose only purpose is to diagnose disallowed uses of the @package attribute. This would add quite a bit of overhead from all of the extra parsing that would be required in multi-file compiles. I think what I'd recommend is something like the following:
by default, swift-frontend always emits an error when it sees the attribute. This can probably be added to one of the AST walkers in MiscDiagnostics.cpp so that it can be done as part of a normal compilation
The -ignore-package-declarations frontend flag is used only when compiling a script with interactive mode swift, and disables the diagnostic
-emit-package-declarations implies -ignore-package-declarations and also ignores the diagnostic
It would be interesting to see how the performance compares between building the script and its dependencies as an executable versus building the dependencies into a shared library (similar to swift run --repl) and running the script in the JIT. The approach here seems totally reasonable though
BTW, the flow charts do a great job of communicating the proposed design - once they're finalized they'd be great additions to the documentation folder in the swift repository IMO
If I understand correctly this builds the script as an executable, and execute it. Am I right?
Have you considered executing the script with frontend immediate mode (i.e. swift-frontend -frontend -interpret <script>) with appropriate -I/-L/-l options after building the packages as dylibs? If this proposal intend to replace swift <script> mode, I feel like JIT execution fits better.
I think it’s quite worth keeping the JIT mode with this feature, thanks for your suggestion!
Meanwhile, swift-script is actually not intended to replace the current behavior of swift <script> — the graph already shows that a script without @package definitions will still be executed using the interpreter. The new behavior of swift <script> serves as an add-on, a shortcut to the whole new feature.
The main goal for swift-script is to easily build tools with a single script file, where we want the script to behave just like an executable. Therefore, I believe pre-building an executable provides better performance here. It would be interesting to compare these two models after the implementation is available, and see which one better meets the need of the whole community.
OK, but I expect swift <script> is executed by JIT regardless of existence of @packages in the script.
To get better performance or an executable file, I would use swift package init —from-script. For me, swift-script (or swift <script>) will be mainly for quick development before packaging, or running onetime scripts. But this is just a personal preference
Being able to choose between compile/run or JIT would be great, though.
I don't know enough about the Swift compiler's internals to judge what is more work, so just a short question: Would it make sense to just use an intermediates folder right next to the script and leave the cache for later? I think SwiftPM already does this with a .build directory?
Feels like a quick way to get something like this bootstrapped and usable would be to just scan over the script for @package declarations, use them and the script file's name to generate a Package.swift, and then just build and run it using regular SwiftPM.
That way you'd leverage a lot of existing infrastructure, and then you could always go in deeper and add more complex features (like making the Swift compiler parse @package declarations, and then either ignore them in script mode or output an error in regular mode), but at every point you'd already have something working.
Anyway, this might already be obvious to you and I might be imagining the wrong parts of the project being the actual difficulty.
One thing I don’t understand about this (otherwise very useful) proposal is: why forbid the use of swiftc in the presence of @package annotations?
Since it is proposed that swift myscript.swift be a shortcut for swift-script run myscript.swift, why not do the same for swiftc and have swiftc myscript.swift be a shortcut for swift-script build myscript.swift -o myscript?
I would find it very useful to be able to compile little, one-source-file tools using one simple command whether they have dependencies or not.
Yeah I think the same. In fact, the swiftly CLI being developed by the SSWG would be a perfect reason to push this work forward too, as getting started writing Swift scripts on any OS could become as easy as:
For anyone who’re interested in continuing the work for GSoC 2023, I would like to talk about it a bit more and provide some useful takeaways for you from the GSoC 2021 work.
SwiftPM support for Swift scripts is not a Swift- or SwiftPM-only feature. It requires interoperability between the Swift compiler, compiler driver and SwiftPM — and the trickiest part is calling SwiftPM from the Swift compiler. Many community members, including me, don’t like the idea of integrating SwiftPM definitions directly into the compiler. Therefore, the feature had better be done by some kind of Swift compiler plug-in provided by SwiftPM.
2 years ago, we didn’t have a stable Swift Syntax package to use from SwiftPM, nor did we have Swift macros, so we ended up with an ugly hack in the compiler. The new reference parser implementation for this feature did the parsing in a far more elegant way, but there’re suggestions that it should be redesigned to use macros or compiler extensions (see the discussion in Pre-Pitch: `@package` argument syntax).
There’re some pieces that you may take away from previous works:
You can refer to the syntax definition described in @package argument syntax. After SwiftPM 5.6 the PackageDescription APIs are mostly stable, so the final syntax is very likely to resemble that pitch.
You can learn how to dynamically check for scripting mode in Swift Driver from this PR. This is used to unblock swift <script.swift> styled calls for SwiftPM-enabled scripts without breaking existing behavior.
You may learn from the design of swift-script tool in this PR. The implementation may change dramatically, but some features (like quiet mode and global cache management) are useful.
Finally, you may read the GSoC 2021 writeups for this project, and try to get some inspiration:)
Welcome (again) to the Swift community and enjoy your GSoC journey😃