Executing processes in Package.swift

Hello,

I've been playing with Swift REPL, Swift scripting (Foundation's Process type) and Swift Package Manager recently.

Package.swift file allows me to write any code inside. I was able to import Foundation and run shell scripts or other unix programs. I've found, that the same is happening when I use such a package as a dependency. Stdout is decorated with yellow "Warning:" string.

Is this feature intentional, or is it something considered as shortcoming of current implementation (or just a bad style)?

In case this behavior is suppored and not subject to future revision, utilizing it would be hugely beneficial to some of my future use cases.

What is our opinion on the matter?

Cheers, Mikolas

The feature is intentional, but using it is bad style. In general, having your Package.swift dynamically reconfigure itself based on environmental parameters is really not an ideal outcome, and it’ll break some assumptions in the tooling: in particular, Package.resolved files become a surprising source of footguns and pain.

Thanks for your reply.

My use cases don't involve dynamic configuration. Configuration of package would remain the same. My use case would execute external application (either interpreted shell/python or build and run another swift package) in order to generate environment files for my middleware (between version of library installed on my machine) and Swift. Ideally, such files would remain concealed in one of the packages targets and would run only when file is missing.

This is also not good style: in general, relying on the idea that your Package.swift will be invoked in this way is not a safe thing to do. That's not to say this won't work well enough for your use-case: it may. However, the use-case here is better served by having SwiftPM become fully aware of external build tools that can be invoked to do this kind of reconfiguration.

Is there some official documentation about this? I'm not a fan of the design either, but adding a feature that nobody should use seems quite silly...

I am aware of Package Manager Extensible Build Tools however I am on certain schedule since the use case is open source project I want to/have to publish before June so even if such a feature would be in review now, it won't be in swiftpm soon enough for me.

I'm not saying I like such an approach, but I think, that swift build should be the only thing you write into terminal in order to build a package. I dislike packages with "install.sh" files more than I dislike some magic inside Package.swift file. At the same time, Xcode is not an issue for me, since I aim at Linux platform.

In case external build systems would be supported directly by SwiftPM I would be more than happy to get advantage of such a feature.

I was wondering, however, whether such an approach is dangerous. As @lukasa mentioned with Package.resolved or outright forbidden.

Thanks for your opinions so far :)

Is it? This paragraph suggests that the ability to run arbitrary code is a bug, not a feature

1 Like

Maybe we're disagreeing on what distinguishes bug and intentional feature. Consider a world in which our package manager accepted JSON, but also said (as this section essentially does), "we will allow you to have a section of your JSON file that is in Javascript, executable at parse time". The fact that this file is therefore executed at build time strikes me as a guaranteed feature of the implementation.

Therefore I assume that the fact that Package.swift is executable is intentional. As noted in the linked section, using this to perform I/O or other weird tasks is against the rules, and may not behave how you want, but the fact that you can use a Turing complete programming language to do this cannot in my mind be considered a bug.

2 Likes

That view makes sense, thanks!

1 Like

OK, then it is safe to assume, that triggering some boilerplate generators edit: during SPM build process from Package.swift (like GYB example: swift/UnsafeBufferPointer.swift.gyb at main · apple/swift · GitHub description: Swift GYB - NSHipster) is forbidden but possible?

That's a good way to characterise it. It's like shoplifting: forbidden, but possible.

2 Likes

There are many valid ways of putting the executability to work, and even of importing Foundation.

It is relatively common to query ProcessInfo for environment variables to switch development modes. Even official packages, such as SwiftPM and SwiftSyntax do so.

It can be used to detect the difference between Xcode’s native support for packages and generate-xcodeproj, such as here. Due to bugs in each, some things are possible in one but not the other at the moment.

Earlier versions of Xcode couldn’t handle URLs that weren’t percent encoded. At that time I used to iterate every URL at the end of the manifest to automatically encode everything for Xcode.

However, care should be taken if you are dynamically adjusting dependencies. As @lukasa said, that will make Package.resolved utterly pointless. It will have no effect on your clients, but it may or may not matter to you for your development strategy. If you look closely at SwiftPM’s manifest, it does change its dependencies, but you’ll note that doesn’t rely Package.resolved for anything and .gitignores it anyway.

But it is a very bad idea to try to adjust anything outside of the package manifest. When your package is a dependency, SwiftPM will be checking it in and out in multiple locations. It will cache your manifest information and then check out the code somewhere else. If you’ve tried to write into repository files, dirty Git states will cause SwiftPM to fail completely, and .gitignored generated files won’t travel with the checkouts. There will be nothing but chaos.