[Pitch] Swift Subprocess

Related Projects Foundation
1

Hi all,

I would like to pitch a new API, Subprocess, that that will eventually replace Process as the canonical way to launch a process in Swift. Please let me know if you have any questions/comments/thoughts!

Introducing Swift Subprocess

Introduction / Motivation

As Swift establishes itself as a general-purpose language for both compiled and scripting use cases, one persistent pain point for developers is process creation. The existing Foundation API for spawning a process, NSTask, originated in Objective-C. It was subsequently renamed to Process in Swift. As the language has continued to evolve, Process has not kept up. It lacks support for async/await, makes extensive use of completion handlers, and uses Objective-C exceptions to indicate developer error. This proposal introduces a new type called Subprocess, which addresses the ergonomic shortcomings of Process and enhances the experience of using Swift for scripting.

Proposed Solution

We propose a new type, struct Subprocess, that will eventually replace Process as the canonical way to launch a process in Swift.

Here's a simple usage Subprocess:

#!/usr/bin/swift

import FoundationEssentials

let gitResult = try await Subprocess.run(   // <- 0
    executing: .named("git"),               // <- 1
    arguments: ["diff", "--name-only"]
)
if let output = gitResult.standardOutput {  // <- 2
    print("Output: \(Strint(data: output, encoding: .utf8))")
}

Let's break down the example above:

  1. Subprocess is constructed entirely on the async/await paradigm. The run() method utilizes await to allow the child process to finish, asynchronously returning an ExecutionResult. Additionally, there is an closure based overload of run() that offers more granulated control, which will be discussed later.
  2. There are two primary ways to configure the executable being run:
    • The default approach, recommended for most developers, is .at(FilePath). This allows developers to specify a full path to an executable.
    • Alternatively, developers can use .named(String) to instruct Subprocess to look up the executable path based on heuristics such as the $PATH environment variable.
  3. The standard output and error are delivered either via AsyncSequence or Data.

I've posted the full pitch including the detailed design here on the swift-foundation repo. Check out the full pitch to read more info about the proposed design.

Trying it out

You can find a experimental (WIP) implementation here: swift-experimental-subprocess

Feel free to play around with it and let me know what you think!

43 Likes
2

Can you clarify what the code would look like to asynchronously enumerate the lines of the output of a subprocess? i.e. consuming each line as it's available, without waiting for the subprocess to exit.

12 Likes
3

I'm quite excited with this, as dealing with Process is definitely annoying. A couple of comments...

I appreciate the convenience of Subprocess.Executable.named(...) to perform which-like lookup for the binary, but I don't see a good way to get the actual discovered FilePath. How would I do that? Also, how does that lookup work? Does it only consider the PATH of the environment, or does it involve any user shell exports?

StandardInputWriter looks exactly like the sort of async NSOutputStream replacement we've been waiting for… Can you talk more about the motivation for having it be a type specific to Subprocess and not a more generally useful thing? It looks perfect for any sort of socket communication APIs (as an example).

5 Likes
4

I’m very excited for this proposal; it will essentially render the module I did to make Process usable useless! \o/
AFAICT all of the peeps I had with Process are fixed in this proposal.

I have a question regarding Linux compatibility: why is it not possible to add the same “escape hatch” than on Darwin? Are you not also using posix_spawn on Linux? Sending other fd than stdin/stdout would probably be useful on Linux too.

5

How is the environment lookup handled? I ask because the canonical spelling for the environment variable is Path on Windows (but environment variables are case insensitive). Do you take into account suffixes? e.g. does git allow the execution of git.cmd, git.bat, git.exe or does it only look for git?

5 Likes
6

The design of the StandardInputWriter is inspired by the design of the NIOAsyncChannelOutboundWriter which provides an almost identical API surface (except finish() being non-async which is something that I want to change in a future release).

We have seen a proliferation of writer types across the server ecosystem e.g. gRPCs [RPCWriter](https://github.com/grpc/grpc-swift/blob/main/Sources/GRPCCore/Streaming/RPCWriter.swift) and hummingbirds HBResponseBodyWriter

I agree that this pattern might be generalized under a common protocol such as an AsyncWriter similar to the AsyncSequence protocol. However, I personally believe before we introduce such a protocol we should investigate if first class async generators might be something that can solve both the reading and writing side behind one abstractions. For now I personally think it’s fine that every package is introducing their own writer types until we gain more experience with them.

3 Likes
7

It seems to me that many of these interfaces would be better placed in the System library - in particular, ProcessIdentifier and Signal seem very system-y to me.

Indeed, there was a draft PR to add ProcessID, Signal, and SignalSet types to swift-system. I also see that this proposal makes use of FilePath (big +1 to that, by the way), so it would already depend on types exposed by swift-system.

What I'd really like to avoid is that each library defines its own types and we have awkward conversion initialisers between them. Both libraries are part of Apple's SDK, and both are important parts of the swift ecosystem on other platforms, so it would be nice if they worked seamlessly together.

It may be that Foundation implements some kind of ergonomic, cross-platform wrapper atop the system types. For instance, @Michael_Ilseman speculated that perhaps we might like to model a process using an actor:

I've been thinking that a Process type would probably be more like an actor with interfaces for interprocess communication.

Maybe that's something that could live in Foundation?

Or maybe all of this goes in System?

12 Likes
8

This looks great! (I worked on Shwift, which solves some similar problems albeit aimed at my own use cases)

Unlike other Process alternatives I’ve seen, this seems to handle the infinite input problem well, great work!

One issue that might deserve closer consideration is the imperfect emulation of POSIX_SPAWN_CLOEXEC_DEFAULT on Linux. The pitch provides for additionalSpawnAttributeConfiguration which would enable setting POSIX_SPAWN_CLOEXEC_DEFAULT on macOS and seems to leave Linux to do its own thing. This is fine as this pitch only seeks to “eventually” replace Process, though we should think about how this impacts the defaults we choose. Some thoughts:

  1. Subprocess behavior should be as consistent as possible across platforms. I believe “what happens to open file descriptors” to be a pretty important thing to keep consistent, at least for macOS and Linux given their process models are so similar. I may be wrong, but either way we should be clear about what we expect to happen to open file descriptors even when something simple like git diff —name-only is executed.
  2. If we think this is a bug, there should be a plan in place to fix it. I have a solution that works for Shwift, but making it general-purpose enough to work for Foundation (which I’m not even sure is possible, and I’m happy to elaborate on this) is something that’s still on my to-do list (I’d also love to brainstorm solutions and work on one!)
  3. If we don’t think this is a bug that can be solved, we should adjust the API to reflect the different outcomes. Specifically, I think we should have a setting along the lines of closeBehaviorForOpenFileDescriptors and have close be unavailable on Linux, instead of allowing it to exist and having an incorrect implementation.
5 Likes
9

I'm very glad this is being worked on and I support the comments from people more involved than me. I wish to offer a minor but powerful bikeshed color: drop the executing: label from the common run(…) method:

let gitResult = try await Subprocess.run(.named("git"), arguments: [
  "diff", "--name-only"
])

In this metaphor the thing being run is the executable rather than the subprocess; both are common ways to talk about this kind of operation, but dropping the argument label leads to simpler call sites.

Other thoughts:

  • "signal" is a Unixism; is sendSignal(…) just unavailable on Windows? Can we have terminate() as a platform-independent thing?
  • Can we see some performance tables? I'm worried that basing things on Sequence-of-bytes and AsyncSequence-of-bytes won't be perfomant enough for input, even though I know they've been well-optimized for output by now.
  • Should there be a StandardInputWriter.write(…) overload for Strings? Sure, you can pass s.utf8, but that kind of stinks.
  • Given that the properties available on PlatformOptions will be different per-platform, it seems weird to offer a public initializer at all. Maybe everyone should start with .default and go from there.
  • The Environment.updating(…) overload taking [Data: Data] is a little weird; I understand that environment variable names may be non-UTF-8, but surely [String: Data] is more common? Maybe three overloads total?
  • The section on TerminationStatus refers to signalled in prose but unhandledException in the code example.
17 Likes
10

One other complication that might be worth considering: we may want to resolve executable paths and (to a lesser extent) environment prior to actually launching the subprocess. For example, a script may want to exit early if an executable is not found, so something like the following won't print "foo" if git isn't in the PATH.

// updated per @nevin’s comment
guard let git: Executable = .named("git") else {
   return nil
}
print("foo")
Subprocess.run(git, ...)

There are a bunch more situations like this, albeit more esoteric ones (including usage of the cursed setenv and friends).

My feeling is that Executable shouldn’t be a thing (for example an executable file can be deleted after the Executable is created… aren’t file systems just the best?), and instead we should have ExecutablePath or maybe just FilePath. There should be a mechanism for searching PATH for an ExecutablePath (could be an extension on FilePath as well). There’s a tiny bit of nuance here (for example, you need to skip non-executable files). So maybe something like the following:

let gitPath: FilePath = .findExecutableInEnvironmentPath(“git”)
print(“git definitely exists”)
Subprocess.run(gitPath, …)
7 Likes
11

I think it would feel more Swifty if this was written like:

guard let git = Executable(named: "git") else {
  ...
  return
}
print("foo")
Subprocess.run(git, ...)
2 Likes
12

I don't think that Process/Subprocess needs to be an actor. It doesn't have to provide its own isolation domain it merely has to make sure the I/O is happening on the right threads. We can solve that with task executor preferences.

I personally don't think this belongs into System since the goal is to provide one unified API across platforms with hooks to fine tune it for a specific platform. This is not what System aims to provide i.e. multi-platform vs cross-platform.

I agree that this is important and IMO we should take ownership of the file descriptors that a user passes in. Requiring a user to do the right descriptor handling is prone to errors.

I agree that perf numbers are important here. The API has been designed here to avoid any AsyncSequence's if we have file descriptors. That means we only pay the overhead of streaming into an AsyncSequence if the user asked us to do. If we just want to pipe from one process to another one we should do this without any in memory copy of the bytes.

Personally, I am not a big fan of adding this overload. There are many things that can be presented as some Sequence<UInt9> and IMO we are better of having that as a standardised API.

Some more comments after having given the proposed APIs another pass:

    public struct StandardInputWriter: Sendable {
        @discardableResult
        public func write<S>(_ sequence: S) async throws -> Int where S : Sequence, S.Element == UInt8

        @discardableResult
        public func write<S>(_ sequence: S) async throws -> Int where S : Sequence, S.Element == CChar

        @discardableResult
        public func write<S: AsyncSequence>(_ asyncSequence: S) async throws -> Int where S.Element == CChar

        @discardableResult
        public func write<S: AsyncSequence>(_ asyncSequence: S) async throws -> Int where S.Element == UInt8

        public func finish() async throws
    }
  • We probably shouldn't make the StandardInputWriter Sendable to enforce single producer patterns. User's can always create a multi producer by transferring the writer into a child task and creating an AsyncChannel
  • Do we need the CChar APIs?
  • Can we drop the Int return parameter. All bytes should be written when you call a write method and it returns without an error
  • Can we add some comments what finish is going to do here i.e. it allows for easy closing of the input fd of the process.
  • Can we add comments around the backpressure behaviour of the write methods and how they react to cancellation please.
public struct Subprocess: Sendable {
    public let processIdentifier: ProcessIdentifier
    // The standard output of the child process, expressed as AsyncSequence<UInt8>
    // This property is `nil` if the standard output is discarded or written to disk
    public var standardOutput: AsyncBytes? { get }
    // The standard error of the child process, expressed as AsyncSequence<UInt8>
    // This property is `nil` if the standard error is discarded or written to disk
    public var standardError: AsyncBytes? { get }

    public func sendSignal(_ signal: Signal) throws
}
  • Should we make Subprocess Sendable? AsyncBytes should probably not be Sendable and with the new region isolation just be transferred forward but never shared.
  • I would love to hear from others what they think about the standardOutput and standardError being optional here. The reason for this is that it depends on what has been passed to the run method. A potential solution that @icharleshu and I discussed is overloading the run method with all combinations of input/output methods (fd, inerhting, some Sequence, some AsyncSequence<UInt8>, etc.). However, that created a ton of overloads.
_ body: (@Sendable @escaping (Subprocess) async throws -> R)

Currently the body is always required to be @Sendable and @escaping. This feels wrong to me since we ought to run the subprocess in the current task. Can you expand where those two constraints come from.

Overall, really excited to see this moving forward!

1 Like
13

For the record, the issue I was bringing up concerns open descriptors which aren’t explicitly passed to the subprocess. On Linux, all open file descriptors are passed to the child process and there is no good thread safe way (as far as I understand) to mirror the Apple-specific “don’t pass any descriptors which aren’t explicitly duped into the child” behavior.

As far as file descriptor ownership goes, I don’t think there is anything wrong with borrowing a descriptor (I.e. not having the API take ownership). The API on Linux and macOS is called “dup2” meaning it’s a semantic duplicate. That said, I can’t think of a great reason to share descriptors between processes but I’m not sure this is a huge problem for the API since most folks will use the input and output types which abstract away the underlying file descriptor.

14

Worth noting that some mutable state (e.g. seek position) is also shared across dup/dup2 copies of a file descriptor as well. Even across processes.

2 Likes
16

You could consume output while the subprocess is running by using the "closure" approach:

let result = try await Subprocess.run(
    executing: .named("..."),
) { subprocess in
    // Here you can stream the standard output via subprocess.standardOutput
    // while the process is running.
    let output = try await Array(subprocess.standardOutput!)
    return output
}
17

That seems oddly complex - can you clarify why it can't be something simpler, like:

for path in try await Subprocess("find", searchPath, "-name", "*.avif").lines {
    …
}

i.e. avoid non-linear or buried execution contexts (closures, especially not escaping ones), use variadic arguments, etc.

I realise there are plenty of more advanced scenarios and behaviours desired, but it'd be nice if the simplest and most common cases are correspondingly simple and easy.

It also makes it easier to do interactive use, e.g.:

let subtool = try Subprocess("sometool", …)

for path in inputs {
    try subtool.stdin.write(path, terminator: "\n")
    let (isError, response) = try await subtool.readLine()
    …
}

(where readLine returns a tuple with a bool indicating if the line read - the next available in chronological order - was from stdout or stderr)

It might be wise to nudge people towards async use by default, to encourage better overall performance (particularly human-visible latency). I realise that there's currently a lot of overhead in basic tasks like async enumeration, but that's been raised in prior Forum threads and assurances provided that it will be addressed.

It'd also be good if the method for reading the output of a subprocess were very similar to reading your own stdin, so that it's easy to share code between programs intended as shell tools (re. pipelining) and other architectures.

5 Likes
18

That's a great question and the proposed API shape here is influenced by the learnings in other libraries such as swift-nio or grpc-swift when we adopted Swift Concurrency. To understand the underlying problem let's look at a fully async bidirectional streaming example:

try await Subprocess.run(executable: "foo") { standardInputWriter, standardOutput, standardError in
    // Push data into the process
    try await standardInputWriter.write("Hello".utf8)
    // Read data out
    for try await in standardOutput {
    ]
}

The reason why the standard input, output and error are only available inside this scope is due to the underlying resource management of the file descriptors and the process. We need to make sure that at the end of the scope of the run methods body closure everything is cleaned up. That means both making sure the file descriptors are properly closed and the process has terminated. At the moment this is only possible to spell through with methods that provide scoped access to a resource and make sure the resource is properly torn down at the end of the scope. Having the guarantee that a resource is torn down at a certain point of a program is really important otherwise it becomes impossible to reason about the current state of your application. Furthermore, worse than not being able to reason you might see sporadic failures like running out of file descriptors.

Let's take your proposed API and see where this becomes problematic:

let subtool = try Subprocess("sometool", …)

for path in inputs {
    try subtool.stdin.write(path, terminator: "\n")
    let (isError, response) = try await subtool.readLine()
    …
}

A few things that stand out:

  • When is the process being spawned here? Is it being spawned when I create a Subprocess?
  • What happens if I just drop the Subprocess on the floor? Will the process be signaled and do we wait for termination?
  • What are the file descriptors closed for stdin and stdout?

I have also tried to explore using ~Copyable + ~Escapable to model this so we don't require the scoping and that worked mostly until the resource teardown becomes async. With ~Copyable types it is possible to do it correctly for resources that require a sync cleanup but it is not yet possible to do it for async cleanups. In our case here, we need async cleans ups for some of the underlying resources and that can only be spelled using the with style scoped approach. Additionally, AsyncSequences can't be ~Copyable right now neither can they carry ~Copyable elements.

Now the proposal already does provide some convenience methods for patterns where we can make it easier for the user without requiring them to do the full blown scoped approach:

    public static func run<S: AsyncSequence>(
        executing executable: Executable,
        arguments: Arguments = [],
        environment: Environment = .inherit,
        workingDirectory: FilePath? = nil,
        platformOptions: PlatformOptions = .default,
        input: S,
        output: CollectedOutputMethod = .collect,
        error: CollectedOutputMethod = .collect
    ) async throws -> CollectedResult where S.Element == UInt8
5 Likes
19

I intentionally left the "how to search for executable path" bit vague in the proposal because this will be platform specific and should be considered an implementation detail. Right now it's mostly considers the PATH environments with some fall back.

There is also no way for you to retrieve the resolved path... I can add that. Thanks for the suggestion!

4 Likes
20

It is 100% possible. I need to explore whether we can use posix_spawn on Linux because I believe its only introduced in some recent versions of Glibc, which may not have WASM support. If we can use posix_spawn, the "escape hatches" will most likely be the same; if not, we might be looking at fork/exec which will have different escape hatches.

1 Like
21

Thank you for identifying and addressing a huge pain point.

I often use Process for streaming & channeled results. Sometimes without any predetermined termination.

Please, whatever implementation Subprocess adopts, be sure to allow a streaming result.

I want an API like:

let processTask = Subprocess.run(executable: "llm-tool", "--prompt", userPrompt)
do {
  for try await streamingResponse in processTask {
     handleResponse(streamingResponse)
  }
} catch {
  handleError(error)
}