# \[Pitch\] Swift Subprocess

**URL:** <https://forums.swift.org/t/pitch-swift-subprocess/69805>\
**Category:** Foundation\
**Created:** [February 2, 2024, 9:38pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805 "2024-02-02T21:38:47Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![icharleshu](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/icharleshu/32/20905_2.png) [@icharleshu](https://forums.swift.org/u/icharleshu)\
**Post date:** [February 2, 2024, 9:38pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/1 "2024-02-02T21:38:47Z")

</div>

Hi all,

I would like to pitch a new API, `Subprocess`, that that will eventually replace `Process` as the canonical way to launch a process in `Swift`. Please let me know if you have any questions/comments/thoughts!

* * *

# Introducing Swift Subprocess

- Proposal: SF-NNNN
- Authors: [Charles Hu](https://github.com/iCharlesHu)
- Review Manager: TBD
- Status: **Draft**
- Bugs: rdar://118127512, [apple/swift-foundation#309](https://github.com/apple/swift-foundation/issues/309)

## Introduction / Motivation

As Swift establishes itself as a general-purpose language for both compiled and scripting use cases, one persistent pain point for developers is process creation. The existing Foundation API for spawning a process, `NSTask`, originated in Objective-C. It was subsequently renamed to `Process` in Swift. As the language has continued to evolve, `Process` has not kept up. It lacks support for `async/await`, makes extensive use of completion handlers, and uses Objective-C exceptions to indicate developer error. This proposal introduces a new type called `Subprocess`, which addresses the ergonomic shortcomings of `Process` and enhances the experience of using Swift for scripting.

## Proposed Solution

We propose a new type, `struct Subprocess`, that will eventually replace `Process` as the canonical way to launch a process in `Swift`.

Here's a simple usage `Subprocess`:

```swift
#!/usr/bin/swift

import FoundationEssentials

let gitResult = try await Subprocess.run( // <- 0
    executing: .named("git"), // <- 1
    arguments: ["diff", "--name-only"]
)
if let output = gitResult.standardOutput { // <- 2
    print("Output: \(Strint(data: output, encoding: .utf8))")
}

```

Let's break down the example above:

1. `Subprocess` is constructed entirely on the `async/await` paradigm. The `run()` method utilizes `await` to allow the child process to finish, asynchronously returning an `ExecutionResult`. Additionally, there is an closure based overload of `run()` that offers more granulated control, which will be discussed later.
2. There are two primary ways to configure the executable being run:
  - The default approach, recommended for most developers, is `.at(FilePath)`. This allows developers to specify a full path to an executable.
  - Alternatively, developers can use `.named(String)` to instruct `Subprocess` to look up the executable path based on heuristics such as the `$PATH` environment variable.

3. The standard output and error are delivered either via `AsyncSequence` or `Data`.

I've posted the full pitch including the detailed design here on the [swift-foundation repo](https://github.com/apple/swift-foundation/pull/397). Check out the [**full pitch**](https://github.com/apple/swift-foundation/pull/397) to read more info about the proposed design.

* * *

# Trying it out

You can find a experimental (WIP) implementation here: [swift-experimental-subprocess](https://github.com/iCharlesHu/swift-experimental-subprocess)

Feel free to play around with it and let me know what you think!

---

<div class="post-metadata">

**Author:** ![wadetregaskis](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/wadetregaskis/32/8494_2.png) [@wadetregaskis](https://forums.swift.org/u/wadetregaskis)\
**Post date:** [February 2, 2024, 9:51pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/2 "2024-02-02T21:51:06Z")

</div>

Can you clarify what the code would look like to asynchronously enumerate the lines of the output of a subprocess? i.e. consuming each line as it's available, without waiting for the subprocess to exit.

---

<div class="post-metadata">

**Author:** ![davedelong](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/davedelong/32/38708_2.png) [@davedelong](https://forums.swift.org/u/davedelong)\
**Post date:** [February 3, 2024, 3:42pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/3 "2024-02-03T15:42:45Z")

</div>

I'm quite excited with this, as dealing with `Process` is definitely annoying. A couple of comments...

I appreciate the convenience of `Subprocess.Executable.named(...)` to perform `which`-like lookup for the binary, but I don't see a good way to get the actual discovered `FilePath`. How would I do that? Also, how does that lookup work? Does it only consider the `PATH` of the environment, or does it involve any user shell exports?

`StandardInputWriter` looks exactly like the sort of async `NSOutputStream` replacement we've been waiting for… Can you talk more about the motivation for having it be a type specific to `Subprocess` and not a more generally useful thing? It looks perfect for any sort of socket communication APIs (as an example).

---

<div class="post-metadata">

**Author:** ![Frizlab](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/frizlab/32/416_2.png) [@Frizlab](https://forums.swift.org/u/Frizlab)\
**Post date:** [February 3, 2024, 5:01pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/4 "2024-02-03T17:01:27Z")

</div>

I’m very excited for this proposal; it will essentially render [the module](https://github.com/xcode-actions/swift-process-invocation) I did to make `Process` usable useless! \o/  
AFAICT all of the peeps I had with `Process` are fixed in this proposal.

I have a question regarding Linux compatibility: why is it not possible to add the same “escape hatch” than on Darwin? Are you not also using `posix_spawn` on Linux? Sending other fd than stdin/stdout would probably be useful on Linux too.

---

<div class="post-metadata">

**Author:** ![compnerd](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/compnerd/32/2165_2.png) [@compnerd](https://forums.swift.org/u/compnerd)\
**Post date:** [February 3, 2024, 5:22pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/5 "2024-02-03T17:22:08Z")

</div>

How is the environment lookup handled? I ask because the canonical spelling for the environment variable is `Path` on Windows (but environment variables are case _insensitive_). Do you take into account suffixes? e.g. does `git` allow the execution of `git.cmd`, `git.bat`, `git.exe` or does it only look for `git`?

---

<div class="post-metadata">

**Author:** ![FranzBusch](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franzbusch/32/19447_2.png) [@FranzBusch](https://forums.swift.org/u/FranzBusch)\
**Post date:** [February 3, 2024, 9:52pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/6 "2024-02-03T21:52:24Z")

</div>

> [@davedelong](#):
>
> `StandardInputWriter` looks exactly like the sort of async `NSOutputStream` replacement we've been waiting for… Can you talk more about the motivation for having it be a type specific to `Subprocess` and not a more generally useful thing? It looks perfect for any sort of socket communication APIs (as an example).

The design of the `StandardInputWriter` is inspired by the design of the [NIOAsyncChannelOutboundWriter](https://github.com/apple/swift-nio/blob/main/Sources/NIOCore/AsyncChannel/AsyncChannelOutboundWriter.swift) which provides an almost identical API surface (except `finish()` being non-async which is something that I want to change in a future release).

We have seen a proliferation of writer types across the server ecosystem e.g. gRPC`s [RPCWriter](https://github.com/grpc/grpc-swift/blob/main/Sources/GRPCCore/Streaming/RPCWriter.swift) and hummingbird`s [HBResponseBodyWriter](https://github.com/hummingbird-project/hummingbird/blob/2.x.x/Sources/HummingbirdCore/Response/ResponseBody.swift)

I agree that this pattern might be generalized under a common protocol such as an `AsyncWriter` similar to the `AsyncSequence` protocol. However, I personally believe before we introduce such a protocol we should investigate if first class async generators might be something that can solve both the reading and writing side behind one abstractions. For now I personally think it’s fine that every package is introducing their own writer types until we gain more experience with them.

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [February 3, 2024, 10:50pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/7 "2024-02-03T22:50:16Z")

</div>

It seems to me that many of these interfaces would be better placed in the `System` library - in particular, `ProcessIdentifier` and `Signal` seem very system-y to me.

Indeed, there was [a draft PR](https://github.com/apple/swift-system/pull/20) to add `ProcessID`, `Signal`, and `SignalSet` types to swift-system. I also see that this proposal makes use of `FilePath` (big +1 to that, by the way), so it would already depend on types exposed by swift-system.

What I'd really like to avoid is that each library defines its own types and we have awkward conversion initialisers between them. Both libraries are part of Apple's SDK, and both are important parts of the swift ecosystem on other platforms, so it would be nice if they worked seamlessly together.

It may be that Foundation implements some kind of ergonomic, cross-platform wrapper atop the system types. For instance, @Michael_Ilseman speculated that perhaps we might like to model a process using an actor:

> I've been thinking that a `Process` type would probably be more like an `actor` with interfaces for interprocess communication.

Maybe that's something that could live in Foundation?

Or maybe _all_ of this goes in System?

---

<div class="post-metadata">

**Author:** ![George](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/george/32/37757_2.png) [@George](https://forums.swift.org/u/George)\
**Post date:** [February 4, 2024, 5:06am UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/8 "2024-02-04T05:06:39Z")

</div>

This looks great! (I worked on [Shwift](https://github.com/GeorgeLyon/Shwift), which solves some similar problems albeit aimed at my own use cases)

Unlike other Process alternatives I’ve seen, this seems to handle the [infinite input](https://github.com/GeorgeLyon/Shwift/blob/6f9377f0c4364adf012868a5e5f87dbb77d4baef/Sources/ScriptExample/Main.swift#L43) problem well, great work!

One issue that might deserve closer consideration is [the imperfect emulation of POSIX\_SPAWN\_CLOEXEC\_DEFAULT on Linux](https://github.com/apple/swift-corelibs-foundation/issues/3946). The pitch provides for `additionalSpawnAttributeConfiguration` which would enable setting POSIX\_SPAWN\_CLOEXEC\_DEFAULT on macOS and seems to leave Linux to do its own thing. This is fine as this pitch only seeks to “eventually” replace `Process`, though we should think about how this impacts the defaults we choose. Some thoughts:

1. `Subprocess` behavior should be as consistent as possible across platforms. I believe “what happens to open file descriptors” to be a pretty important thing to keep consistent, at least for macOS and Linux given their process models are so similar. I may be wrong, but either way we should be clear about what we expect to happen to open file descriptors even when something simple like `git diff —name-only` is executed.
2. If we think this is a bug, there should be a plan in place to fix it. I have [a solution that works for Shwift](https://github.com/GeorgeLyon/Shwift/tree/main/Sources/CLinuxSupport), but making it general-purpose enough to work for Foundation (which I’m not even sure is possible, and I’m happy to elaborate on this) is something that’s still on my to-do list (I’d also love to brainstorm solutions and work on one!)
3. If we _don’t_ think this is a bug that can be solved, we should adjust the API to reflect the different outcomes. Specifically, I think we should have a setting along the lines of `closeBehaviorForOpenFileDescriptors` and have `close` be unavailable on Linux, instead of allowing it to exist and having an incorrect implementation.

---

<div class="post-metadata">

**Author:** ![jrose](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jrose/32/408_2.png) [@jrose](https://forums.swift.org/u/jrose)\
**Post date:** [February 4, 2024, 8:10am UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/9 "2024-02-04T08:10:29Z")

</div>

I'm very glad this is being worked on and I support the comments from people more involved than me. I wish to offer a minor but powerful bikeshed color: drop the `executing:` label from the common `run(…)` method:

```swift
let gitResult = try await Subprocess.run(.named("git"), arguments: [
  "diff", "--name-only"
])

```

In this metaphor the thing being `run` is the executable rather than the subprocess; both are common ways to talk about this kind of operation, but dropping the argument label leads to simpler call sites.

* * *

Other thoughts:

- "signal" is a Unixism; is `sendSignal(…)` just unavailable on Windows? Can we have `terminate()` as a platform-independent thing?
- Can we see some performance tables? I'm worried that basing things on Sequence-of-bytes and AsyncSequence-of-bytes won't be perfomant enough for _input,_ even though I know they've been well-optimized for output by now.
- Should there be a `StandardInputWriter.write(…)` overload for Strings? Sure, you can pass `s.utf8`, but that kind of stinks.
- Given that the properties available on PlatformOptions will be different per-platform, it seems weird to offer a public initializer at all. Maybe everyone should start with `.default` and go from there.
- The `Environment.updating(…)` overload taking `[Data: Data]` is a little weird; I understand that environment variable names may be non-UTF-8, but surely `[String: Data]` is more common? Maybe three overloads total?
- The section on TerminationStatus refers to `signalled` in prose but `unhandledException` in the code example.

---

<div class="post-metadata">

**Author:** ![George](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/george/32/37757_2.png) [@George](https://forums.swift.org/u/George)\
**Post date:** [February 4, 2024, 5:04pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/10 "2024-02-04T17:04:50Z")

</div>

One other complication that might be worth considering: we may want to resolve executable paths and (to a lesser extent) environment prior to actually launching the subprocess. For example, a script may want to exit early if an executable is not found, so something like the following won't print "foo" if `git` isn't in the `PATH`.

```swift
// updated per @nevin’s comment
guard let git: Executable = .named("git") else {
   return nil
}
print("foo")
Subprocess.run(git, ...)

```

There are a bunch more situations like this, albeit more esoteric ones (including usage of the cursed `setenv` and friends).

My feeling is that `Executable` shouldn’t be a thing (for example an executable file can be deleted after the `Executable` is created… aren’t file systems just the best?), and instead we should have `ExecutablePath` or maybe just `FilePath`. There should be a mechanism for searching `PATH` for an `ExecutablePath` (could be an extension on `FilePath` as well). There’s a tiny bit of nuance here (for example, you need to skip non-executable files). So maybe something like the following:

```swift
let gitPath: FilePath = .findExecutableInEnvironmentPath(“git”)
print(“git definitely exists”)
Subprocess.run(gitPath, …)

```

---

<div class="post-metadata">

**Author:** ![Nevin](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nevin/32/617_2.png) [@Nevin](https://forums.swift.org/u/Nevin)\
**Post date:** [February 4, 2024, 6:18pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/11 "2024-02-04T18:18:34Z")

</div>

> [@George](#):
>
> For example, a script may want to exit early if an executable is not found, so something like the following won't print "foo" if `git` isn't in the `PATH`.
> 
> ```swift
> let git: Executable = .named("git")
> print("foo")
> Subprocess.run(git, ...)
> 
> ```

I think it would feel more Swifty if this was written like:

```swift
guard let git = Executable(named: "git") else {
  ...
  return
}
print("foo")
Subprocess.run(git, ...)

```

---

<div class="post-metadata">

**Author:** ![FranzBusch](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franzbusch/32/19447_2.png) [@FranzBusch](https://forums.swift.org/u/FranzBusch)\
**Post date:** [February 5, 2024, 3:15pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/12 "2024-02-05T15:15:36Z")

</div>

> [@Karl](#):
>
> > I've been thinking that a `Process` type would probably be more like an `actor` with interfaces for interprocess communication.
> 
> Maybe that's something that could live in Foundation?

I don't think that `Process/Subprocess` needs to be an `actor`. It doesn't have to provide its own isolation domain it _merely_ has to make sure the I/O is happening on the right threads. We can solve that with task executor preferences.

> [@Karl](#):
>
> Or maybe _all_ of this goes in System?

I personally don't think this belongs into `System` since the goal is to provide one unified API across platforms with hooks to fine tune it for a specific platform. This is not what `System` aims to provide i.e. `multi-platform` vs `cross-platform`.

> [@George](#):
>
> `Subprocess` behavior should be as consistent as possible across platforms. I believe “what happens to open file descriptors” to be a pretty important thing to keep consistent, at least for macOS and Linux given their process models are so similar. I may be wrong, but either way we should be clear about what we expect to happen to open file descriptors even when something simple like `git diff —name-only` is executed.

I agree that this is important and IMO we should take ownership of the file descriptors that a user passes in. Requiring a user to do the right descriptor handling is prone to errors.

> [@jrose](#):
>
> Can we see some performance tables? I'm worried that basing things on Sequence-of-bytes and AsyncSequence-of-bytes won't be perfomant enough for _input,_ even though I know they've been well-optimized for output by now.

I agree that perf numbers are important here. The API has been designed here to avoid any `AsyncSequence`'s if we have file descriptors. That means we only pay the overhead of streaming into an `AsyncSequence` if the user asked us to do. If we just want to pipe from one process to another one we should do this without any in memory copy of the bytes.

> [@jrose](#):
>
> Should there be a `StandardInputWriter.write(…)` overload for Strings? Sure, you can pass `s.utf8`, but that kind of stinks.

Personally, I am not a big fan of adding this overload. There are many things that can be presented as `some Sequence<UInt9>` and IMO we are better of having that as a standardised API.

Some more comments after having given the proposed APIs another pass:

```swift
    public struct StandardInputWriter: Sendable {
        @discardableResult
        public func write<S>(_ sequence: S) async throws -> Int where S : Sequence, S.Element == UInt8

        @discardableResult
        public func write<S>(_ sequence: S) async throws -> Int where S : Sequence, S.Element == CChar

        @discardableResult
        public func write<S: AsyncSequence>(_ asyncSequence: S) async throws -> Int where S.Element == CChar

        @discardableResult
        public func write<S: AsyncSequence>(_ asyncSequence: S) async throws -> Int where S.Element == UInt8

        public func finish() async throws
    }

```

- We probably shouldn't make the `StandardInputWriter` `Sendable` to enforce single producer patterns. User's can always create a multi producer by transferring the writer into a child task and creating an `AsyncChannel`
- Do we need the `CChar` APIs?
- Can we drop the `Int` return parameter. All bytes should be written when you call a `write` method and it returns without an error
- Can we add some comments what `finish` is going to do here i.e. it allows for easy closing of the input fd of the process.
- Can we add comments around the backpressure behaviour of the `write` methods and how they react to cancellation please.

```swift
public struct Subprocess: Sendable {
    public let processIdentifier: ProcessIdentifier
    // The standard output of the child process, expressed as AsyncSequence<UInt8>
    // This property is `nil` if the standard output is discarded or written to disk
    public var standardOutput: AsyncBytes? { get }
    // The standard error of the child process, expressed as AsyncSequence<UInt8>
    // This property is `nil` if the standard error is discarded or written to disk
    public var standardError: AsyncBytes? { get }

    public func sendSignal(_ signal: Signal) throws
}

```

- Should we make `Subprocess` `Sendable`? `AsyncBytes` should probably not be `Sendable` and with the new region isolation just be transferred forward but never shared.
- I would love to hear from others what they think about the `standardOutput` and `standardError` being optional here. The reason for this is that it depends on what has been passed to the `run` method. A potential solution that @icharleshu and I discussed is overloading the `run` method with all combinations of input/output methods (fd, inerhting, some Sequence, some `AsyncSequence<UInt8>`, etc.). However, that created a ton of overloads.

```swift
_ body: (@Sendable @escaping (Subprocess) async throws -> R)

```

Currently the body is always required to be `@Sendable` and `@escaping`. This feels wrong to me since we ought to run the subprocess in the current task. Can you expand where those two constraints come from.

Overall, really excited to see this moving forward!

---

<div class="post-metadata">

**Author:** ![George](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/george/32/37757_2.png) [@George](https://forums.swift.org/u/George)\
**Post date:** [February 5, 2024, 4:31pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/13 "2024-02-05T16:31:05Z")

</div>

> [@FranzBusch](#):
>
> I agree that this is important and IMO we should take ownership of the file descriptors that a user passes in. Requiring a user to do the right descriptor handling is prone to errors.

For the record, the issue I was bringing up concerns open descriptors which _aren’t_ explicitly passed to the subprocess. On Linux, all open file descriptors are passed to the child process and there is no good thread safe way (as far as I understand) to mirror the Apple-specific “don’t pass any descriptors which aren’t explicitly duped into the child” behavior.

As far as file descriptor ownership goes, I don’t think there is anything wrong with borrowing a descriptor (I.e. **not** having the API take ownership). The API on Linux and macOS is called “dup2” meaning it’s a semantic duplicate. That said, I can’t think of a great reason to share descriptors between processes but I’m not sure this is a huge problem for the API since most folks will use the input and output types which abstract away the underlying file descriptor.

---

<div class="post-metadata">

**Author:** ![David\_Smith](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/david_smith/32/37508_2.png) [@David\_Smith](https://forums.swift.org/u/David_Smith)\
**Post date:** [February 5, 2024, 10:17pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/14 "2024-02-05T22:17:34Z")

</div>

Worth noting that some mutable state (e.g. seek position) is also shared across dup/dup2 copies of a file descriptor as well. Even across processes.

---

<div class="post-metadata">

**Author:** ![icharleshu](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/icharleshu/32/20905_2.png) [@icharleshu](https://forums.swift.org/u/icharleshu)\
**Post date:** [February 7, 2024, 10:08pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/16 "2024-02-07T22:08:23Z")

</div>

You could consume output while the subprocess is running by using the "closure" approach:

```swift
let result = try await Subprocess.run(
    executing: .named("..."),
) { subprocess in
    // Here you can stream the standard output via subprocess.standardOutput
    // while the process is running.
    let output = try await Array(subprocess.standardOutput!)
    return output
}

```

---

<div class="post-metadata">

**Author:** ![wadetregaskis](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/wadetregaskis/32/8494_2.png) [@wadetregaskis](https://forums.swift.org/u/wadetregaskis)\
**Post date:** [February 8, 2024, 12:58am UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/17 "2024-02-08T00:58:51Z")

</div>

That seems oddly complex - can you clarify why it can't be something simpler, like:

```swift
for path in try await Subprocess("find", searchPath, "-name", "*.avif").lines {
    …
}

```

i.e. avoid non-linear or buried execution contexts (closures, especially not escaping ones), use variadic arguments, etc.

I realise there are plenty of more advanced scenarios and behaviours desired, but it'd be nice if the simplest and most common cases are correspondingly simple and easy.

It also makes it easier to do interactive use, e.g.:

```swift
let subtool = try Subprocess("sometool", …)

for path in inputs {
    try subtool.stdin.write(path, terminator: "\n")
    let (isError, response) = try await subtool.readLine()
    …
}

```

(where `readLine` returns a tuple with a bool indicating if the line read - the next available in chronological order - was from stdout or stderr)

It might be wise to nudge people towards async use by default, to encourage better overall performance (particularly human-visible latency). I realise that there's currently a lot of overhead in basic tasks like async enumeration, but that's been raised in prior Forum threads and assurances provided that it will be addressed.

It'd also be good if the method for reading the output of a subprocess were very similar to reading your own stdin, so that it's easy to share code between programs intended as shell tools (re. pipelining) and other architectures.

---

<div class="post-metadata">

**Author:** ![FranzBusch](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franzbusch/32/19447_2.png) [@FranzBusch](https://forums.swift.org/u/FranzBusch)\
**Post date:** [February 8, 2024, 11:10am UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/18 "2024-02-08T11:10:15Z")

</div>

> [@wadetregaskis](#):
>
> That seems oddly complex - can you clarify why it can't be something simpler, like:
> 
> ```swift
> for path in try await Subprocess("find", searchPath, "-name", "*.avif").lines {
> …
> }
> 
> ```

That's a great question and the proposed API shape here is influenced by the learnings in other libraries such as `swift-nio` or `grpc-swift` when we adopted Swift Concurrency. To understand the underlying problem let's look at a fully async bidirectional streaming example:

```swift
try await Subprocess.run(executable: "foo") { standardInputWriter, standardOutput, standardError in
    // Push data into the process
    try await standardInputWriter.write("Hello".utf8)
    // Read data out
    for try await in standardOutput {
    ]
}

```

The reason why the standard input, output and error are only available inside this scope is due to the underlying resource management of the file descriptors and the process. We need to make sure that at the end of the scope of the `run` methods `body` closure everything is cleaned up. That means both making sure the file descriptors are properly closed and the process has terminated. At the moment this is only possible to spell through `with` methods that provide scoped access to a resource and make sure the resource is properly torn down at the end of the scope. Having the guarantee that a resource is torn down at a certain point of a program is really important otherwise it becomes impossible to reason about the current state of your application. Furthermore, worse than not being able to reason you might see sporadic failures like running out of file descriptors.

Let's take your proposed API and see where this becomes problematic:

```swift
let subtool = try Subprocess("sometool", …)

for path in inputs {
    try subtool.stdin.write(path, terminator: "\n")
    let (isError, response) = try await subtool.readLine()
    …
}

```

A few things that stand out:

- When is the process being spawned here? Is it being spawned when I create a `Subprocess`?
- What happens if I just drop the `Subprocess` on the floor? Will the process be signaled and do we wait for termination?
- What are the file descriptors closed for stdin and stdout?

I have also tried to explore using `~Copyable` + `~Escapable` to model this so we don't require the scoping and that worked mostly until the resource teardown becomes `async`. With `~Copyable` types it is possible to do it correctly for resources that require a `sync` cleanup but it is not yet possible to do it for `async` cleanups. In our case here, we need `async` cleans ups for some of the underlying resources and that can only be spelled using the `with` style scoped approach. Additionally, `AsyncSequence`s can't be `~Copyable` right now neither can they carry `~Copyable` elements.

Now the proposal already does provide some convenience methods for patterns where we can make it easier for the user without requiring them to do the full blown scoped approach:

```swift
    public static func run<S: AsyncSequence>(
        executing executable: Executable,
        arguments: Arguments = [],
        environment: Environment = .inherit,
        workingDirectory: FilePath? = nil,
        platformOptions: PlatformOptions = .default,
        input: S,
        output: CollectedOutputMethod = .collect,
        error: CollectedOutputMethod = .collect
    ) async throws -> CollectedResult where S.Element == UInt8

```

---

<div class="post-metadata">

**Author:** ![icharleshu](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/icharleshu/32/20905_2.png) [@icharleshu](https://forums.swift.org/u/icharleshu)\
**Post date:** [February 8, 2024, 6:53pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/19 "2024-02-08T18:53:11Z")

</div>

> [@davedelong](#):
>
> I appreciate the convenience of `Subprocess.Executable.named(...)` to perform `which`-like lookup for the binary, but I don't see a good way to get the actual discovered `FilePath`. How would I do that? Also, how does that lookup work? Does it only consider the `PATH` of the environment, or does it involve any user shell exports?

I intentionally left the "how to search for executable path" bit vague in the proposal because this will be platform specific and should be considered an implementation detail. Right now it's mostly considers the `PATH` environments with some fall back.

There is also no way for you to retrieve the resolved path... I can add that. Thanks for the suggestion!

---

<div class="post-metadata">

**Author:** ![icharleshu](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/icharleshu/32/20905_2.png) [@icharleshu](https://forums.swift.org/u/icharleshu)\
**Post date:** [February 8, 2024, 6:56pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/20 "2024-02-08T18:56:12Z")

</div>

> [@Frizlab](#):
>
> I have a question regarding Linux compatibility: why is it not possible to add the same “escape hatch” than on Darwin? Are you not also using `posix_spawn` on Linux? Sending other fd than stdin/stdout would probably be useful on Linux too.

It is 100% possible. I need to explore whether we can use `posix_spawn` on Linux because I believe its only introduced in some recent versions of `Glibc`, which may not have `WASM` support. If we can use `posix_spawn`, the "escape hatches" will most likely be the same; if not, we might be looking at `fork/exec` which will have different escape hatches.

---

<div class="post-metadata">

**Author:** ![mcritz](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/mcritz/32/7094_2.png) [@mcritz](https://forums.swift.org/u/mcritz)\
**Post date:** [February 8, 2024, 6:57pm UTC](https://forums.swift.org/t/pitch-swift-subprocess/69805/21 "2024-02-08T18:57:45Z")

</div>

Thank you for identifying and addressing a huge pain point.

I often use Process for streaming & channeled results. Sometimes without any predetermined termination.

Please, **whatever implementation Subprocess adopts, be sure to allow a streaming result.**

I want an API like:

```swift
let processTask = Subprocess.run(executable: "llm-tool", "--prompt", userPrompt)
do {
  for try await streamingResponse in processTask {
     handleResponse(streamingResponse)
  }
} catch {
  handleError(error)
}

```

[Next page](https://forums.swift.org/t/pitch-swift-subprocess/69805.md?page=2)
