# \[Review\] SF-0007: Introducing Swift Subprocess

**URL:** <https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337>\
**Category:** Foundation\
**Created:** [February 28, 2024, 6:51pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337 "2024-02-28T18:51:43Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\
**Post date:** [March 2, 2024, 6:46pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/41 "2024-03-02T18:46:23Z")

</div>

Great. This is the deadlock issue you were talking about above.

```swift
import Foundation

func child() {
    let output = FileHandle.standardOutput
    let error = FileHandle.standardError
    try! error.write(contentsOf: Data(repeating: 0x41, count: 80_000))
    try! output.write(contentsOf: Data(repeating: 0x42, count: 10))
    RunLoop.current.run(until: .distantFuture)
}

func parent() {
    let process = Process()
    let inputPipe = Pipe()
    let outputPipe = Pipe()
    let errorPipe = Pipe()
    process.standardInput = inputPipe
    process.standardOutput = outputPipe
    process.standardError = errorPipe
    process.executableURL = Bundle.main.executableURL
    process.currentDirectoryPath = NSTemporaryDirectory()
    process.arguments = ["child"]
    try! process.run() // The file doesn’t exist.
    
    Task {
        print("before await loop")
        for try await v in outputPipe.fileHandleForReading.bytes {
            print("\(String(format: "%c", v)).", terminator: "")
        }
        print("after await loop")
    }
    RunLoop.current.run(until: .distantFuture)
}

func main() {
    if CommandLine.arguments.count > 1 && CommandLine.arguments[1] == "child" {
        child()
    } else {
        parent()
    }
}

main()

```

Turns out the error buffer size is around 64K.

Basically what's happening here is the child process blocks on writing to `stderr` and the parent process blocks on reading from `stdin`.

* * *

> **Same app rewritten without async await**
>
> ```swift
> import Foundation
> 
> func child() {
> let output = FileHandle.standardOutput
> let error = FileHandle.standardError
> try! error.write(contentsOf: Data(repeating: 0x41, count: 200_000))
> try! output.write(contentsOf: Data(repeating: 0x42, count: 10))
> RunLoop.current.run(until: .distantFuture)
> }
> 
> func parent() {
> let process = Process()
> let inputPipe = Pipe()
> let outputPipe = Pipe()
> let errorPipe = Pipe()
> process.standardInput = inputPipe
> process.standardOutput = outputPipe
> process.standardError = errorPipe
> process.executableURL = Bundle.main.executableURL
> process.currentDirectoryPath = NSTemporaryDirectory()
> process.arguments = ["child"]
> try! process.run() // The file doesn’t exist.
> let outputFile = outputPipe.fileHandleForReading
> let errorFile = errorPipe.fileHandleForReading
> 
> NotificationCenter.default.addObserver(forName: FileHandle.readCompletionNotification, object: nil, queue: nil) { note in
> guard let file = note.object as? FileHandle else { return }
> let fileName = if file == outputFile { "output" }
> else if file == errorFile { "error" }
> else { "other" }
> let data = note.userInfo![NSFileHandleNotificationDataItem] as! Data
> print("received \(fileName) data, count: \(data.count)")
> file.readInBackgroundAndNotify()
> }
> outputPipe.fileHandleForReading.readInBackgroundAndNotify()
> errorPipe.fileHandleForReading.readInBackgroundAndNotify()
>     
> RunLoop.current.run(until: .distantFuture)
> }
> 
> func main() {
> if CommandLine.arguments.count > 1 && CommandLine.arguments[1] == "child" {
> child()
> } else {
> parent()
> }
> }
> 
> main()
> 
> ```

Example output:

```swift
received error data, count: 131072
received error data, count: 65536
received output data, count: 10
received error data, count: 3392

```

No deadlock. Note that output and error data is interleaved. If I spend too much time within the notification callback the writer is (expectedly) blocked.

---

<div class="post-metadata">

**Author:** ![wadetregaskis](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/wadetregaskis/32/8494_2.png) [@wadetregaskis](https://forums.swift.org/u/wadetregaskis)\
**Post date:** [March 2, 2024, 8:45pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/42 "2024-03-02T20:45:32Z")

</div>

> [@tera](#):
>
> No deadlock. Note that output and error data is interleaved.

Right. And you can construct similar workarounds for async APIs too, e.g. using `merge` as was noted before (which is fortunately significantly less work than the older callback-based `Process` API, albeit likely less efficient).

The point is not so much that you _can't_ work around these deadlock issues, it's that most people don't realise they exist, so they _don't_. They then unknowingly roll the dice on whether their app is going to mysteriously deadlock out in the wild, possibly far away from their debugger. As is the nature of deadlocks - where things just come to a grinding halt without apparent explanation - debugging them is particularly fiendish and so it's _really_ valuable to not have deadlocks be possible to begin with.

Thus why I think it's worth `Subprocess` doing the extra little bit of work to merge the two pipes into one. If users want to then separate those back into separate `AsyncByte`s - though I'm not sure how likely that is or what its merits are, but for argument's sake - they _can_, without much difficulty. The result might be a little less efficient due to the intermediary merging - although not necessarily, depending on how well the optimiser does - but it will be safe from deadlocks (other than any the user introduces, but that's on them).

---

<div class="post-metadata">

**Author:** ![George](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/george/32/37757_2.png) [@George](https://forums.swift.org/u/George)\
**Post date:** [March 2, 2024, 8:58pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/43 "2024-03-02T20:58:59Z")

</div>

[`posix_spawn_file_actions_addclosefrom_np`](https://www.gnu.org/software/gnulib//manual/html_node/posix_005fspawn_005ffile_005factions_005faddclosefrom_005fnp.html) is interesting, but seems tricky. If you pass it a lowfiledes of 0 and then call the posix dup methods on some file descriptors are those guaranteed to remain open?

I assume the suggestion here is to `fork`, close all open file descriptors, then `exec` (similar to my solution using [clone\_vm](https://github.com/GeorgeLyon/Shwift/tree/main/Sources/CLinuxSupport) here). The big issue is that you may fork while another thread holds the malloc lock, or a swift runtime lock (I’ve run into both of these cases in practice). This is why I had to do the slow thing in my solution and iterate over all possible file descriptors (other APIs may allocate memory). Its a significant performance hit but the correctness was worth it in my case. Maybe there is some way we can “reset” the state of the program while preserving the subprocess configuration (like execing a trampoline) but these feel like research projects to me that shouldn’t hold up this important API.

I would recommend we make it an explicit API which would default to “close by default” on macOS, but on linux you’d have to pass `existingFileDescriptors: .leaveOpen` for now (there would be no other option). Once we address this issue (and I’m happy to contribute here) we can add `.close` as an option on Linux (and make it the default).

---

<div class="post-metadata">

**Author:** ![FranzBusch](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franzbusch/32/19447_2.png) [@FranzBusch](https://forums.swift.org/u/FranzBusch)\
**Post date:** [March 2, 2024, 10:21pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/44 "2024-03-02T22:21:37Z")

</div>

> [@wadetregaskis](#):
>
> Well, a lot of the things I've mentioned - like the restrictive closure, which - while it doesn't preclude any particular designs, technically - is not friendly to use-cases that inherently need to interact with the subprocess from many places. It imposes a burden, of having to manage a long-lived closure and set up yet more communication channels to talk to it. That sort of thing is easier (IMO) if done by e.g. an actor, not a closure.

I disagree with this statement and in my opinion this is something that we are going to see more and more with structured concurrency. Resources are going to be owned by individual tasks and when we need to interact with such a resource from a different task we need to use some kind of "channel" to communicate between the tasks. Most often I have been using `AsyncChannel` or `AsyncStream` for such an inter task communication.

> [@wadetregaskis](#):
>
> Remember that this isn't just 'daemons' (meaning processes that live indefinitely). Having the child process outlive the parent is also a common occurrence in helper tools, pipelines, and task systems.

This should be possible with the proposed APIs here. While the proposal does explicitly terminate the `Subprocess` on cancellation you can just spawn the `Subprocess` in an unstructured task that never gets cancelled and this will lead to the `Subprocess` outliving the parent process as long as the child process is not handling the parent process dying itself.

---

<div class="post-metadata">

**Author:** ![wadetregaskis](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/wadetregaskis/32/8494_2.png) [@wadetregaskis](https://forums.swift.org/u/wadetregaskis)\
**Post date:** [March 2, 2024, 11:06pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/45 "2024-03-02T23:06:17Z")

</div>

> [@FranzBusch](#):
>
> Resources are going to be owned by individual tasks and when we need to interact with such a resource from a different task we need to use some kind of "channel" to communicate between the tasks. Most often I have been using `AsyncChannel` or `AsyncStream` for such an inter task communication.

Those mechanisms are more difficult to use - and still immature - though. It takes quite a lot of care and research to figure out which to use, when, and what their limitations are (e.g. multiple consumers? back-pressure? etc). Academically I love them but pragmatically I minimise their use.

Working through closures introduces conceptual complexity from the most basic level (e.g. what does this `return` now mean?), problems of escaping, and retain cycles. Not to mention sendability, an area where Swift is still maturing. Swift 6 looks promising in that last regard, but it's going to take quite some time actually living on it for the jury to rule on whether sendability is 'solved'.

The Swift optimiser is also not yet great at making async-heavy code like that efficient. From what I've seen it mostly relies on being able to trivially inline everything and then sort through the result, but that rapidly becomes non-viable as you introduce multiple modules, more complex lifetimes and relationships, etc. And it seems like it's going to require manual assistance for peak performance (e.g. micro-managing executors).

Even if _none_ of the above ultimately turns out to be a practical concern, the fact that it has to be considered is a big problem with that approach.

There's a lot to be said for the simplicity of a simple class, or even an actor.

In my mind the big promise of async/await is the ability to write _straight-forward_ I/O code, that's easy to read because it lays everything out _sequentially_ (even though the implementation details may be quite complex). Use of channels, callbacks, escaping closures, etc, tends to damage that simplicity.

---

<div class="post-metadata">

**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\
**Post date:** [March 2, 2024, 11:20pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/46 "2024-03-02T23:20:14Z")

</div>

> [@wadetregaskis](#):
>
> Thus why I think it's worth `Subprocess` doing the extra little bit of work to merge the two pipes into one.

I don't see this as a full proof solution. Consider a more complicated case when you deal with two sub processes and have to merge the output stream of one and the error stream of another to not deadlock. Or you have to do a 4-way merge of the streams of the two subprocesses.

Or consider another simpler example:

child:

```swift
    try input.read(upToCount: 1)
    try output.write(contentsOf: Data([1]))

```

parent:

```swift
    try outputPipe.fileHandleForReading.read(upToCount: 1)
    try inputPipe.fileHandleForWriting.write(contentsOf: Data([1]))

```

Here "read" could be exchanged with "async await" loop without changing the meaning significantly. In any case this will deadlock. In general this deadlock is not avoidable, but in those cases when write doesn't depend on the result of read the deadlock could be avoided and callback/closure based approach solves this without a problem.

I agree that working with closures is more complicated.. We are just weighting them against something worse: deadlocks, and anything that minimises the chance of deadlocks is worth considering.

---

<div class="post-metadata">

**Author:** ![George](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/george/32/37757_2.png) [@George](https://forums.swift.org/u/George)\
**Post date:** [March 3, 2024, 12:03am UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/47 "2024-03-03T00:03:43Z")

</div>

> [@icharleshu](#):
>
> This is a great point and thank you for bringing it up. IMO `POSIX_SPAWN_CLOEXEC_DEFAULT` is essential enough (I believe `NSTask` uses it by default) that we should emulate on Linux

Sorry for the double-post but I just wanted to respond to this directly. I agree with this statement, but I'm not sure there is a clear answer for _how_ to emulate this on Linux. I think `Process`'s current, flawed solution **should not** be carried over to the new API. I do believe that "close other file descriptors" should be default. For Linux, in the absence of a workable solution (which is where I think we are), we should require explicitly providing that argument (`existingFileDescriptors: .leaveOpen` as recommended in my previous post, though I'm sure we can find better phrasing) until such a time that acceptable emulation (that is robust and performant enough to be a general-purpose solution) is available.

---

<div class="post-metadata">

**Author:** ![ksluder](https://avatars.discourse-cdn.com/v4/letter/k/eada6e/32.png) [@ksluder](https://forums.swift.org/u/ksluder)\
**Post date:** [March 3, 2024, 1:01am UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/48 "2024-03-03T01:01:56Z")

</div>

What about an argument that allowed you to pass in all the file descriptors you know about and want to take action on?

```swift
enum InheritedFileDisposition {
  @unavailable(platform: windows)
  enum Action {
    case close

    @unavailable(platform: linux)
    case closeAllOthers

    case inheritIfPossible

    case duplicate
  }

  @unavailable(platform: linux)
  case closeAll

  case inheritAllPossible

  @unavailable(platform: windows)
  case actions([FileHandle : Action])
}

extension InheritedFileDisposition : ExpressibleByDictionaryLiteral {
  init(dictionaryLiteral actions: [FileHandle : Action]) { .actions(actions) }
}

```

On Darwin:

- `closeAll` means “pass `POSIX_SPAWN_CLOEXEC_DEFAULT` to `posix_spawn`.”
- `inheritAllPossible` means “do not pass `POSIX_SPAWN_CLOEXEC_DEFAULT` or a `posix_spawn_file_actions_t` to `posix_spawn`.”
- A `close` action means “call `posix_spawn_file_actions_addclose()` with this file handle.”
- A `closeAllOthers` action means “in addition to the `posix_spawn_file_actions_t`, pass `POSIX_SPAWN_CLOEXEC_DEFAULT` to `posix_spawn`.”
- An `inheritIfPossible` action means “call `posix_spawn_file_actions_addinherit_np()` with this file descriptor.”
- A `duplicate` action means “call `posix_spawn_file_actions_adddup2()` with this file descriptor.”

On Linux:

- `closeAll` is not available.
- `inheritAllPossible` means “do not pass a `posix_spawn_actions_t` to `posix_spawn`.”
- A `close` action means “call `posix_spawn_file_actions_addclose` with this file descriptor.”
- `Action.closeAllOthers` is unavailable. Use `.closeAll` instead.
- An `inheritIfPossible` action does nothing with the provided file descriptor.
- A `duplicate` action means “call `posix_spawn_file_actions_adddup2()` with this file descriptor.”

On Windows:

- `closeAll` means “pass `FALSE` for the `bInheritHandles` argument to `CreateProcess`.”
- `inheritAllPossible` means “pass `TRUE` for the `bInheritHandles` argument to `CreateProcess`.”
- `Action` is unavailable:
  - To emulate `Action.close`, call `CloseHandle` in the spawned process.
  - Instead of `Action.closeAllOthers`, use `.closeAll` instead.
  - Instead of `Action.inheritIfPossible`, use `.inheritAllPossible` instead.
  - Instead of `Action.duplicate`, after `.run()` returns, call `DuplicateHandle()`, passing the newly created process’s handle as the target handle. Use IPC to send the handle value to the new process.

---

<div class="post-metadata">

**Author:** ![fclout](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/fclout/32/36155_2.png) [@fclout](https://forums.swift.org/u/fclout)\
**Post date:** [March 3, 2024, 1:24am UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/49 "2024-03-03T01:24:19Z")

</div>

I don't have a lot of comments, but I'm not sure how to feel that termination codes are `DWORD` on Windows and `CInt` on everything else. Would it be better to have the same type on all platforms?

---

<div class="post-metadata">

**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\
**Post date:** [March 3, 2024, 2:10am UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/50 "2024-03-03T02:10:01Z")

</div>

> [@tera](#):
>
> ~~BTW, why they can't be infinite? Not literally, but limited only by available memory. We have no problem that when we create a class the whole process could be terminated on out of memory error, same could happen if memory overflows in read/write buffers.~~ On the second thought not a good idea.

On the third thought, why can't those buffers be "infinite" indeed?

I can use an array in my app, and append to the end of that array from one thread and remove from the head of that array from another thread (let's consider the array synced with a lock so thread safety is not an issue, and let's ignore the O(n) behaviour of removing from head). Obviously when the writer speed is higher than the reader speed eventually the array will overflow memory and crash the app by doing so. Blocking the too speedy writer while possible is not necessarily the only right approach (\*) (\*\*). I wonder why the "infinite" buffer size approach is not considered when writing to another process over a pipe or stream? Can it not be an option? e.g. `"pipe.bufferSize = 1234"` or `"pipe.bufferSize = nil"` (for unbounded buffer).

(\*) and if we ever introduce FixedArray I doubt we would block (in a broad meaning of that word) when appending to a full array.

(\*\*) another approach could be exposing the number of currently "unread" bytes and let the writer decide what to do if it considers that number to be too high.

---

<div class="post-metadata">

**Author:** ![wadetregaskis](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/wadetregaskis/32/8494_2.png) [@wadetregaskis](https://forums.swift.org/u/wadetregaskis)\
**Post date:** [March 3, 2024, 2:25am UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/51 "2024-03-03T02:25:01Z")

</div>

> [@tera](#):
>
> I don't see this as a full proof solution. Consider a more complicated case when you deal with two sub processes and have to merge the output stream of one and the error stream of another to not deadlock. Or you have to do a 4-way merge of the streams of the two subprocesses.

Indeed, it's not fool proof. I'm not sure how `Subprocess` could reasonably protect against that scale of problem, though. It seems it'd require contextual knowledge, or a very different API (that _enforces_ that all pipes for all subprocesses are `select`ed upon simultaneously, never separately…?).

> [@tera](#):
>
> Or consider another simpler example:
> 
> child:
> 
> ```swift
> try input.read(upToCount: 1)
> try output.write(contentsOf: Data([1]))
> 
> ```
> 
> parent:
> 
> ```swift
> try outputPipe.fileHandleForReading.read(upToCount: 1)
> try inputPipe.fileHandleForWriting.write(contentsOf: Data([1]))
> 
> ```

That's just a bug; a logic error. It's similar to e.g. having two locks and not always ordering them consistently. Or opening a network connection and then both sides sit there in silence waiting for the other one to speak first. There has to be an agreed-upon and followed protocol between the two parties.

Aside from the principle of it (that such a bug is the user's fault, not the library's), practically-speaking I don't know that `Subprocess` could do much to defend against this. It has no insight into the other end of the pipes, so it can't even detect the deadlock at runtime. It similarly couldn't tell you that you're doing it wrong if you fail to issue the write entirely (short of changing the API to require _all_ input up-front, which would be terribly limiting).

> [@tera](#):
>
> …in those cases when write doesn't depend on the result of read the deadlock could be avoided and callback/closure based approach solves this without a problem.

Maybe. But probably there'll be a synchronisation point, a "[needs to] happen before" dependency (between writing and reading) at _some_ place in the parent program, so the potential for these kinds of ordering errors is always there.

If writing and reading truly are independent, then the natural thing to do would be to use them independently, from different tasks / threads - i.e. it's intuitive to not couple writing and reading. So it's less likely that a user will unwittingly create a bug.

---

<div class="post-metadata">

**Author:** ![wadetregaskis](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/wadetregaskis/32/8494_2.png) [@wadetregaskis](https://forums.swift.org/u/wadetregaskis)\
**Post date:** [March 3, 2024, 2:41am UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/52 "2024-03-03T02:41:04Z")

</div>

> [@tera](#):
>
> On the third thought, why can't those buffers be "infinite" indeed?

It's hard to say. A bit like whether callstacks should be [virtually] infinite or not.

There's always the possibility of pathological cases, e.g. a subprocess that's buggy or badly designed such that it goes into an infinite loop of continuous output. It's not great if those get dragged out, noticeably impacting the entire computer in the process (by causing swapping etc).

I do think having the buffer size be configurable is wise, as that can be simply necessary to achieve good performance (64 kB is _really_ small in terms of how long it takes to fill that space). That configurability could also be abused in attempts to fix deadlock issues, which is unfortunate but probably not justification to disallow it.

Having it be finite might be helpful in encouraging people to _not_ naively or lazily try to buffer things; it's generally better if they at least _try_ to process inputs as they come in, for overall performance (both throughput and latency) and to minimise negative effects on the computer as a whole (like unnecessary swapping).

Also, it's entirely possible for a program's output to be genuinely infinite. e.g. a program that emits as many pseudorandom bytes as you like, much like /dev/random. To be able to safely invoke such programs the API _must_ support finite buffers.

---

<div class="post-metadata">

**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\
**Post date:** [March 3, 2024, 2:55am UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/53 "2024-03-03T02:55:51Z")

</div>

> [@wadetregaskis](#):
>
> It's hard to say. A bit like whether callstacks should be [virtually] infinite or not.

Or heap!

> [@wadetregaskis](#):
>
> There's always the possibility of pathological cases, e.g. a subprocess that's buggy or badly designed such that it goes into an infinite loop of continuous output. It's not great if those get dragged out, noticeably impacting the entire computer in the process (by causing swapping etc).

I found it inconsistent that we don't apply this logic in a simpler example:

```swift
    var array = [0]
    for i in 1 ..< Int.max {
        array.append(i)
    }

```

If we did – the append would either block or crash or it would be throwable and throw well before the app runs out of RAM and causes excessive swapping.

---

<div class="post-metadata">

**Author:** ![Jean-Daniel](https://avatars.discourse-cdn.com/v4/letter/j/977dab/32.png) [@Jean-Daniel](https://forums.swift.org/u/Jean-Daniel)\
**Post date:** [March 3, 2024, 8:08am UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/54 "2024-03-03T08:08:49Z")

</div>

> [@George](#):
>
> [`posix_spawn_file_actions_addclosefrom_np`](https://www.gnu.org/software/gnulib//manual/html_node/posix_005fspawn_005ffile_005factions_005faddclosefrom_005fnp.html) is interesting, but seems tricky. If you pass it a lowfiledes of 0 and then call the posix dup methods on some file descriptors are those guaranteed to remain open?

I'm not sure, but the common use case which is to close all files descriptors can be reliably implemented using it.

And if you need more control over stdin/stdout/stderr, it is easy to use `posix_spawn_file_actions_addclosefrom_np` with an argument of 3, and add other actions for fd 0, 1 and 2.

---

<div class="post-metadata">

**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\
**Post date:** [March 3, 2024, 1:25pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/55 "2024-03-03T13:25:00Z")

</div>

> [@icharleshu](#):
>
> Our goal is to eventually replace the existing `Process`

Given this goal – Subprocesses must support (as an opt-in or opt-out) the use case of Process: child subprocesses outliving parent process.

* * *

Something that's unclear to me after reading the pitch. I can see the async/await version for writing the child process input from the parent process. How would the child process write to its output/error in a non blocking async/await manner? We have `"print("hello")"` or `"FileHandle.standardOutput.write(...)"` but both are blocking calls.

* * *

I'm not sure I like the proliferation of the run methods. Would it be possible to modify (combine) input / output methods like so:

```swift
enum InputMethod {
    case noInput
    case readFrom(FileDescriptor, closeWhenDone: Bool)
    case readFrom(Pipe)
    case readFrom(sequence: any Sequence<UInt8>)
    case readFromWriter(Subprocess, StandardInputWriter)
}

enum OutputMethod {
    case discard
    case redirect
    case writeToData
    case writeToDataWithLimit(limit: Int)
    case writeTo(FileDescriptor, closeWhenDone: Bool)
    case writeTo(Pipe)
}

```

(using the enum version of those just for simplicity, they could be structs)

Otherwise I don't think the API is flexible enough to support, say, getting input from a file, outputting output to the pipe (or async/await reader) and outputting error to a file.

* * *

With Process we have the flexibility of:

```swift
let child = Process(...)
child.run()
...
// in another part of the app:
child.input.write(...)

```

without being constraint by doing that in a closure of "run". I believe that this flexibility is a good thing.

---

<div class="post-metadata">

**Author:** ![George](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/george/32/37757_2.png) [@George](https://forums.swift.org/u/George)\
**Post date:** [March 3, 2024, 8:42pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/56 "2024-03-03T20:42:10Z")

</div>

I did a quick-and-dirty test of this API in my Shwift package:

> <https://github.com/GeorgeLyon/Shwift/blob/b4d55b14e4bc2f523caeec70a350f72e258e5f93/Sources/Shwift/Process.swift#L123-L130>

Passing `3` for `lowfiledes` does indeed close file descriptors (can be tested via ScriptExample's stressTest). Unfortunately, passing `0` and then duping some file descriptors does not result in those descriptors being open in the child process.

What this means is for the common case where we just care about stdin, stdout and stderr this will work fine. In the less common case where you are passing another file descriptor things get murky. As you see in the example, simply closing a range of file descriptors seems to work, so if the file descriptor you want to pass has a value of "42" you can do `for i in 3..<42 { ps_..._close(42) }; ps_closefrom(43)`. If you want to pass a file descriptor with the value of `UInt32.max` for some ungodly reason, this becomes less reasonable.

Given all of this, maybe the right approach is to use `closefrom` if the file descriptors that you want to pass are consecutive from 0, and throw `ENOSYS` if the file descriptors are nonconsecutive (similar to how `ENOSYS` is thrown here in [tools-support-core](https://github.com/apple/swift-tools-support-core/blob/main/Sources/TSCclibc/process.c)). This would make it work for almost all use cases (I can't come up with a practical example where you would want to pass an extra file descriptor with a high numeric value) but will fail explicitly if the API consumer requests something weird that can't be robustly supported.

**Update:** Polished this up and added it to Shwift, seems to be working great!

> <https://github.com/GeorgeLyon/Shwift/blob/edc8e7651d50e92588cecd785755f19a41672d08/Sources/Shwift/Process.swift#L121-L141>

---

<div class="post-metadata">

**Author:** ![prachi](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/prachi/32/10581_2.png) [@prachi](https://forums.swift.org/u/prachi)\
**Post date:** [March 4, 2024, 3:38am UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/57 "2024-03-04T03:38:53Z")

</div>

> [@icharleshu](#):
>
> I'm torn between using `executable` or not having one at all.

I’d prefer no label, but I think that would necessitate renaming the enum cases or function to help it read better:

```swift
// Only dropping label
_ = try await Subprocess.run(.at("/usr/bin/yes"), …) 
_ = try await Subprocess.run(.named("yes"), …)

// Renamed function
_ = try await Subprocess.runExecutable(.at("/usr/bin/yes"), …)
_ = try await Subprocess.runExecutable(.named("yes"), …)

// Renamed cases
_ = try await Subprocess.run(.executableAt("/usr/bin/yes"), …)
_ = try await Subprocess.run(.executableNamed("yes"), …)

```

The dropped label solution still has the PWD confusion I mentioned before. As such, I prefer the renamed function, but it also feels redundant: when aren’t you running an executable?

---

<div class="post-metadata">

**Author:** ![LiarPrincess](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/liarprincess/32/22594_2.png) [@LiarPrincess](https://forums.swift.org/u/LiarPrincess)\
**Post date:** [March 5, 2024, 4:08pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/58 "2024-03-05T16:08:20Z")

</div>

In the old thread I wrote a [long post](https://forums.swift.org/t/pitch-swift-subprocess/69805/53) about some issues that I see in the proposal. I think most of the problems there are still relevant, so I would love to see them discussed here.

This post works differently. Each section starts with a `Subprocess.run` snippet that will do one of:

- crash child
- crash parent
- deadlock child-parent
- deadlock parent
- make Swift Concurrency inoperable

This is going to be a long one, but you can run all of the code examples by yourself. "Pride and Prejudice.txt" is available at Project Gutenberg. Ubuntu 22.04.4 LTS, should work on macOS (it is actually even easier to deadlock there).

## `Collected` - `limit` deadlocks

```swift
try await Subprocess.run(
  executing: .at(FilePath("\(bin)/cat")),
  arguments: ["Pride and Prejudice.txt"]
)

print("after")

```

**Expected:** prints "after"

**Result:**

- nothing is printed
- it seems to be hanging
- `cat` is visible in System Monitor even after a few minutes; deadlock

If we open in in the debugger we will see that it hangs on:

```swift
internal func captureStandardError() throws -> Data? {
  guard case .collected(let limit, let readFd, _) = self.executionError else {
    return nil
  }
  let captured = try readFd.read(upToLength: limit) // <-- here; limit is 131072
  return Data(captured)
}

```

This is a blocking `FileDescritor.read` on a cooperative thread. It should not deadlock, because when the `cat` finishes it should just resume. (That said, a blocking read would not be my 1st choice in this case.)

The culprit is somewhere else. As it turns out the `limit` that we specify does not only regulate the amount of bytes that we care about, but also sets an indirect limit on how much the process can write in total:

- `stdout <= limit` - 🟢 ok
- `stdout <= limit + pipe size` - 🟢 ok, but the output is truncated
- `stdout > limit + pipe size` - 🔴 deadlock

Am I the only one surprised by it? When I read the proposal I though of: `git status limit(500)` will just take the first 500 bytes of the output and forget about the rest. But it turns out that "the rest" is critical, as it decides about the deadlock.

And remember that we have to set a limit. Either directly via the `collect(limit: Int)` or indirectly via `collect`. That said the proposal says: "By default, this limit is 16kb", but in code we have `.collected(128 * 1024)`. So, to get 16kb I have to specify `128 * 1024` as an argument. What is the unit?

How about using `Int.max` as a limit? Or `Int.min`? Or `-1`? All will crash with `SIGILL`: illegal instruction operand. The code in PR will actually try to allocate `Int.max` of memory:

```swift
extension FileDescriptor {
  internal func read(upToLength maxLength: Int) throws -> [UInt8] {
    let buffer: UnsafeMutableBufferPointer<UInt8> = .allocate(capacity: maxLength)
    let readCount = try self.read(into: .init(buffer))
    let resizedBuffer: UnsafeBufferPointer<UInt8> = .init(start: buffer.baseAddress, count: readCount)
    return Array(resizedBuffer)
  }
}

```

But what if we guess the total size of the `stdout` wrong? It not only deadlocks, but also takes 2 cooperative threads with it: the `read(stderr)` one and the `waitpid` one.

The conditions for a deadlock are:

- we have a "chatty program" - it prints a lot
- we underestimated the output by pipe buffer size

Enter… the Swift compiler. It is a chatty program, and immediately upon seeing the proposal I thought of:

1. git clone
2. cd
3. swift build

The standard CI stuff. This will work… for a while. But then our app will grow bigger: we will add dependencies, we will add more `.swift` files, and it will cross the max `stdout` and deadlock. If this does not convince you then just use `swift test`, this will 100% deadlock after a few weeks.

But you can always cancel the `Task` which will kill the process and reclaim the cooperative threads! No, you can't. It is a race condition. Going back to our CI example: lets say we implement a monitor in Swift that will `kill` the `Process/Task` after 30 min (seems like a reasonable upper bound for most of the `swift build` uses). What if during those 30 min users submitted enough builds to choke the Swift concurrency? Will the Monitor work correctly?

The only _reliable_ way to find out about this is from the external sources. Maybe the row in the database did not appear? Maybe the users are frustrated that the CI is not working? Etc.

Also, look how easy this was! I just used `swift build`. No maliciously crafted arguments. No specially prepared programs. Everything with default values, and it deadlocked. Where does this put our user? Do they know what went wrong? Do they have some information? Can they at least link this with `stdout`? No.

In the long post in the pitch thread I proposed changing the defaults to: `input: .noInput` and `output: .discard` for this exact reason. At least it will not deadlock. To deadlock they have to explicitly set the `output: .collect(limit: …)`, which is more visible. I had a discussion about this with @wadetregaskis, and their stance is (I cropped it to save space, please go see the old thread):

> .`discard` is a terrible default, especially for stderr, because it will cause many people (mostly but not exclusively beginners) to unwittingly miss that their subprocess error'd out.

I would agree with this if we had a "read all and never deadlock" or "read 64kb and never deadlock" option. That would be a perfect default value. But we don't. Btw. why not? As I explained in the long post this should be technically possible. Currently we only have: "read X, but sometimes deadlock when X + _platform\_specific\_value_ do not match".

Whatever option we choose (as far as the default values go) can we at least make them the same? In the proposal we have (this is the variant with the `body` closure):

```swift
public static func run<R>(
  …
  output: RedirectedOutputMethod = .redirect,
  error: RedirectedOutputMethod = .discard,
  _ body: (@Sendable @escaping (Subprocess) async throws -> R)
) async throws -> Result<R>

```

I had a 100% deadlock scenario that I wanted to include in this post. The problem is that it did not deadlock. Hmm… It took me a few minutes to realize that the default for `output` is `redirect` and for `error` is `discard`. I forgot about this, I just copied the `standardOutput` code, changed to `standardError` and thought it would work. In other words: I used the discarding `standardError` thinking that it is `redirect`, because that was the default for `standardOutput`.

This makes it easy for users to miss `stderr`, because they forgot that the default is `discard`. No compiler error, no precondition, no exception, no assertion, it just returns `nil` that my autocomplete discarded with `p.standardError?.xxx`. It looked correct, but in fact it was a dead code, because the stream was redirected to `/dev/null` and not to my handler.

## `Collected` - reading deadlocks

This will be our child program, it will write 70\*1000 bytes to `stderr`:

```swift
import SystemPackage

let buffer = [UInt8](repeating: 0, count: 1000)

for _ in 0..<70 {
  try FileDescriptor.standardError.writeAll(buffer)
}

```

Parent:

```swift
let result = try await Subprocess.run(
  executing: .at(FilePath(".build/debug/princess_is_in_another_stream"))
)

```

**Expected:** In the result variable:

- `stdout` is empty
- `stderr` has 70\_000 bytes

**Result:** deadlock

We are doing a blocking read on `stdout`, while the child writes to `stderr`, fills the pipe and waits for somebody to read.

How probable is this? I don't know. Can we guarantee that no such program exist? No.

I kind of feel that the difference between `stdout` and `stderr` is blurry. [glibc says](https://www.gnu.org/software/libc/manual/html_node/Standard-Streams.html):

> Variable: FILE \* stderr
> 
> The standard error stream, which is used for error messages and diagnostics issued by the program.

For example Swift Package Manager writes this to _stderr_:

```swift
Fetching https://github.com/apple/swift-system from cache
Computing version for https://github.com/apple/swift-system
Fetched https://github.com/apple/swift-system (0.65s)
Computed https://github.com/apple/swift-system at 1.2.1 (0.29s)
Creating working copy for https://github.com/apple/swift-system
Working copy of https://github.com/apple/swift-system resolved at 1.2.1

```

For them this is a diagnostic, and programs are allowed to print as many of those as they want. Nothing wrong has happened, there is no error. But once they reach the pipe buffer size it will deadlock. And I will not blame the Swift Package Manager team for this.

Maybe we should read the `stdout/stderr` in parallel. Maybe we should do something else. But I do not feel like a deadlock is a valid thing to do. Again, notice that I am using the default values for all of the `Subprocess.run` arguments.

## `body` closure - SIGPIPE in child

```swift
let result = try await Subprocess.run(
  executing: .at(FilePath("\(bin)/cat")),
  arguments: ["Pride and Prejudice.txt"]
) { _ in
  return 1
}

print(result.terminationStatus)

```

**Expected:** prints "0"

**Result:** prints "unhandledException(13)" \<-- `SIGPIPE`

In general this is will happen when `body` closure exits before the last `stdout` from the child. We are basically using `stdout/stderr` as a synchronization primitives.

I'm not really sure if by looking at the code I would expect crashing the child with `SIGPIPE`. This behavior may be valid, but neither the documentation nor the proposal mention it. I would call it _surprising_. And it leaves user with no information about the _why_. Where is the error? I could see somebody wasting many hours on this.

## `body` closure - `stdin` deadlock

```swift
let result = try await Subprocess.run(
  executing: .at(FilePath("program"))
) { p, writer in
  return 1
}

```

**Expected:** Did you notice the change from the program in the previous section? It is very subtle.

**Result:** Possible deadlock depending on what the `program` is.

The call to `try await writer.finish()` is missing. If `program` loops infinitely reading `stdin` and performing some actions, then it will never finish, because we forgot to close the `stdin`. And because, `writer.finish()` is `try await`, we can't put it in `defer`.

In the long post I mentioned that I do not see the reason for `finish/close` call to be mandatory. It should be optional. We could make the this method idempotent and make sure that the `Subprocess` will ALWAYS call it.

Scenario 1:

1. User calls `close` -\> we close the file
2. `Subprocess` calls `close` -\> nothing happens

Scenario 2:

1. User forgets to call `close`
2. `Subprocess` calls `close` -\> we close the file

For example:

```swift
actor StandardInputWriterActor {
  private let fileDescriptor: FileDescriptor
  private var isClosed = false

  public func close() async throws {
    if !self.isClosed {
      self.isClosed = true
      try self.fileDescriptor.close()
    }
  }
}

```

Now we just need to find a good spot for the `Subprocess` to call it. We can call `close` as many times as we want, only the 1st one matters, this way we go from:

> User HAS to call `close` otherwise possible deadlock.

To:

> User MAY call `close`. Subprocess will call it ALWAYS. No deadlock.

In the old thread I had a talk about this with @wadetregaskis, and their final stance was:

> It's not about calling it multiple times, it's about ensuring it's called [at least] once.
> 
> Some cases don't care if it's called - maybe the subprocess in question doesn't wait on stdin and will exit regardless. But it doesn't hurt to close stdin in that case, and it's safer to err that way.

`Subprocess` will call it! That's the whole point of making it idempotent. We can call it even if the user has already called it. But if they forgot, then there is always a safety-call done by the `Subprocess`.

## `body` closure - `stdout` deadlock

For this one let's ignore the `SIGPIPE`-ing the child for a second:

```swift
let result = try await Subprocess.run(
  executing: .at(FilePath("\(bin)/cat")),
  arguments: ["Pride and Prejudice.txt"],
  // I typed the explicit '.redirect', because the default values are
  // different for 'output' and 'error'. Just to avoid confusion.
  output: .redirect,
  error: .redirect
) { p in
  return 1
}

```

Deadlock. Oh… we need to read the pipes. This will also prevent `SIGPIPE`.

```swift
let result = try await Subprocess.run(
  executing: .at(FilePath("\(bin)/cat")),
  arguments: ["Pride and Prejudice.txt"],
  output: .redirect,
  error: .redirect
) { p in
  // There is no "read all and discard" or "forEach", I will just use 'reduce'.
  // '!' because YOLO. It's not like I can handle 'nil' meaningfully anyway.
  try await p.standardOutput!.reduce(()) { (_, _) in () }
  try await p.standardError!.reduce(()) { (_, _) in () }
  return 1
}

```

It will work… in this case. But at this point we all know that it is not correct. The `stderr` may fill and deadlock. I will use the same program as before to write 70\*1000 bytes to `stderr`:

```swift
let result = try await Subprocess.run(
  executing: .at(FilePath(".build/debug/princess_is_in_another_stream")),
  output: .redirect,
  error: .redirect
) { /* Same as above. */ }

```

Deadlock. We need to read them in parallel!

```swift
let result = try await Subprocess.run(
  executing: .at(FilePath(".build/debug/princess_is_in_another_stream")),
  output: .redirect,
  error: .redirect
) { p in
  // 'withDiscardingTaskGroup' is only on macOS 14
  try await withThrowingTaskGroup(of: Void.self) { group in
    group.addTask { try await p.standardOutput!.reduce(()) { (_, _) in () } }
    group.addTask { try await p.standardError!.reduce(()) { (_, _) in () } }
    try await group.waitForAll()
  }

  return 1
}

```

It works! As far as I know this is deadlock proof.

Now, if we use the closure with `StandardInputWriter` then our total boilerplate is:

```swift
try await writer.finish()

try await withThrowingTaskGroup(of: Void.self) { group in
  group.addTask { try await p.standardOutput!.reduce(()) { (_, _) in () } }
  group.addTask { try await p.standardError!.reduce(()) { (_, _) in () } }
  try await group.waitForAll()
}

```

We have to do this every single time we use the `body` closure overload to avoid `SIGPIPE` or deadlock. Are we sure that our users will be able to write it? It looks quite complicated.

## SIGPIPE - parent

There is an obvious `SIGPIPE` on the parent, but we can't do anything about it. If we tried to convert it into `EBADF` that would be a race condition. I would take "always `SIGPIPE`" over "`EBADF` but once in a blue moon `SIGPIPE`".

That said, I see no reason why we allow sending signals after we confirmed the process termination. Pid reuse is a "theoretical" issue, but still an issue. (I mentioned this in my long post.)

## Multiple processes - deadlock/crash/custom scheduler?

This module does a lot of blocking on cooperative threads: `waitpid`, `read`, `write`. From what I see only `AsyncBytes.read` is done using `DispatchIO`. And there are many possible deadlocks. Personally I would try to move everything outside of the Swift Concurrency, but as long as it does not leak outside of the module it is fine. But it leaks.

In this section we will assume that deadlocks are not possible, and everything always works perfectly. This will start `sleep 5` 10 times:

```swift
let SLEEP_COUNT = 10
let start = DispatchTime.now()

try await withThrowingTaskGroup(of: Void.self) { group in
  for _ in 0..<SLEEP_COUNT {
    group.addTask {
      let result = try await Subprocess.run(
        executing: .at(FilePath("\(bin)/sleep")),
        arguments: ["5"]
      )

      print(result.terminationStatus)
    }
  }

  try await group.waitForAll()
}

let duration = DispatchTime.now().uptimeNanoseconds - start.uptimeNanoseconds
print("Total: \(duration / 1_000_000_000)s")

```

**Expected:** It finishes in ~5s.

**Result:** Possible deadlock/crash. Even if it didn't it would be more than 5s.

My original idea for this test was to show that running more than 2/4/8/… processes at the same time is not possible because of the blocking calls.

**Use case 1:** We have a program that subscribes to a Message Queue. After getting a message it starts a process depending on the message content. If we receive a lot of messages we may need to start multiple processes at the same time. This would not be possible with `Subprocess.run`. We end up with a back-pressure scenario. I think that OS scheduler would deal with it just fine, but this module overrides the OS scheduler.

**Use case 2:** Olympics are coming and I am interested in running, swimming and rhythmic gymnastic. Unfortunately during the competitions I am at work, so I decided to write an app that will scrape html page and download a video with `ffmpeg`. You can't do this with `Subprocess.run` because it may happen that when you are recording running and swimming it will not be able to start rhythmic gymnastic. We just lost user data.

That was the theory and my initial assumption about this test case. What **actually** happens depends on the `SLEEP_COUNT` value. If you go too far it will deadlock/crash. I tested this on Intel Pentium G4560 (Ubuntu 22.04.4 LTS) which has only 2 cores and `SLEEP_COUNT = 2` would crash.

You know how computer games have those minimum requirements? Like Cyberpunk 2077 requires Core i7-6700 etc. With this module our programs have minimum requirements.

Each process requires 2 parallel tasks. Maybe 3 for `waitpid` and 2 parallel reads from `stdout/stderr`. There is some math formula that would allow us to calculate minimum requirements of a given program, but it is not in the proposal. I think: `3 x number_of_processes` is safe.

Similar thing happens then you start piping data from one process to another. All of them have to be alive at the same time. I have no example because this requires indentation (nesting `Subprocess.run` within another `Subprocess.run` closure), and that makes it too complicated and visually noisy/distracting for this post.

Anyway, now you have this number that you have remember and if you (by mistake) deploy your program to the machine that does not satisfy the minimum requirements then it will deadlock.

Btw. We just deadlocked without filling the pipe buffer. There are some other pipe buffer related deadlocks that I do not show here, as at this point everyone knows the drill, so it would be repetitive. I wanted to show something different.

## Documentation

There is none?

## End

I'm going to stop here, as at this point both of my posts added together are longer than the proposal itself. Check out my [old post](https://forums.swift.org/t/pitch-swift-subprocess/69805/53), if you want more.

I see that the proposal authors are trying to push a lot of things as "user errors", but I'm not sure if I agree. I think we should try to tackle as many of those scenarios as possible, or at least give users some tools to handle them. `Subprocess.run` looks deceptively simple, but it has multiple corner cases. It is declarative-like, users jus say `git status limit 500`, and they do not care about the details. Neither should they! For example we could:

- do not deadlock after the `limit + platform_specific_value`
- close `stdin` if they forget
- do not `SIGPIPE` the child, or at least mention it in documentation
- do not require the users to write parallel read from `stdout/stderr`, give them a method to do it with 1 call

Also, I just can't fail to notice that the authors of the proposal fall for the same pitfalls that they expect the users to avoid. For example blocking read on `stdout` while ignoring `stderr`. Or finite pipe buffer size. Users do not even know what pipes are! They can't account for this. And, from what I see, deadlocks are never mentioned in either the proposal or the documentation.

---

<div class="post-metadata">

**Author:** ![FranzBusch](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franzbusch/32/19447_2.png) [@FranzBusch](https://forums.swift.org/u/FranzBusch)\
**Post date:** [March 5, 2024, 4:26pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/59 "2024-03-05T16:26:54Z")

</div>

> Disclaimer: I have provided feedback to @icharleshu during the proposal and helped shape some of the APIs

I want to provide my review on this as well after having spent a significant time looking at the proposed APIs and trying them out. Overall, I agree that this is an important problem to solve and I think the proposed solution takes the right approach; however, I think there are a few things that need to be reconsidered or require further clarification.

- `run` methods that take explicit arguments vs `run` methods that take a configuration. I personally think that we don't need the `run` methods that take explicit arguments like `executable` and rather all methods should take a `configuration`. The only overloads should be the various input/output/error methods.
- The `run` methods that take a `body` currently require the `body` to be `@escpaing` and `@Sendable`. That implies they run in some child task. This is unexpected to me and makes using the APIs significantly harder in certain contexts like actors. I see no reason that the `body` closure must run in a child task and rather expect in the calling task.
- `Subprocess` is marked as `Sendable`. This seems wrong to me and implies that the `Subprocess` can be shared across multiple isolation contexts. However, the standard output and error **must not** be called from multiple tasks concurrently. IMO, `Subprocess` should not be `Sendable` and might actually want to be `~Escapable`.\*
- Similarly the `StandardInputWriter` currently is `Sendable` and IMO should not be `Sendable` to enforce single producer patterns. It probably should be `~Copyable` and `~Escapable` if possible.\*
- Similarly the `InputMethod`, `RedirectedOutputMethod`, and `CollectedOutputMethod` are marked as `Sendable` but can contain `FileDescriptor` so they **must not** be `Sendable`. I also think they should not be `Hashable` since that might restrict future API design and I personally don't think anybody should equate/hash those types.
- This has already been brought up in this thread but the methods should have explicit documentation about how they handle the file descriptors and cancellation. Without documentation it is hard to reason if the proposed APIs are working or not.
- The `StandardInputWriter` `write` and `finish` method should document that they are suspending if the write to the subprocess would be blocking.
- `TerminationStatus.Code` exposes the underlying platform difference in the type alias. Could we hide this better and just expose an `Int` backing instead?
- We should adopt `some AsyncSequence<UInt8, any Error` for `standardOutput` and `standardError`. This would allow us to define yet another `AsyncBytes` type.

Thanks again for pushing on this @icharleshu!

\*`~Escapable` and `~Copyable` are currently not nicely usable with Swift Concurrency so we might not be able to mark some types here correctly. However, I think we should at least semantically mimic this and call it out in documentation e.g. the `run` method that takes a `body` closure should call out that the `subprocess` **must not** be escaped out of the body closure.

---

<div class="post-metadata">

**Author:** ![ksluder](https://avatars.discourse-cdn.com/v4/letter/k/eada6e/32.png) [@ksluder](https://forums.swift.org/u/ksluder)\
**Post date:** [March 5, 2024, 4:36pm UTC](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337/60 "2024-03-05T16:36:06Z")

</div>

> [@LiarPrincess](#):
>
> ```swift
> try await Subprocess.run(
> executing: .at(FilePath("\(bin)/cat")),
> arguments: ["Pride and Prejudice.txt"]
> )
> 
> print("after")
> 
> ```
> 
> **Expected:** prints "after"
> 
> **Result:**
> 
> - nothing is printed
> - it seems to be hanging
> - `cat` is visible in System Monitor even after a few minutes; deadlock

Is this really the whole example script? If the motivation is to make Swift more competitive with shell scripting, hanging on the most straightforward translation of `cat PrideAndPrejudice.txt ; echo after` seems like a pretty bad footgun.

I’m sure explicitly discarding the output would fix the issue, but the fact it works sometimes without doing so and only fails if the output is above a certain size is a dangerous default.

[Previous page](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337.md?page=2)

[Next page](https://forums.swift.org/t/review-sf-0007-introducing-swift-subprocess/70337.md?page=4)
