[Accepted as version 0.1] SF-0007: Subprocess

[LiarPrincess]

I looked into that repository, and it is identical to the old proposal. It seems that a long time ago Apple decided on the final api, and nothing the community says will change it.

At a certain point the proposal author(s) just stopped responding to people questions/issues (they were completely non existent in the discussion), and started creating new threads with the same proposal over and over again. In the most recent one, the only real discussion was the run function naming. At the same time, the solution has the same unaddressed functional problems as the pitch from 1+ year ago.

Windows

@ahoppen

When launching a process that itself might launch subprocesses, it is possible to terminate the entire process tree on Posix platforms using send(signal: .kill, toProcessGroup: true). However, there is no way to terminate an entire process tree on Windows.

This is nothing new.

You can also reproduce this problem on Linux. (<-- highly recommend read). As @ahoppen mentioned the way to solve this is to send signal to the group leader via negative pid. This is not available in TeardownStep:

// Available on Execution:
extension Execution {
  public func send(
    signal: Signal,
    toProcessGroup shouldSendToProcessGroup: Bool = false
  ) throws
}

// Not available on TeardownStep:
public struct TeardownStep: Sendable, Hashable {
  public static func send(
    signal: Signal,
    allowedDurationToNextStep: Duration
  ) -> Self

  // This one has a typo in the argument name: 'allowe' -> 'allowed'.
  static func gracefulShutDown(
    alloweDurationToNextStep: Duration
  ) -> Self
}

So BOTH Windows and Linux are missing this functionality.

This was discussed in either the pitch or the 1st review thread, so at this point we have to assume that this is a conscious decision by the proposal authors. IDK why we have 2 different api for sending signals (which is a root of this problem).

Race

From the previous thread:

@LiarPrincess

waitid documentation says: If idtype is P_ALL, waitid() shall wait for any children and id is ignored.

How this will work if I already use another Subprocess like solution? What is the migration path?

(…)

Which wait[p]id will trigger for the process started with other Subprocess like code? The Subprocess -> waitid or Custom code->waitpid?

I meant this as a conversation starter, but I received no answers. The answer is: this is a race condition:

1. Precondition: start some Subprocess to init the signal watcher
2. Start a new process without using the Subprocess
3. Child process finishes
4. Subprocess receives a signal and waitpid(-1)
5. Start a waitpid on that process -> it not exist as it was reaped by the Subprocess

It will break any other subprocess like solution. The waitpid in step 5 will return -1 with ECHILD, which breaks any code that expected an exit status.

I mentioned this in the pitch thread more than 1 year ago. I even linked the Python code that solves this problem. But that was completely ignored, without even a comment from the proposal authors.

Compilation errors

As of 06-05-2025 16:00 CEST the package does not even compile (Swift 6.0.3, Linux). It has syntax errors:

internal struct CreatedPipe {
    internal init(
        readFileDescriptor: TrackedFileDescriptor?,
        writeFileDescriptor: TrackedFileDescriptor?,
    ) {
//  ^
// Sources/Subprocess/Configuration.swift:876:5: error: unexpected ',' separator

I understand that this is a work in progress, but as a general rule the code on github should compile. How did the authors even build/run/test the code? Especially when CreatedPipe.init is not gated behind some #if, meaning that it will happen for everyone regardless of the OS.

What if another person (probably an Apple employee) wants to work on this? Or when the main author gets sick?

cat "Pride and prejudice.txt"

// “Pride and Prejudice” is available at project Gutenberg:
// https://www.gutenberg.org/ebooks/42671
let result = try await Subprocess.run(
  .path("/usr/bin/cat"),
  arguments: ["Pride and Prejudice.txt"]
)

This code will not work. I reported this issue (and ~8 other similar issues) more than a year ago:

@LiarPrincess

As it turns out the limit that we specify does not only regulate the amount of bytes that we care about, but also sets an indirect limit on how much the process can write in total:

• stdout <= limit - ok
• stdout <= limit + pipe size - ok, but the output is truncated
• stdout > limit + pipe size - deadlock

It is nice to see all of that being completely ignored. And again, the proposal authors never commented on the issue.

cat "Pride and Prejudice.txt" should either work correctly or be documented.

UTF-8 is completely broken

This is now an official package under the swiftlang, so lets look at the README:

import Subprocess

let result = try await run(
    .path("/bin/dd"),
    arguments: ["if=/path/to/document"]
) { execution in
    var contents = ""
    for try await chunk in execution.standardOutput {
        let string = chunk.withUnsafeBytes { String(decoding: $0, as: UTF8.self) }
        contents += string
        if string == "Done" {
            // Stop execution
            await execution.teardown(
                using: [
                    .gracefulShutDown(
                        allowedDurationToNextStep: .seconds(0.5)
                    )
                ]
            )
            return contents
        }
    }
    return contents
}

This will not compile because of the following error:

PATH/swift-subprocess-main/Sources/Main/main.swift:110:3: error: trailing closure passed to parameter of type 'Environment' that does not accept a closure
) { execution in
  ^~~~~~~~~~~~~~

But this is far from the biggest problem in this code. Look at the following line:

let string = chunk.withUnsafeBytes { String(decoding: $0, as: UTF8.self) }

This is wrong!

To put it simply: we received a sequence of bytes from the child process, and we expect it to be a valid UTF-8. What if it is not a valid UTF-8? This can happen in at least the following cases:

1. Split in child - child process has the whole String, but it splits it by PIPE_BUF bytes to guarantee atomic write
2. Split by OS - pipe is full, so the OS writes only a part of the message and returns the number of bytes written
3. Split by parent - parent process issues a read with a number of bytes

In all of those situations, the bytes representing the message may be split at a random point becoming invalid UTF-8.

That's the theory, in practice:

private let replacementCharacter = UnicodeScalar(65533)!

func unicodeCheck(book: String) async throws {
  // Code from the README:
  let executionResult = try await Subprocess.run(
    .path("/usr/bin/cat"),
    arguments: [book],
    output: .sequence,
    error: .discarded
  ) { execution in
    var contents = ""

    for try await chunk in execution.standardOutput {
      let string = chunk.withUnsafeBytes { String(decoding: $0, as: UTF8.self) }
      contents += string
    }

    return contents
  }

  let result = executionResult.value
  let expected = try String(contentsOf: URL(filePath: book), encoding: .utf8)

  print("===", book, "===")
  print("equal:", result == expected ? "TRUE" : "FALSE")
  print("result.utf8  ", result.utf8.count)
  print("expected.utf8", expected.utf8.count)
  print("� count      ", result.unicodeScalars.count { $0 == replacementCharacter })
}

If we tried to run it on:

• Pride and Prejudice - english text
• Romance of the Three Kingdoms - chinese text

=== Pride and Prejudice.txt ===
equal: TRUE
result.utf8     725472
expected.utf8   725472
utf8.count diff 0
� count         0

=== Romance of the Three Kingdoms.txt ===
equal: FALSE
result.utf8     1864981
expected.utf8   1863697
utf8.count diff 1284
� count         710

To put it simply: the code in README is correct only for ASCII texts, because a single character always occupies 1 byte.

It makes no sense to use String(decoding: $0, as: UTF8.self), as it replaces all of the invalid characters with "�". Just the fact that it promises a valid conversion for every input should raise some flags. String(data: data, encoding: .utf8) is much better as it returns an String?, though the correct code would be more complicated than that.

Tbh. I have no idea what the authors of the original code wanted to archive.

Stdlib

@FranzBusch

All the engagement throughout shows how important this is to many Swift contributors and that it fills a gap in the current ecosystem.

Reason 1: Swift does not have a standard library, especially on the server. Swift only has what Apple needs; anything else is immediately rejected by the core team. For example, if we wanted to unzip something:

1. There is no official package.
2. Community packages do not compile on Ubuntu 24.04.2 LTS - no biggie because they are maintained by unpaid developers.
3. We can write a wrapper around some C library, but pretty soon we will be maintaining 25+ wrappers, which wastes a lot of time.

The only viable alternative is to launch a unzip in a separate process. This is what Swift package manager does. Though the error messages will suck.

Reason 2:
I asked a simple question 7 times and have not received an answer:

Why stdout is limited to 64kb by default? This causes deadlock in "Pride and Prejudice" test case.

I never got an answer. Even the last 3 posts in the previous thread are unanswered questions. Unanswered questions from 1 thread go to the next one, and the whole discussion halts.

To put it simply: there was never a will to work with the community to solve the problems. Too many open ends that were not addressed on the forum or in the next proposal version.

But it does not matter anyway, because Apple will lock this thread. Then in a true spirit of Swift evolution, they will unveil the final api.