After several months and multiple rounds of review on the Subprocess proposal, I'd like to accept it as version 0.1. @icharleshu has updated it significantly based on all of the great feedback you've given throughout the months. The latest version includes several platform specific API and new design patterns and features, such as top-level global run functions, package traits, and SPAN. It would be very useful that everyone tries them in practice.
@icharleshu will work on moving the implementation into a repository that you'll be able to depend on and test, including documentation and sample code to help you get started. We will collect feedback through GitHub issues. We will revisit this in four months to assess the feedback and determine whether to proceed to version 1.0, possibly with another community review depending on the volume and nature of API changes.
I will update this post when the repository is ready. Please stay tuned. If you have any questions in the meantime, please feel free to reply here, or contact me directly as the review manager by email or DM. Thank you.
I thought we would have a different repo, perhaps in the swiftlang GH org, and a tagged 0.1 version, which the current repository doesn't have at the moment.
That's right. We're working with the core team to set it up, including adding preliminary documentation and setting up CI. The current plan is to have it ready this month. Please stay tuned and thanks for asking!
I looked into that repository, and it is identical to the old proposal. It seems that a long time ago Apple decided on the final api, and nothing the community says will change it.
At a certain point the proposal author(s) just stopped responding to people questions/issues (they were completely non existent in the discussion), and started creating new threads with the same proposal over and over again. In the most recent one, the only real discussion was the run function naming. At the same time, the solution has the same unaddressed functional problems as the pitch from 1+ year ago.
Windows
This is nothing new.
You can also reproduce this problem on Linux. (<-- highly recommend read). As @ahoppen mentioned the way to solve this is to send signal to the group leader via negative pid. This is not available in TeardownStep:
// Available on Execution:
extension Execution {
public func send(
signal: Signal,
toProcessGroup shouldSendToProcessGroup: Bool = false
) throws
}
// Not available on TeardownStep:
public struct TeardownStep: Sendable, Hashable {
public static func send(
signal: Signal,
allowedDurationToNextStep: Duration
) -> Self
// This one has a typo in the argument name: 'allowe' -> 'allowed'.
static func gracefulShutDown(
alloweDurationToNextStep: Duration
) -> Self
}
So BOTH Windows and Linux are missing this functionality.
This was discussed in either the pitch or the 1st review thread, so at this point we have to assume that this is a conscious decision by the proposal authors. IDK why we have 2 different api for sending signals (which is a root of this problem).
Race
From the previous thread:
I meant this as a conversation starter, but I received no answers. The answer is: this is a race condition:
Precondition: start some Subprocess to init the signal watcher
Start a new process without using the Subprocess
Child process finishes
Subprocess receives a signal and waitpid(-1)
Start a waitpid on that process -> it not exist as it was reaped by the Subprocess
It will break any other subprocess like solution. The waitpid in step 5 will return -1 with ECHILD, which breaks any code that expected an exit status.
I mentioned this in the pitch thread more than 1 year ago. I even linked the Python code that solves this problem. But that was completely ignored, without even a comment from the proposal authors.
Compilation errors
As of 06-05-2025 16:00 CEST the package does not even compile (Swift 6.0.3, Linux). It has syntax errors:
I understand that this is a work in progress, but as a general rule the code on github should compile. How did the authors even build/run/test the code? Especially when CreatedPipe.init is not gated behind some #if, meaning that it will happen for everyone regardless of the OS.
What if another person (probably an Apple employee) wants to work on this? Or when the main author gets sick?
cat "Pride and prejudice.txt"
// “Pride and Prejudice” is available at project Gutenberg:
// https://www.gutenberg.org/ebooks/42671
let result = try await Subprocess.run(
.path("/usr/bin/cat"),
arguments: ["Pride and Prejudice.txt"]
)
This code will not work. I reported this issue (and ~8 other similar issues) more than a year ago:
It is nice to see all of that being completely ignored. And again, the proposal authors never commented on the issue.
cat "Pride and Prejudice.txt" should either work correctly or be documented.
UTF-8 is completely broken
This is now an official package under the swiftlang, so lets look at the README:
import Subprocess
let result = try await run(
.path("/bin/dd"),
arguments: ["if=/path/to/document"]
) { execution in
var contents = ""
for try await chunk in execution.standardOutput {
let string = chunk.withUnsafeBytes { String(decoding: $0, as: UTF8.self) }
contents += string
if string == "Done" {
// Stop execution
await execution.teardown(
using: [
.gracefulShutDown(
allowedDurationToNextStep: .seconds(0.5)
)
]
)
return contents
}
}
return contents
}
This will not compile because of the following error:
PATH/swift-subprocess-main/Sources/Main/main.swift:110:3: error: trailing closure passed to parameter of type 'Environment' that does not accept a closure
) { execution in
^~~~~~~~~~~~~~
But this is far from the biggest problem in this code. Look at the following line:
let string = chunk.withUnsafeBytes { String(decoding: $0, as: UTF8.self) }
This is wrong!
To put it simply: we received a sequence of bytes from the child process, and we expect it to be a valid UTF-8. What if it is not a valid UTF-8? This can happen in at least the following cases:
Split in child - child process has the whole String, but it splits it by PIPE_BUF bytes to guarantee atomic write
Split by OS - pipe is full, so the OS writes only a part of the message and returns the number of bytes written
Split by parent - parent process issues a read with a number of bytes
In all of those situations, the bytes representing the message may be split at a random point becoming invalid UTF-8.
That's the theory, in practice:
private let replacementCharacter = UnicodeScalar(65533)!
func unicodeCheck(book: String) async throws {
// Code from the README:
let executionResult = try await Subprocess.run(
.path("/usr/bin/cat"),
arguments: [book],
output: .sequence,
error: .discarded
) { execution in
var contents = ""
for try await chunk in execution.standardOutput {
let string = chunk.withUnsafeBytes { String(decoding: $0, as: UTF8.self) }
contents += string
}
return contents
}
let result = executionResult.value
let expected = try String(contentsOf: URL(filePath: book), encoding: .utf8)
print("===", book, "===")
print("equal:", result == expected ? "TRUE" : "FALSE")
print("result.utf8 ", result.utf8.count)
print("expected.utf8", expected.utf8.count)
print("� count ", result.unicodeScalars.count { $0 == replacementCharacter })
}
=== Pride and Prejudice.txt ===
equal: TRUE
result.utf8 725472
expected.utf8 725472
utf8.count diff 0
� count 0
=== Romance of the Three Kingdoms.txt ===
equal: FALSE
result.utf8 1864981
expected.utf8 1863697
utf8.count diff 1284
� count 710
To put it simply: the code in README is correct only for ASCII texts, because a single character always occupies 1 byte.
It makes no sense to use String(decoding: $0, as: UTF8.self), as it replaces all of the invalid characters with "�". Just the fact that it promises a valid conversion for every input should raise some flags. String(data: data, encoding: .utf8) is much better as it returns an String?, though the correct code would be more complicated than that.
Tbh. I have no idea what the authors of the original code wanted to archive.
Stdlib
Reason 1: Swift does not have a standard library, especially on the server. Swift only has what Apple needs; anything else is immediately rejected by the core team. For example, if we wanted to unzip something:
There is no official package.
Community packages do not compile on Ubuntu 24.04.2 LTS - no biggie because they are maintained by unpaid developers.
We can write a wrapper around some C library, but pretty soon we will be maintaining 25+ wrappers, which wastes a lot of time.
The only viable alternative is to launch a unzip in a separate process. This is what Swift package manager does. Though the error messages will suck.
Reason 2:
I asked a simple question 7 times and have not received an answer:
Why stdout is limited to 64kb by default? This causes deadlock in "Pride and Prejudice" test case.
I never got an answer. Even the last 3 posts in the previous thread are unanswered questions. Unanswered questions from 1 thread go to the next one, and the whole discussion halts.
To put it simply: there was never a will to work with the community to solve the problems. Too many open ends that were not addressed on the forum or in the next proposal version.
But it does not matter anyway, because Apple will lock this thread. Then in a true spirit of Swift evolution, they will unveil the final api.
Thanks for taking the time to write up a detailed post. I think the best way to address many of these concerns is by filing issues on the GitHub repository, so they can be tracked and fixed individually. For example, updating the sample code in the README to handle non-UTF8 data (or just annotating it so that it's clear what the limitation is, while providing a clear example of what is possible) seems like a good starter issue.
The reason this was accepted as 0.1 was so that the package could get some more real-world use. Feedback from that use is critical information to inform a 1.0 API. That is the opposite of simply unveiling a final version of the package. Instead, we get a tag that people can actually try out and the package will be allowed to iterate on its implementation and API with early adopters.
english text
chinese text