I'm having a problem with Process (aka NSTask) that's been hard to diagnose.
I have a Swift program that runs several different tasks during its lifetime, using the Process API. One of these tasks involves calling the Swift compiler.
Usually the tasks run quickly, finish, and the program moves on. However, when the task involves calling the Swift compiler, the process often never finishes. Tasks that take milliseconds using the terminal can go on for hours until I eventually kill the program.
I tried boiling it down into a simple test case, but simple test cases always finish quickly.
Problems like this are often related to the size of the output pipe buffer. When you run a program via Process, there are two async events in play:
Reading data from the pipe connected to the process’s stdout (A)
Waiting for the process to complete (B)
If you do these synchronously, you can run into problems. Specifically, if you do B before A and the program outputs a lot of data, the program’s output pipe buffer fills up and the program then blocks waiting for space in that buffer to become available. At that point you’ve deadlocked, because you’re waiting for the program to terminate and the program is waiting for you to read some of its output.
To do this properly you have to handle both events asynchronously. This can be tricky. I recently posted my code for this in a thread on DevForums.
I have a package you can use for that if it already does what you want. Otherwise you have my permission to plagiarize from it. The relevant source that finds and uses Swift is here and here, and the handling of processes (synchronously) in general is here.
Thanks, for the answers, I think I'm starting to understand things better now.
My problem is that I'm trying to get the result from the standard output and standard error (separately). I noticed the program deadlocks when I invoke a process that outputs text only to standard error. In particular, the example below prints "Reading output" but never "Output read".
import Foundation
enum Shell {
public struct CommandOutput {
public let standardOutput: String
public let standardError: String
public let status: Int32
}
@discardableResult public static func run(_ arguments: [String]) -> CommandOutput {
let process = Process()
process.launchPath = "/usr/bin/env"
process.arguments = arguments
process.qualityOfService = Thread.current.qualityOfService
let outputPipe = Pipe()
let errorPipe = Pipe()
process.standardOutput = outputPipe
process.standardError = errorPipe
process.launch()
var outputStream = Data()
var errorStream = Data()
var outputIsDone = false
var errorIsDone = false
while true {
if !outputIsDone {
let fileHandle = outputPipe.fileHandleForReading
print("Reading output")
let newData = fileHandle.availableData
print("Output read")
outputStream.append(newData)
if newData.isEmpty {
outputIsDone = true
}
}
if !errorIsDone {
let fileHandle = outputPipe.fileHandleForReading
let newData = fileHandle.availableData
errorStream.append(newData)
if newData.isEmpty {
errorIsDone = true
}
}
if outputIsDone && errorIsDone {
break
}
}
while process.isRunning {}
let outputString = String(data: outputStream, encoding: .utf8) ?? ""
let errorString = String(data: outputStream, encoding: .utf8) ?? ""
let status = process.terminationStatus
return CommandOutput(
standardOutput: outputString,
standardError: errorString,
status: status)
}
}
It seems to me that the availableData property is blocking because the process doesn't print anything to the standard output. This was kind of predictable since the documentation says it blocks, so I tried replacing it with readData(ofLength:) and readDataToEndOfFile(), but they all cause the same problem. Is there a method that doesn't block? Or a way to know if the method is going to block before calling it?
I also thought about using the async methods (readInBackgroundAndNotify() and readToEndOfFileInBackgroundAndNotify()) but the documentation said they use the methods above in their implementation, so I'm not sure it would work.
Ah yes. I originally had the input and error streams separate. For that you have to read from both streams at once. Rewinding the Git blame, this is what it looked like back then. It was changed away from that because it was impossible to reconstitute the interspersion of the two after the fact. It was deemed more useful for the log to be accurate chronologically. But if you only care about one or the other, it work just fine like that.
(This is a very good example of how much effort it takes to corral an API that assumes asynchronicity back into serial behaviour, whereas it is trivial to boot a synchronous API away onto a separate queue. It is wise to alway design your APIs for synchronous use.)
As some constructive criticism, I think the documentation could have been clearer about this part (unless there's a programming guide about this somewhere I didn't see!). Specifically, it would have been better if Xcode's docs had explained that a process with a pipe with a full buffer will block while waiting for space to clear up. (That is what happens, right? I'm still not 100% sure).
I think the documentation could have been clearer about this part
Absolutely. My recommendation is that you file a bug describing the problems you hit, where you looked for a solution, and what would have helped you. That’ll get your feedback to the docs folks here at Apple.
I also think an enhancement request against Process itself is warranted. The problem you hit is ‘obvious’ to anyone who’s familiar with the UNIX underpinnings of Process, Pipe, and FileHandle, but the whole point of those abstractions is to make this stuff easier. A higher-level abstraction would help a lot of folks I think.