Calling the Swift compiler using Process/NSTask never finishes

Hi all,

I'm having a problem with Process (aka NSTask) that's been hard to diagnose.

I have a Swift program that runs several different tasks during its lifetime, using the Process API. One of these tasks involves calling the Swift compiler.

Usually the tasks run quickly, finish, and the program moves on. However, when the task involves calling the Swift compiler, the process often never finishes. Tasks that take milliseconds using the terminal can go on for hours until I eventually kill the program.

I tried boiling it down into a simple test case, but simple test cases always finish quickly.

Here's the code that calls the Process API, in case anyone wants to take a look: https://github.com/vinivendra/Gryphon/blob/master/Sources/GryphonLib/Shell.swift

Anyone have any ideas about what might be going on?

Problems like this are often related to the size of the output pipe buffer. When you run a program via Process, there are two async events in play:

  • Reading data from the pipe connected to the process’s stdout (A)

  • Waiting for the process to complete (B)

If you do these synchronously, you can run into problems. Specifically, if you do B before A and the program outputs a lot of data, the program’s output pipe buffer fills up and the program then blocks waiting for space in that buffer to become available. At that point you’ve deadlocked, because you’re waiting for the program to terminate and the program is waiting for you to read some of its output.

To do this properly you have to handle both events asynchronously. This can be tricky. I recently posted my code for this in a thread on DevForums.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

2 Likes

I have a package you can use for that if it already does what you want. Otherwise you have my permission to plagiarize from it. The relevant source that finds and uses Swift is here and here, and the handling of processes (synchronously) in general is here.

Thanks, for the answers, I think I'm starting to understand things better now.

My problem is that I'm trying to get the result from the standard output and standard error (separately). I noticed the program deadlocks when I invoke a process that outputs text only to standard error. In particular, the example below prints "Reading output" but never "Output read".

import Foundation

enum Shell {
	public struct CommandOutput {
		public let standardOutput: String
		public let standardError: String
		public let status: Int32
	}

	@discardableResult public static func run(_ arguments: [String]) -> CommandOutput {
		let process = Process()
		process.launchPath = "/usr/bin/env"
		process.arguments = arguments
		process.qualityOfService = Thread.current.qualityOfService

		let outputPipe = Pipe()
		let errorPipe = Pipe()
		process.standardOutput = outputPipe
		process.standardError = errorPipe

		process.launch()

		var outputStream = Data()
		var errorStream = Data()

		var outputIsDone = false
		var errorIsDone = false
		while true {
			if !outputIsDone {
				let fileHandle = outputPipe.fileHandleForReading
				print("Reading output")
				let newData = fileHandle.availableData
				print("Output read")

				outputStream.append(newData)
				if newData.isEmpty {
					outputIsDone = true
				}
			}


			if !errorIsDone {
				let fileHandle = outputPipe.fileHandleForReading
				let newData = fileHandle.availableData
				errorStream.append(newData)
				if newData.isEmpty {
					errorIsDone = true
				}
			}

			if outputIsDone && errorIsDone {
				break
			}
		}

		while process.isRunning {}

		let outputString = String(data: outputStream, encoding: .utf8) ?? ""
		let errorString = String(data: outputStream, encoding: .utf8) ?? ""
		let status = process.terminationStatus

		return CommandOutput(
			standardOutput: outputString,
			standardError: errorString,
			status: status)
	}
}

It seems to me that the availableData property is blocking because the process doesn't print anything to the standard output. This was kind of predictable since the documentation says it blocks, so I tried replacing it with readData(ofLength:) and readDataToEndOfFile(), but they all cause the same problem. Is there a method that doesn't block? Or a way to know if the method is going to block before calling it?

I also thought about using the async methods (readInBackgroundAndNotify() and readToEndOfFileInBackgroundAndNotify()) but the documentation said they use the methods above in their implementation, so I'm not sure it would work.

Ah yes. I originally had the input and error streams separate. For that you have to read from both streams at once. Rewinding the Git blame, this is what it looked like back then. It was changed away from that because it was impossible to reconstitute the interspersion of the two after the fact. It was deemed more useful for the log to be accurate chronologically. But if you only care about one or the other, it work just fine like that.

(This is a very good example of how much effort it takes to corral an API that assumes asynchronicity back into serial behaviour, whereas it is trivial to boot a synchronous API away onto a separate queue. It is wise to alway design your APIs for synchronous use.)

Yes! This is exactly what I needed. I adapted your code into my APIs and it works perfectly, thank you very much.

I added a comment with a link to your original file for credit too, but let me know if you prefer it differently.

Here's a link to my final solution, if anyone happens upon this thread and wants to check it out.

As some constructive criticism, I think the documentation could have been clearer about this part (unless there's a programming guide about this somewhere I didn't see!). Specifically, it would have been better if Xcode's docs had explained that a process with a pipe with a full buffer will block while waiting for space to clear up. (That is what happens, right? I'm still not 100% sure).

I think the documentation could have been clearer about this part

Absolutely. My recommendation is that you file a bug describing the problems you hit, where you looked for a solution, and what would have helped you. That’ll get your feedback to the docs folks here at Apple.

I also think an enhancement request against Process itself is warranted. The problem you hit is ‘obvious’ to anyone who’s familiar with the UNIX underpinnings of Process, Pipe, and FileHandle, but the whole point of those abstractions is to make this stuff easier. A higher-level abstraction would help a lot of folks I think.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

1 Like

Ok, I filed the bug describing my issues with the docs and asking for an enhancement for the Process API.

Thanks for everything, both of you.

Terms of Service

Privacy Policy

Cookie Policy