"Loops" with Futures
Frequently, the SwiftNIO team get asked how you can loop over a bunch of asynchronous operations, doing one after the other. In synchronous programming that's basically a for thing in things { do(thing) }. We've got a "rule of three" so when we get asked the same question three times, we need to do something to document it and today was at least the third time I personally got asked the "looping over futures" question.
Some may also still remember from their CS courses that as a theoretical concept, loops and recursion are equally expressive. That means that you can express any loop with recursion too but in imperative languages there is usually no point in turning a loop into recursion. In fact, sometimes you would do the opposite for performance reasons.
For future-based asynchronous programming however, loops aren't very useful to do things one after the other because for that to work we would need to block and that's verboten. Hence, for future-based asynchronous programs, turning loops into recursion is actually a very useful skill and is necessary whenever you want to do one asynchronous operation after the other.
Synchronous with a loop
To explain this a bit better, let's start with a (working, the code uses AsyncHTTPClient) synchronous version that uses a straightforward loop:
let urls: [String] = (0..<10).map { _ in "http://httpbin.org/get" }
let client = HTTPClient(eventLoopGroupProvider: .createNew)
defer {
try! client.syncShutdown()
}
var remaining = urls[...]
while let first = remaining.popFirst {
do {
try client.get(url: first).wait()
} catch {
throw error
}
}
Synchronous with recursion
The above program can be transformed (still synchronous) to use recursion instead of a loop:
let urls: [String] = (0..<10).map { _ in "http://httpbin.org/get" }
let client = HTTPClient(eventLoopGroupProvider: .createNew)
defer {
try! client.syncShutdown()
}
func doRemainingRequests(_ remaining: ArraySlice<String>, eventLoop: EventLoop) throws {
var remaining = remaining
if let first = remaining.popFirst() {
do {
try client.get(url: first).wait()
// Recurse to do the next request.
doRemainingRequests(remaining, eventLoop: eventLoop)
} catch {
throw error
}
}
}
try! doRemainingRequests(urls[...], eventLoop: client.eventLoopGroup.next())
Asynchronous
and once you've got the synchronous version that's rewritten using recursion, it's pretty straightforward to make it asynchronous:
let urls: [String] = (0..<10).map { _ in "http://httpbin.org/get" }
let client = HTTPClient(eventLoopGroupProvider: .createNew)
defer {
try! client.syncShutdown()
}
func doRemainingRequests(_ remaining: ArraySlice<String>, eventLoop: EventLoop) -> EventLoopFuture<Void> {
var remaining = remaining
if let first = remaining.popFirst() {
return client.get(url: first).flatMap { [remaining] _ in
// Recurse to do the next request.
doRemainingRequests(remaining, eventLoop: eventLoop)
}
} else {
return eventLoop.makeSucceededFuture(())
}
}
// This kicks it off and in this case waits for the result. In an asynchronous context
// you would return the resulting future instead.
try! doRemainingRequests(urls[...], eventLoop: client.eventLoopGroup.next()).wait()
Asynchronous (better)
The above program works just fine but it keeps all the futures in memory until the last request has terminated, only then will they all deinit. If you also want to get rid of that temporary memory over-use, you'd use
let urls: [String] = (0..<10).map { _ in "http://httpbin.org/get" }
let client = HTTPClient(eventLoopGroupProvider: .createNew)
defer {
try! client.syncShutdown()
}
func doRemainingRequests(_ remaining: ArraySlice<String>, overallResult: EventLoopPromise<Void>, eventLoop: EventLoop) {
var remaining = remaining
if let first = remaining.popFirst() {
client.get(url: first).map { [remaining] _ in
doRemainingRequests(remaining, overallResult: overallResult, eventLoop: eventLoop)
}.whenFailure { error in
overallResult.fail(error)
}
} else {
return overallResult.succeed(())
}
}
let promise = client.eventLoopGroup.next().makePromise(of: Void.self)
// Kick off the process
doRemainingRequests(urls[...], overallResult: promise, eventLoop: client.eventLoopGroup.next())
// Again, in this case this waits synchronously for everything to terminate, in
// an asynchronous context you would return `promise.futureResult` which is fulfilled when all the
// 10 requests are done (or there was an error).
try! promise.futureResult.wait()
It's probably best to only consider the doRemainingRequests function in the middle of the code. In these examples, I did add the (synchronous) setup & teardown so it's easier for folks to try out the code by just pasting this in a SwiftPM project or an XCTest. The only dependencies you need are AsyncHTTPClient and SwiftNIO.
I hope this small code transformation journey from synchronous loop version via synchronous recursive version to an asynchronous recursive version makes sense and helps you next time you want to do one asynchronous task after another "in a loop".
If anything remains unclear or you have suggestions on how to improve the code, please feel free to comment right here 