Implementing async functions at the "bottom level"

Using Swift
1

This is a question about async/await that has been on my mind ever since it was mentioned, but that I haven't found an answer for: Does the async system automatically know when my code is blocked and return control to another task?

I'm trying to implement a breadth-first filesystem enumerator as an AsyncSequence. When I block on reading a directory, will control automatically be transferred to another task, or should I be using non-blocking reads and calling Task.yield() myself?

Here's my code sample:

    mutating func next() async throws -> String? {
        while true {
            if self.entries.count == 0 && self.directoryQueue.count == 0 {
                return nil
            }
            
            if self.entries.count > 0 {
                return self.entries.removeFirst()
            }
                            
            let currentDirectory = directoryQueue.removeFirst()
            guard let unixDirPtr = opendir(currentDirectory) else {
                throw HXErrors.general("Could not open directory \(currentDirectory)")
            }
            defer {
                closedir(unixDirPtr)
            }
            
            var dirListing = [String]()
            var directoriesInListing = [String]()
            while let entry = readdir(unixDirPtr) {
                let name = withUnsafePointer(to:entry.pointee.d_name) {
                    $0.withMemoryRebound(to:UInt8.self, capacity:Int(entry.pointee.d_namlen)) {
                        String(cString:$0)
                    }
                }
                if name == "." || name == ".." {
                    continue
                }
                dirListing.append(name)
                if entry.pointee.d_type == DT_DIR {
                    directoriesInListing.append(name)
                }
            }
            
            self.entries.append(
                contentsOf:dirListing.lazy
                    .sorted {$0.localizedCompare($1) == .orderedAscending}
                    .map    {currentDirectory + "/" + $0}
            )
            
            let prune = self.prune  // need to assign to a local variable so we don't capture self
            self.directoryQueue.append(
                contentsOf:directoriesInListing.lazy
                    .sorted {$0.localizedCompare($1) == .orderedAscending}
                    .map    {currentDirectory + "/" + $0}
                    .filter {!prune($0)}
            )
        }
    }

Thanks in advance for any enlightenment.

2

Potential suspension points in an async function are always explicit. This is a fundamental design principal of Swift concurrency.

No. "Blocking" is not a well-defined term, so I don't even understand how you would implement this feature even if you wanted to.

Your function is completely synchronous.

3

This is the essence of my question. I've seen many examples of async functions calling other async functions, but no examples of what to do when there are no more async functions to call. Down at "the bottom", as it were.

4

I have no idea what you're talking about.

5

Are you perhaps looking for continuations, such as withUnsafeContinuation?

6

This maybe it. I'll read more about it, but I am not looking to interface with a callback-based API, as you can see.

With async/await, you write an async function that calls await on another async function, which presumably calls await on another async function, and so on. You can't call await on another async function ad infinitum. What does the last function that has nothing to await do?

7

It either does not block, or it throws the work onto a different thread.

Concretely, the bottom layer of the code must be callback-based or it must be essentially non-blocking (that is, if it does block it will only be brief, e.g. making a non-IO system call or using a non-blocking variant). The effect of this is that all of the async/await world is necessary built on top of the continuation primitives: they're the only thing that allow you to escape async/await land.

In your case, as there is no non-blocking variant, your while true loop must kick the work off to a different thread and use a continuation to return the result.

3 Likes
8

Thanks for the guidance. I guess with all the hype about async/await, it's important to remember that it's not the panacea for all problems. Still, as an exercise, I'll try to make this fit into the mold more closely.

Also, I'll take this opportunity to officially complain about the dearth of documentation/examples/doc on this topic.