AsyncStream for CMSampleBuffer in Camera Delegate: Ensuring Smooth Performance Across Devices

Using Swift
1

Hi everyone! I’m implementing a didOutputSampleBuffer camera delegate and ran into an issue regarding smooth camera preview performance when using AsyncStream. I need to send CMSampleBuffer to my models, analyze it, and get back a result. Here’s my setup::

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
  guard CMSampleBufferIsValid(sampleBuffer) else { return }
  let awaiter = DispatchSemaphore(value: 0)
  addToBufferStream?(SampleBuffer(sampleBuffer, awaiter))
  awaiter.wait()
}

where I have:

    private var continuation: AsyncStream<SampleBuffer>.Continuation?
    private var addToBufferStream: ((SampleBuffer) -> Void)?

I’ve wrapped CMSampleBuffer in a SampleBuffer class:

/// Wrapped non Sendable `CMSampleBuffer`
public final class SampleBuffer: @unchecked Sendable {
    public let buffer: CMSampleBuffer
    private let awaiter: DispatchSemaphore

    init(_ buffer: CMSampleBuffer, _ awaiter: DispatchSemaphore) {
        self.buffer = buffer
        self.awaiter = awaiter
    }

    deinit {
        awaiter.signal()
    }
}

Why do I use semaphore here? Without it, AVFoundation may reuse the same CMSampleBuffer before my analyzer finishes processing it, causing performance issues, especially with camera preview blocking. Using a semaphore ensures smooth preview performance.
It seems, when adding SampleBuffer to AsyncStream, the async sequence here is not behaving like async. I need a blocking async context to ensure camera preview runs smoothly (AVFoundation detects if this delegate takes too long) and automatically allocates separate buffers for purposes of preview. However, if we just launch the task with captured CMSampleBuffer, AVFoundation will see immediate exit of the delegate function and will try using the same buffer for new frame, but since it's locked due to being in use, this causes camera preview to block.

I’ve tested this on various iPhones from the iPhone 8 onwards. Interestingly, the issue seems to occur only on Pro Max models.

Is there a more idiomatic way to handle this? Any suggestions or feedback would be greatly appreciated!

1 Like
2

Blocking meaning backpressure? I believe there was talk from @FranzBusch about something along those lines. You might also want to look at AsyncChannel from AsyncAlgorithms.

3

Thank you for the links. But I see that send is also async here, so producer needs to be inside the task. So it’s still a non blocking thing, I still need Semaphore here