AVFoundation Swift 6

Using Swift
, ,
1

Hi. I'm having trouble in migrating AVFoundation implementation to Swift 6, the AVCaptureVideoDataOutputSampleBufferDelegate to be exact.

The following code gives an error:

Call to main actor-isolated instance method 'handleBuffer(sampleBuffer:)' in a synchronous nonisolated context

extension CameraViewController: @preconcurrency AVCaptureVideoDataOutputSampleBufferDelegate {
    func captureOutput(_: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from _: AVCaptureConnection) {
        DispatchQueue.global(qos: .background).async { [weak self] in
            guard let self = self else { return }
            self.handleBuffer(sampleBuffer: sampleBuffer)
        }
    }
}

How can I solve this?

2

In this case, the error states the problem pretty clearly. handleBuffer is main actor isolated (probably inherited by @MainActor from your UIViewController) however DispatchQueue.global.async's closure is nonisolated. In order to switch isolation you have to await at some point (which cannot be done in a DispatchQueue async closure).

I don't think I can provide an exact solution to your problem, because I do not know what handleBuffer does, but you could try making your handleBuffer a nonisolated func.

PS: Weakly capturing self in a DispatchQueue.async is not really needed.

3

It seems converting it to a Task {} fixed the error.

extension CameraViewController: @preconcurrency AVCaptureVideoDataOutputSampleBufferDelegate {
    
    func captureOutput(_: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from _: AVCaptureConnection) {
        Task(priority: .background) {
            self.handleBuffer(sampleBuffer: sampleBuffer)
        }
    }
}

As I understand, since the CMSampleBuffer cannot become Sendable, there is no other way to make it thread-safe, apart from using the @preconcurrency annotation.

4

All depends on which queue you set AVCaptureVideoDataOutputSampleBufferDelegate. Is it main queue (and, therefore, actor)? Then you can use MainActor.assumeIsolated inside captureOutput.

5

It fixed because Task.init by default inherits actor context, and runs it (in this case) on main actor as view controller is isolated to main actor, making call to handleBuffer to be in the same actor in result.

It is not you to make it thread-safe, but just to ensure (on your own) that it remains in the same isolation. If you set your delegate to be on main queue, captureOutput will be called on the main queue as well, making sampleBuffer stay in the same concurrency context. The issue for Swift here is that AVCaptureVideoDataOutputSampleBufferDelegate communicates nothing about its isolation, so Swift cannot be sure that it is safe, you have to opt-out its check (using @preconcurrency) and rely on yourself and some preconditions.

1 Like
6

That is correct, I'm using main queue, thank you.

7

I missed that you have @preconcurrency with my first answer, so you actually don't need assumeIsolated as well, you can just call handleBuffer inside as usual. For more details, see SE-0423.

8

@vns Your solution omits the fact that previously handleBuffer was called in DispatchQueue.global, which is now done on the main queue. If this was there for a reason (for instance heavy work) then you are now doing heavy work on the main queue.

But yeah, CMSampleBuffer being explicitly marked as non-sendable makes AVCaptureVideoDataOutputSampleBufferDelegate hard to use. If you use a non-main-queue Q1 for the sample buffers and need to do heavy work, you will still block Q1. At least it would not be the main thread...

1 Like
9

Since delegate is set to be on main queue, previous call to DispatchQueue.global (apart from using global) wasn’t correct as well, and it called main actor-isolated method anyway. To address heavy operations it would need to set delegate to be not on main queue in the first place, which might involve much larger changes to the code.

It is kinda OK with custom actor executors, as you now can create a separate actor and set it to specific queue, offloading main thread and keeping Swift Concurrency checks in place, except that you’ll need to handle delegate conformance with opt-outs, but I consider this minor to the case. So at least this part is actually pretty OK. There are other APIs in AVFoundation specifically that I’ve found harder to use with Swift 6 checks.

10

I still think the cleanest solution IF you do not explicitly need @MainActor isolated stuff is to stick to nonisolation:

// Set a non-main queue here
output.setSampleBufferDelegate(self, queue: someQueue)

extension CameraViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
    
    nonisolated func handleBuffer(sampleBuffer: CMSampleBuffer) {
        
    }
    
    nonisolated func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        handleBuffer(sampleBuffer: sampleBuffer)
    }
}

No preconcurrency or unsafe workarounds necessary. And if you want to switch to main actor for setting some properties or updating some views, you can still keep doing heavy work in nonisolation and do DispatchQueue.main.async or Task { @MainActor in } after that.

11

I know, but this is not what I meant. Even if you have an actor with a custom executor backed by a dispatch queue, you cannot transfer CMSampleBuffer at all. Whatever work AVCaptureVideoDataOutput does with its sample buffer delegate queue, has to be shared with your post-processing.

But I guess this is how this API was intended to be used anyways.

12

Yes, it wasn’t designed to be able to transferred. In the same way as NSManagedObject isn’t thread-safe (and this was carried to SwiftData). Not every design and object should be sendable and safe to pass around threads, sometimes it makes sense to limit this.