Safely use AVCaptureSession + Swift 6.2 Concurrency

Using Swift
, , ,
1

Hey Folks,

I’m in the middle of modernizing an older AVFoundation-based QR/Barcode scanner and pulling it into SwiftUI using Swift 6.2 concurrency, and I’ve hit a wall that I’m hoping someone can help me understand.

While researching how to move my AVFoundation pipeline into SwiftUI, I came across this great example that uses UIViewControllerRepresentable, Vision, and AVCaptureVideoDataOutput for real-time scanning:

(Reference: https://www.createwithswift.com/reading-qr-codes-and-barcodes-with-the-vision-framework/)

The example is extremely close to what I need, but it no longer compiles or runs under Swift 6.2 because AVFoundation is much stricter about threading. Specifically, I get this runtime warning/error:

Thread Performance Checker: -[AVCaptureSession startRunning] should be called from background thread.
Calling it on the main thread can lead to UI unresponsiveness.

This is the same issue I ran into while trying to update my company’s actual QR code scanner. The core problem seems to be:

How do you correctly integrate an AVCaptureSession into SwiftUI using Swift 6.2 concurrency without hitting data-race warnings or the startRunning() threading violation?

What I think I understand

After digging through related discussions from this forum.swift, like this one:

https://forums.swift.org/t/avcapturesession-and-concurrency/72681

…it seems clear that the “modern Swift” solution is to:

  • Move all AVFoundation session setup (inputs/outputs/session configuration)

  • And all start/stop operations

…into an actor, ideally with a custom executor, so that AVFoundation work runs on a dedicated serial queue but is still Swift-concurrency-safe.

The problem is:
I don’t know how to correctly implement the actor + custom executor pattern in a way that plays nicely with SwiftUI and UIViewControllerRepresentable.

What I’m looking for

If anyone can provide:

  • A minimal example actor wrapping AVCaptureSession (with custom executor)

  • Guidance on how to call it from SwiftUI without tripping thread-checker warnings

  • Or a pattern you use in production for AVFoundation + SwiftUI + Swift 6.2 concurrency
    (especially for camera capture + Vision processing)

…I’d really appreciate it.

Sample Code

Here is the makeUIViewController I’m trying to modernize (shortened for readability see above URL for full code):

func makeUIViewController(context: Context) -> UIViewController {
        let viewController = UIViewController()
        
        guard let videoCaptureDevice = AVCaptureDevice.default(for: .video),
              let videoInput = try? AVCaptureDeviceInput(device: videoCaptureDevice),
              captureSession.canAddInput(videoInput) else { return viewController }
        
        captureSession.addInput(videoInput)
        
        let videoOutput = AVCaptureVideoDataOutput()
        
        if captureSession.canAddOutput(videoOutput) {
            videoOutput.setSampleBufferDelegate(context.coordinator, queue: DispatchQueue(label: "videoQueue"))
            captureSession.addOutput(videoOutput)
        }
        
        let previewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
        previewLayer.frame = viewController.view.bounds
        previewLayer.videoGravity = .resizeAspectFill
        viewController.view.layer.addSublayer(previewLayer)
        
        captureSession.startRunning()   // <- triggers thread checker warning in Swift 6.2
        
        return viewController
    }

There’s also a Coordinator sending Vision results back to SwiftUI, and VideoDataOutput delegating to a background queue.

Everything works perfectly in older Swift versions — but Swift 6.2’s stricter concurrency model makes this no longer viable.

Any guidance, examples, or explanations of how to properly architect this under Swift 6.2 would be hugely helpful. Thanks in advance! :folded_hands:

1 Like
2

I haven’t had a chance to dig into this new API, but I think runDeferredStartWhenNeeded() might help here.

3

Ok so I found figured out how to make this compile and run. @calicoding runDeferedStartWhenNeeded()was a good jumping off point but the actual issue was just getting the threading/concurrency correct in the delegate methods and the AVCaptureSessionsetup. It seems super super hacky. Here’s the full solution I ended up on:

import SwiftUI
import Vision
@preconcurrency import AVFoundation

struct ContentView: View {
    @State private var scannedString: String = "Scan a QR code or barcode"

    var body: some View {
        ZStack(alignment: .bottom) {
            ScannerView(scannedString: $scannedString)
                .edgesIgnoringSafeArea(.all)

            Text(scannedString)
                .padding()
                .background(.ultraThinMaterial)
                .clipShape(RoundedRectangle(cornerRadius: 10))
                .padding()
        }
    }
}

struct ScannerView: UIViewControllerRepresentable {
    @Binding var scannedString: String

    // Serial queue for all AVCaptureSession operations
    private let sessionQueue = DispatchQueue(label: "sessionQueue")
    // Separate queue for Vision barcode processing
    private let videoQueue = DispatchQueue(label: "videoQueue")

    let captureSession = AVCaptureSession()

    func makeUIViewController(context: Context) -> UIViewController {
        let viewController = UIViewController()

        sessionQueue.async { [captureSession] in
            guard let videoCaptureDevice = AVCaptureDevice.default(for: .video),
                  let videoInput = try? AVCaptureDeviceInput(device: videoCaptureDevice),
                  captureSession.canAddInput(videoInput) else { return }

            captureSession.addInput(videoInput)

            let videoOutput = AVCaptureVideoDataOutput()
            if captureSession.canAddOutput(videoOutput) {
                Task { @MainActor in

                    videoOutput.setSampleBufferDelegate(context.coordinator, queue: self.videoQueue)
                }

                captureSession.addOutput(videoOutput)
            }

            captureSession.startRunning()
        }

        let previewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
        previewLayer.frame = viewController.view.bounds
        previewLayer.videoGravity = .resizeAspectFill
        viewController.view.layer.addSublayer(previewLayer)

        return viewController
    }

    func updateUIViewController(_ uiViewController: UIViewController, context: Context) {}
    func makeCoordinator() -> Coordinator {
        Coordinator(self)
    }

    class Coordinator: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {
        var parent: ScannerView

        init(_ parent: ScannerView) {
            self.parent = parent
        }

        nonisolated func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
            // Detect barcode here
        }
    }
}

I feel like there’s gotta be a better way to do all of this especially the setup wrapped inside sessionQueue.async. The entire time while figuring this out I was thinking there needs to be a way to put the initialization and starting of the capturing and even just other basic camera operations into an actor. I feel like @vns was onto something with his reply here.

@calicoding Happy this runs but thoughts on how I might be able to put this into an actor/clean this up a bit?

4

Can you repro from the AVCam sample?

1 Like
5

[Resolved] Safely use AVCaptureSession + Swift 6.2 Concurrency

Ok so I was finally able to get the QR/barcode example working under Swift 6.2, strict concurrency checking, and SwiftUI. It just about killed me but I was able to get it to work with little compromise. I didn't need to rely on DispatchQueue, @unchecked Sendable, or weakening isolation.

Key takeaways:

  • Apple’s AVCam sample still (as of January 6th) isn’t using Swift 6.2 or strict concurrency.
  • @Published is not concurrency-safe when exposed from an actor.
  • I used AsyncStream, consumed via for await in the ViewModel.
  • The AVCapture delegate works best as an internal private class fully encapsulated inside the actor.

I was originally going to post a dramatic retelling of how I got here to the forums, but it turned into a novel, the tone was less than professional and didn't really fit with general forum vibe, so I put it up on my substack. Feel free to check it out if you have the time.

Here's the final solution I came to:

import SwiftUI
import Vision
@preconcurrency import AVFoundation

struct ContentView: View {
    @State private var scannerViewModel = BarcodeScannerViewModel()
    
    var body: some View {
        ZStack(alignment: .bottom) {
            BarcodeScannerView(session: scannerViewModel.session)
                .ignoresSafeArea()
                .task {
                    await scannerViewModel.start()
                }
                
            Text(scannerViewModel.scannedCode ?? "Scan a code")
                .padding()
                .background(.ultraThinMaterial)
                .clipShape(RoundedRectangle(cornerRadius: 10))
                .padding()
        }
    }
}

struct BarcodeScannerView: UIViewRepresentable {
    let session: AVCaptureSession
    
    func makeUIView(context: Context) -> PreviewView {
        let view = PreviewView()
        view.previewLayer.session = session
        view.previewLayer.videoGravity = .resizeAspectFill
        return view
    }
    
    func updateUIView(_ uiView: PreviewView, context: Context) { }
}

class PreviewView: UIView {
    override class var layerClass: AnyClass {
        AVCaptureVideoPreviewLayer.self
    }
    
    var previewLayer: AVCaptureVideoPreviewLayer {
        layer as! AVCaptureVideoPreviewLayer
    }
}

@MainActor
@Observable
class BarcodeScannerViewModel {
    private(set) var isRunning = false
    private(set) var scannedCode: String?
    private(set) var error: Error?
    
    private let captureService = BarcodeScannerCaptureService()
    var session: AVCaptureSession { captureService.captureSession }
    
    func start() async {
        do {
            try await captureService.start()
            isRunning = true
            scannedStringListener()
        } catch {
            self.error = error
        }
    }
    
    private func scannedStringListener() {
        Task {
            guard let scannedStringStream = await captureService.scannedStringStream else { return }
            for await codeString in scannedStringStream {
                scannedCode = codeString
            }
        }
    }
}

actor BarcodeScannerCaptureService {
    nonisolated let captureSession = AVCaptureSession()
    private let outputSampleDelegate = OutputSampleDelegate()
    var scannedStringStream: AsyncStream<String>?
    
    private let videoQueue = DispatchQueue(label: "videoQueue")
    private let sessionQueue = DispatchSerialQueue(label: "sessionQueue")
    
    nonisolated var unownedExecutor: UnownedSerialExecutor {
        sessionQueue.asUnownedSerialExecutor()
    }
    
    func start() async throws {
        guard let videoCaptureDevice = AVCaptureDevice.default(for: .video),
              let videoInput = try? AVCaptureDeviceInput(device: videoCaptureDevice),
              captureSession.canAddInput(videoInput) else { return }
        
        scannedStringStream = outputSampleDelegate.scannedStringStream
        captureSession.addInput(videoInput)
        
        let videoOutput = AVCaptureVideoDataOutput()
        if captureSession.canAddOutput(videoOutput) {
            videoOutput.setSampleBufferDelegate(outputSampleDelegate, queue: self.videoQueue)
            captureSession.addOutput(videoOutput)
        }
        
        captureSession.startRunning()
    }
    
    private class OutputSampleDelegate: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {
        let scannedStringStream: AsyncStream<String>
        private let continuation: AsyncStream<String>.Continuation
        
        override init() {
            let (stream, continuation) = AsyncStream.makeStream(of: String.self)
            self.scannedStringStream = stream
            self.continuation = continuation
        }
        
        func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
            guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
                return
            }
            
            self.detectBarcode(in: pixelBuffer)
        }
        
        private func detectBarcode(in pixelBuffer: CVPixelBuffer) {
            let request = VNDetectBarcodesRequest()
            
            /*
             
             Use the following line to restrict the type of symbologies (scannable code types)
             that the scanner will detect
             
             request.symbologies = [.qr, .ean13, .code128]
             
             Scannable symbologies are as follows
             
             1D Barcodes:
             codabar, code128, code39, code39CheckSum, code39FullASCII, code39FullASCIIChecksum, code93, code93i, i2of5, i2of5Checksum, msiPlessey, upce
             
             2D Barcodes:
             aztec, dataMatrix, microPDF417, microQR, pdf417, qr
             
             Product Codes:
             ean13, ean8, gs1DataBar, gs1DataBarExpanded, gs1DataBarLimited, itf14
             */
            
            let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up, options: [:])
            
            do {
                try handler.perform([request])
                if let results = request.results, let payload = results.first?.payloadStringValue {
                    AudioServicesPlaySystemSound(SystemSoundID(kSystemSoundID_Vibrate))
                    continuation.yield(payload)
                }
            } catch {
                print("Barcode detection failed: \(error)")
            }
        }
    }
}

/*
 
 Uncomment this section to display SwiftUI preview
 
 #Preview {
     ContentView()
 }
 */

If you're lazy like me and don't want to take the time to copy paste it into a project I pushed it to a public Git. Feel free to use how you like.

Hopefully this saves someone else a few weeks of pain. Thanks to everyone who replied — especially @vanvoorden for pointing me to the AVCam.

2 Likes
6

The way I did it:

  1. An actor model where all AV thingies live. It maintains the state. State is sendable.
  2. KV observation to keep that state up to date with AV.
  3. a MainActor ObservableObject view model with published properties. One of the published properties is state.
  4. The view connected to the view model.
  5. A simple synchronisation machinery to keep the state in the actor model and in the view model in sync. For example in the view model I have something like this:
@Published var state: State {
    didSet {
        Task { await actorModel.updateState(state) }
    }
}

and to sync the state the other way around from the actor to the view model it's similar:

    Task { await viewModel.updateState(state) }

Turned out to be quite clean.

7

This is pretty clean. I'm not a huge fan of Combine, but it's been growing on my lately, especially with this deep dive and the @MainActor and @Observable you can add to your entire ViewModel class. @tera will keep this pattern in mind for future use! Great callout.

8

AVCaptureVideoPreviewLayer might still be an issue: one can keep it in the actor as AV thingy but then UI cannot access it. Or keep it on main actor, but then setting the session property needs the (unsendable) AVCaptureSession to cross actor isolation.

Using @preconcurrency and nonisolated makes the compiler happy, but are not making it actually safe. In the end the AVCaptureSession instance is exposed on the main thread.