Migrating captureOutput to Swift 6

Using Swift
, ,
1

The method captureOutput captures both video and audio frames inside my class CameraViewModel (not a good name). Inside the method there are a lot of references to isolated properties declared in CameraViewModel and that generates a lot of warnings since is captureOutput nonisolated. For example self.videoWriterInput: AVAssetWriterInput and self.audioWriterInput: AVAssetWriterInput gives

Main actor-isolated property 'videoWriterInput' can not be referenced from a nonisolated context

.

My question: I could move all of the code inside captureoutput to be executed on the MainActor, but is this really the most optimal way of doing it? Since it's called from another thread it feels like it would be more efficient to keep it on that thread.

class CameraViewModel: UIViewController, AVCaptureDepthDataOutputDelegate, AVCaptureVideoDataOutputSampleBufferDelegate, AVCaptureAudioDataOutputSampleBufferDelegate {

// ...

  nonisolated func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
          guard CMSampleBufferDataIsReady(sampleBuffer), self.canWrite() else { return }
          
          // Extract the presentation timestamp (PTS) from the sample buffer
          let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
          
          //sessionAtSourceTime is the first buffer we will write to the file
          if self.sessionAtSourceTime == nil {
              //Make sure we start by capturing the videoDataOutput (if we start with the audio the file gets corrupted)
              guard output == self.videoDataOutput else { return }
              //Make sure we don't start recording until the buffer reaches the correct time (buffer is always behind, this will fix the difference in time)
              guard sampleBuffer.presentationTimeStamp >= self.recordFromTime! else { return }
              self.sessionAtSourceTime = sampleBuffer.presentationTimeStamp
              self.videoWriter!.startSession(atSourceTime: sampleBuffer.presentationTimeStamp)
              self.audioWriter!.startSession(atSourceTime: sampleBuffer.presentationTimeStamp)
          }
          
          if output == self.videoDataOutput {
              if self.videoWriterInput!.isReadyForMoreMediaData {
                  self.videoWriterInput!.append(sampleBuffer)
                  self.videoTimestamps.append(
                      Timestamp(
                          frame: videoTimestamps.count,
                          value: timestamp.value,
                          timescale: timestamp.timescale
                      )
                  )
              }
          } else if output == self.audioDataOutput {
              if self.audioWriterInput!.isReadyForMoreMediaData {
                  self.audioWriterInput!.append(sampleBuffer)
                  self.audioTimestamps.append(
                      Timestamp(
                          frame: audioTimestamps.count,
                          value: timestamp.value,
                          timescale: timestamp.timescale
                      )
                  )
              }
          }
      }

// ...

}
2

Does captureOutput need to be nonisolated?

3

Yes in Swift 6. You get this error otherwhise:
Main actor-isolated instance method 'captureOutput(_:didOutput:from:)' cannot be used to satisfy nonisolated protocol requirement

4

That is correct, we want to keep the captureOutput on that background queue. The whole point of specifying a queue in our AVCaptureVideoDataOutput is to keep it off the main thread.

The traditional solution is to:

  • make sure the code is thread-safe (i.e., no accessing of any properties from other queues while being mutated from another); and
  • having done that, mark the class doing this as @unchecked Sendable (i.e., you vouch for the thread safety).

So, in your case, to minimize the impact of this approach, perhaps the AVCaptureVideoDataOutputSampleBufferDelegate could be isolated into its own object. When we do that, we might:

  • Give this AVCaptureVideoDataOutputSampleBufferDelegate a queue property:

    class VideoCaptureService: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate, @unchecked Sendable {
    let queue = DispatchQueue(label: Bundle.main.bundleIdentifier! + ".VideoCaptureService")

    // make sure to only update mutable properties from the above queue
    private var …                                        

    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        // verify our queue assumptions with dispatch preconditions
        dispatchPrecondition(condition: .onQueue(queue))

        …
    }
}

  • When configuring the AVCaptureVideoDataOutput, I would use that queue:

    let videoOutput = AVCaptureVideoDataOutput()
videoOutput.setSampleBufferDelegate(captureService, queue: captureService.queue)

  • As you can see, we would guarantee thread-safety with liberal use dispatchPrecondition to make we never accidentally called any functions from something other than its custom queue:

    // MARK: - Private utility methods

private extension VideoCaptureService {
    func foo() {
        // again, always verify our queue assumptions with dispatch preconditions
        dispatchPrecondition(condition: .onQueue(queue))

        …
    }
}

  • Having ensured the thread-safety of the object, I would mark it as @unchecked Sendable as shown in the first snippet above.

By moving this capture code into its own object, we limit amount the code that we have to manually ensure thread-safety.

For what it’s worth, it strikes me that a more elegant solution would be to make a custom global actor that is isolated to a particular dispatch queue (e.g. with a custom executor) and then add some custom executor checking, but I couldn't get that to work as elegantly on a custom global actor as it does on the main actor.

Really, the right solution is for Apple just fix this API to support Swift concurrency. [Insert editorial here about having concurrency checking rammed down our throats while so much of the framework APIs have not yet been refactored.]

2 Likes
5

This is a great response, thank you!

I agree, their API makes it awkward in combination with swift concurrency right now.