# Realtime threads with Swift

**URL:** <https://forums.swift.org/t/realtime-threads-with-swift/40562>\
**Category:** Discussion\
**Created:** [September 23, 2020, 10:20pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562 "2020-09-23T22:20:35Z")\
**Posts on this page:** 20\
**Page:** 5

<div class="post-metadata">

**Author:** ![young](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/young/32/7878_2.png) [@young](https://forums.swift.org/u/young)\
**Post date:** [September 12, 2022, 4:30am UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/82 "2022-09-12T04:30:30Z")

</div>

May see some sample code of how you use unsafe memory allocation?

---

<div class="post-metadata">

**Author:** ![audulus](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/audulus/32/40637_2.png) [@audulus](https://forums.swift.org/u/audulus)\
**Post date:** [December 31, 2022, 4:21am UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/83 "2022-12-31T04:21:53Z")

</div>

Just bumping this thread because I happened upon this:

```swift
/*!	@property	renderContextObserver
	@brief Block called by the OS when the rendering context changes.
	
	Audio Units which create auxiliary realtime rendering threads should implement this property to
	return a block which will be called by the OS when the render context changes. Audio Unit hosts
	must not attempt to interact with the AudioUnit through this block; it is for the exclusive use
	of the OS. See <AudioToolbox/AudioWorkInterval.h> for more information.
*/
@property (nonatomic, readonly) AURenderContextObserver renderContextObserver
	API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0))
	__SWIFT_UNAVAILABLE_MSG("Swift is not supported for use with audio realtime threads");

```

Here we are 2+ years later. Still can't do realtime audio in swift. Sad.

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [December 31, 2022, 10:32am UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/85 "2022-12-31T10:32:16Z")

</div>

> [@TeamPuzel](#):
>
> I'm curious, and I asked about this before, what's stopping Swift from just having a normal array? Is it because having 2 types of arrays would make the language more complicated, even though there's multiple (and arguably less useful) array types already? I feel like Swift is way past the point of being a "simple" language anyway.

We do also want to add _fixed-size_ arrays at some point (and there have been [discussions](https://forums.swift.org/t/approaches-for-fixed-size-arrays/58894) about the best approach). Which one is "normal" depends on your perspective, but AFAIK there has been _no rejection_ of fixed-size arrays as a concept, even if that requires a new type.

> [@TeamPuzel](#):
>
> Something else possibly related, aren't for loops also less optimal than C in this area?  
> C style loops seem to have been removed a long time ago for some reason, and  
> if I have to write `for i in 0...10` doesn't it need to allocate the range just to do the loop? That sounds a bit inefficient, unless there's some kind of optimisation done to these.

`Range` is a struct, so its values are stored directly. Most of the time, it will just get inlined and vanish. It is safe to use in realtime code. I would expect integer literals, as you've shown here, to compile to exactly the same code as C. For non-literals, I would expect the differences to be limited to range correctness checks (checking that `lower <= upper` when you write `for x in lower..<upper`) and integer overflow checks.

---

<div class="post-metadata">

**Author:** ![ksluder](https://avatars.discourse-cdn.com/v4/letter/k/eada6e/32.png) [@ksluder](https://forums.swift.org/u/ksluder)\
**Post date:** [December 31, 2022, 3:29pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/86 "2022-12-31T15:29:45Z")

</div>

> [@TeamPuzel](#):
>
> And what about memory allocation in the first place? as far as I can tell Swift provides no way of handling out of memory errors.

Neither does C on systems that overcommit memory like Linux. Even under extreme memory contention, `malloc()` will happily return a non-null pointer and then the VM system will kill your process when you try to wire those pages by writing to them.

Practically speaking, your opportunities to handle memory conditions are restricted to direct interactions with the VM system. Swift could definitely make it easier to allocate instances out of manually managed memory, but a system which was completely safe in the presence of VM would require handling OOM on every memory access.

---

<div class="post-metadata">

**Author:** ![tcldr](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tcldr/32/24405_2.png) [@tcldr](https://forums.swift.org/u/tcldr)\
**Post date:** [December 31, 2022, 5:25pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/87 "2022-12-31T17:25:59Z")

</div>

How realistic is it to use Swift for simple real-time audio processing today?

I see that AVFoundation's `AVAudioSourceNode` and `AVAudioSinkNode` released in 2019 both mention realtime audio processing applications with a caveat about avoiding mutexes and libdispatch. But no mention now of avoiding Swift explicitly.

In fact, the video that mentions avoiding Swift seems to have been removed from the archives (Session 501 WWDC 2017) and is quite hard to find now.

With Swift Atomics now available, would it be reasonable to use `ManagedBuffer` to preallocate a buffer upfront to implement various producer-consumer queues suitable for realtime applications? If not statically, is there a simple way to instrument our realtime sections to ensure the compiler isn't adding something that may obstruct progress of our real time threads?

EDIT: And also, how much of an impact will the realisation of Swift's ownership manifesto have on using Swift for realtime applications?

---

<div class="post-metadata">

**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\
**Post date:** [December 31, 2022, 6:40pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/88 "2022-12-31T18:40:18Z")

</div>

> [@tcldr](#):
>
> How realistic is it to use Swift for simple real-time audio processing today?
> 
> I see that AVFoundation's `AVAudioSourceNode` and `AVAudioSinkNode` released in 2019 both mention realtime audio processing applications with a caveat about avoiding mutexes and libdispatch. But no mention now of avoiding Swift explicitly.
> 
> In fact, the video that mentions avoiding Swift seems to have been removed from the archives (Session 501 WWDC 2017) and is quite hard to find now.

Don't know the current official word on that, last time I paid attention it was: "don't do this" in 2015 and "yes you can" in 2016 (see the proof-links in the [message above](https://forums.swift.org/t/realtime-threads-with-swift/40562/57)).

I think - practically you can. Just remember some audio glitches are unavoidable even if you do it in C (e.g. the data or code you are touching happen to be paged out and need to be loaded from disk - that's I/O → which can take unbound time (or just enough time to miss the deadline). Good idea is to count the number of glitches per a time interval to see how good / bad it works in practice.

---

<div class="post-metadata">

**Author:** ![audulus](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/audulus/32/40637_2.png) [@audulus](https://forums.swift.org/u/audulus)\
**Post date:** [December 31, 2022, 6:52pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/89 "2022-12-31T18:52:33Z")

</div>

Yeah, I suppose if you program like you are programming in C, then it should be fine. After all, why would the compiler need to generate runtime calls?

So you can't use swift arrays (gotta preallocate anything and use UnsafePointer), no escaping closures, etc. It's fairly restrictive but I can get quite a bit done.

Still, apple blocked us from trying with that `__SWIFT_UNAVAILABLE_MSG`. I've gotten around it using `class_replaceMethod`: [AudioKit/EngineAudioUnit.swift at 3e5003bde0ea9d36e48e8512778cbfbb9d3ae243 · AudioKit/AudioKit · GitHub](https://github.com/AudioKit/AudioKit/blob/3e5003bde0ea9d36e48e8512778cbfbb9d3ae243/Sources/AudioKit/Internals/Engine/EngineAudioUnit.swift#L117)

I'm sick of waiting around for this. We're doing AudioKit v6's DSP with Swift.

---

<div class="post-metadata">

**Author:** ![audulus](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/audulus/32/40637_2.png) [@audulus](https://forums.swift.org/u/audulus)\
**Post date:** [December 31, 2022, 6:56pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/90 "2022-12-31T18:56:54Z")

</div>

> [@tcldr](#):
>
> With Swift Atomics now available, would it be reasonable to use `ManagedBuffer` to preallocate a buffer upfront to implement various producer-consumer queues suitable for realtime applications?

Yeah doable. We wanted to use Swift Atomics but they use a C shim and we wanted compatibility with the Swift Playgrounds app since AudioKit is beginner-friendly ([Swift Atomics can't be used in Swift Playgrounds.app · Issue #62 · apple/swift-atomics · GitHub](https://github.com/apple/swift-atomics/issues/62))

Instead I used the (deprecated) OSAtomic APIs: [https://github.com/AudioKit/AudioKit/blob/v6/Sources/AudioKit/Internals/Engine/RingBuffer.swift](https://github.com/AudioKit/AudioKit/blob/v6/Sources/AudioKit/Internals/Engine/RingBuffer.swift)

---

<div class="post-metadata">

**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\
**Post date:** [December 31, 2022, 9:42pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/91 "2022-12-31T21:42:19Z")

</div>

> [@TeamPuzel](#):
>
> I'm curious, and I asked about this before, what's stopping Swift from just having a normal array?

Swift's Array is not normal 🙂?

I implemented a few "stack" based fixed sized arrays via tuples. It's far from convenient:

```swift
struct Array16<T> {
    var elements: (T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T)
}

```

and then you define your own subscript operations, etc. But what if I need an array of 1234 elements? I guess I can define it as Array64\<Array32\> – quite a hassle to maintain and lot's of wasted space.

> [@audulus](#):
>
> Instead I used the (deprecated) OSAtomic APIs:

I assume you only have a single writer (e.g. a microphone) and a single reader (e.g. a speaker) hence it is ok to not protect the head/tail variables, only the fillCount variable. However it looks like your code is missing a load barrier when reading "fillCount" variable. For example this (a bit overkill but should work): "OSAtomicAdd32(0, &fillCount)".

OTOH, with that barrier in place, or with all barriers (atomics) removed completely it won't sound too different I guess – What happens with proper overflow / underflow checks when overflow/underflow happens? A glitch. What happens on overflow/underflow without checks? Also a glitch 🙂

---

<div class="post-metadata">

**Author:** ![audulus](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/audulus/32/40637_2.png) [@audulus](https://forums.swift.org/u/audulus)\
**Post date:** [January 1, 2023, 12:13am UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/92 "2023-01-01T00:13:40Z")

</div>

That code is just adapted from TPCircularBuffer, which is used all over the place: [TPCircularBuffer/TPCircularBuffer.h at master · michaeltyson/TPCircularBuffer · GitHub](https://github.com/michaeltyson/TPCircularBuffer/blob/master/TPCircularBuffer.h)

---

<div class="post-metadata">

**Author:** ![tcldr](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tcldr/32/24405_2.png) [@tcldr](https://forums.swift.org/u/tcldr)\
**Post date:** [January 1, 2023, 4:24pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/93 "2023-01-01T16:24:41Z")

</div>

That's really useful info, thank you.

Some further research on `ManagedBuffer` [kicked up this issue](https://forums.swift.org/t/managedbuffer-header-and-exclusivity/55013) by @Karl, which shows that unless you access `ManagedBuffer`'s header through `withUnsafeMutablePointerToHeader`, the Swift runtime is used to enforce exclusivity.

So maybe it's better for me to do as you have done in your RingBuffer and use `UnsafeMutableBufferPointer ` directly as I'm not confident that the implementation of `ManagedBuffer` won't change and suddenly rely on the runtime.

Looks like I'll be copying @Karl 's trick of pasting my real time code into [Godbolt](https://swift.godbolt.org) and checking for emission of Swift runtime calls to verify realtime safety.

Wish there was a better way!

---

<div class="post-metadata">

**Author:** ![audulus](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/audulus/32/40637_2.png) [@audulus](https://forums.swift.org/u/audulus)\
**Post date:** [January 1, 2023, 4:47pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/94 "2023-01-01T16:47:28Z")

</div>

> [@tcldr](#):
>
> Wish there was a better way!

There's this: [Performance annotations - #64 by tera](https://forums.swift.org/t/performance-annotations/54441/64)

I haven't gotten it to work yet.

---

<div class="post-metadata">

**Author:** ![tcldr](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tcldr/32/24405_2.png) [@tcldr](https://forums.swift.org/u/tcldr)\
**Post date:** [January 1, 2023, 5:12pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/95 "2023-01-01T17:12:24Z")

</div>

Yeah, that would be great. Looks like a pre-pitch though.

Any chance we might see the [performance annotations mentioned here](https://forums.swift.org/t/performance-annotations/54441), coming to fruition in 2023, @Erik_Eckstein ? Lots of appetite for them here!

---

<div class="post-metadata">

**Author:** ![audulus](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/audulus/32/40637_2.png) [@audulus](https://forums.swift.org/u/audulus)\
**Post date:** [January 3, 2023, 5:25pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/96 "2023-01-03T17:25:57Z")

</div>

If anyone is curious, or wants to tell me the hundreds of ways I've violated realtime safety via the swift runtime, my WIP code is here (new engine for a popular open source framework) [AudioKit/Sources/AudioKit/Internals/Engine at v6 · AudioKit/AudioKit · GitHub](https://github.com/AudioKit/AudioKit/tree/v6/Sources/AudioKit/Internals/Engine)

---

<div class="post-metadata">

**Author:** ![Max\_Desiatov](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/max_desiatov/32/20654_2.png) [@Max\_Desiatov](https://forums.swift.org/u/Max_Desiatov)\
**Post date:** [January 3, 2023, 8:09pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/97 "2023-01-03T20:09:30Z")

</div>

These are already [available as underscored attributes](https://github.com/apple/swift/blob/main/docs/ReferenceGuides/UnderscoredAttributes.md#_noallocation-_nolocks), aren't they?

---

<div class="post-metadata">

**Author:** ![tcldr](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tcldr/32/24405_2.png) [@tcldr](https://forums.swift.org/u/tcldr)\
**Post date:** [January 3, 2023, 8:30pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/98 "2023-01-03T20:30:30Z")

</div>

Oh, interesting. Thanks! Will give these a try later this week.

---

<div class="post-metadata">

**Author:** ![audulus](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/audulus/32/40637_2.png) [@audulus](https://forums.swift.org/u/audulus)\
**Post date:** [January 3, 2023, 8:40pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/99 "2023-01-03T20:40:47Z")

</div>

Something tells me it doesn't yet work 🤣

 ![image](https://global.discourse-cdn.com/swift/original/3X/9/c/9c25807122577ed2cbbc4f463c98e97dc38fcb0b.jpeg)

---

<div class="post-metadata">

**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\
**Post date:** [January 3, 2023, 9:46pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/100 "2023-01-03T21:46:33Z")

</div>

> [@audulus](#):
>
> > [@tera](#):
> >
> > However it looks like your code is missing a load barrier when reading "fillCount" variable.
> 
> That code is just adapted from TPCircularBuffer, which is used all over the place: [TPCircularBuffer/TPCircularBuffer.h](https://github.com/michaeltyson/TPCircularBuffer/blob/master/TPCircularBuffer.h)

Perhaps the error was always there and propagated from source to source over the years. Lock-free algorithms are very easy to do wrong and bugs reproducible on "weaker memory model" architectures could go unnoticed on "stronger memory model" architectures (e.g. x86 or first iPhones that weren't multicore at all). In case of audio – bugs (audio glitched) could get unnoticed when happen rarely.

It's easy to google to get results like [here](https://preshing.com/20120710/memory-barriers-are-like-source-control-operations/) or [here](https://preshing.com/20130823/the-synchronizes-with-relation/) showing the need of load barrier on the reading side. By no means I am an expert on this _huge_ topic so I have to trust experts and test results.

> [@audulus](#):
>
> If anyone is curious, or wants to tell me the hundreds of ways I've violated realtime safety via the swift runtime, my WIP code is here (new engine for a popular open source framework) [AudioKit/Sources/AudioKit/Internals/Engine](https://github.com/AudioKit/AudioKit/tree/v6/Sources/AudioKit/Internals/Engine)

I see, you've changed from OSAtomicAdd32 that I saw in a previous version of your code to Atomics. Does it work ok in playground or did you drop that support?

Looking at RingBuffer file (and only that file) It feels almost alright to me (but I'd like to hear from real subject experts!) other than these two things plus one extra:

- you are using "relaxed" for both reading and writing, it should be `acquiring` on read and `releasing` on write.
- classes are huge no-no for realtime algorithms (retain/release are outlawed), and ManagedAtomic is a class. I'd use `UnsafeAtomic` which is a struct. _(the nomenclature in atomics library could've been better)_. I'll live it with you to check the implications of having the RingBuffer itself as a class (does it cause ARM traffic when you call its instance methods? worth checking the asm).
- something from my experience, but may be not important in your case: for audio I prefer to have APIs that give me ability to write audio frames (one or more channels), audio samples or just bytes, as sometimes the blocks of audio data I have to write is variable length (useful for things like Varispeed component or codecs like mp3 or opus with variable input to output ratio expressed in bytes). Your API enforces fixed size, unless the idea is to use T = UInt8 and use the "pop(to ptr: UnsafeMutableBufferPointer" / "push(from ptr: UnsafeBufferPointer" - which is not convenient (IRT variable the size still: would be quite hard to vary the count of existing UnsafeMutableBufferPointer).

---

<div class="post-metadata">

**Author:** ![audulus](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/audulus/32/40637_2.png) [@audulus](https://forums.swift.org/u/audulus)\
**Post date:** [January 3, 2023, 10:19pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/101 "2023-01-03T22:19:27Z")

</div>

> [@tera](#):
>
> Perhaps the error was always there and propagated from source to source over the years. Lock-free algorithms are very easy to do wrong and bugs reproducible on "weaker memory model" architectures could go unnoticed on "stronger memory model" architectures (e.g. x86 or first iPhones that weren't multicore at all). In case of audio – bugs (audio glitched) could get unnoticed when happen rarely.

Yeah, I think you're right about that. I've been updating my code to take into account memory ordering.

> [@tera](#):
>
> you are using "relaxed" for both reading and writing, it should be `acquiring` on read and `releasing` on write.

Yeah, I've got a new version that I think its correct.

> [@tera](#):
>
> classes are huge no-no for realtime algorithms (retain/release are outlawed), and ManagedAtomic is a class. I'd use `UnsafeAtomic` which is a struct.

Oh sheesh. That's rough, since I'm using classes in various places. I can use `std::shared_ptr` in C++ (also reference counted), so why are retain/release a problem? In C++ I just need to ensure the object isn't deallocated on the rt thread (various approaches to that).

> [@tera](#):
>
> for audio I prefer to have APIs that give me ability to write audio frames (one or more channels),

Good points. Here's the C++ version which I'm going to port over: [AudioKitEX/Sources/CAudioKitEX/Internals/RingBuffer.h at main · AudioKit/AudioKitEX · GitHub](https://github.com/AudioKit/AudioKitEX/blob/main/Sources/CAudioKitEX/Internals/RingBuffer.h)

> [@tera](#):
>
> Does it work ok in playground or did you drop that support?

No, we dropped that support for now. I wanted to get it passing TSAN and thought maybe the OSAtomic calls weren't good enough.

---

<div class="post-metadata">

**Author:** ![ksluder](https://avatars.discourse-cdn.com/v4/letter/k/eada6e/32.png) [@ksluder](https://forums.swift.org/u/ksluder)\
**Post date:** [January 3, 2023, 10:32pm UTC](https://forums.swift.org/t/realtime-threads-with-swift/40562/102 "2023-01-03T22:32:12Z")

</div>

> [@tera](#):
>
> classes are huge no-no for realtime algorithms (retain/release are outlawed)

Hold up. Retain and release are not “outlawed”. The normal path for retain/release atomically increments/decrements the inline refcount. There are circumstances where retain will cause a side allocation (overflowing retain count), and allocations can take the malloc lock. But as we already know, if your program touches memory at all, it’s impossible to avoid every single thing that can cause a hitch.

(I’d be more concerned about the overhead of dynamic exclusivity checking.)

[Previous page](https://forums.swift.org/t/realtime-threads-with-swift/40562.md?page=4)

[Next page](https://forums.swift.org/t/realtime-threads-with-swift/40562.md?page=6)