Realtime threads with Swift

Yep, that is an advantage. It’s been a while since I worked on SIL, so that might only be true early in the pipeline, but there are already diagnostic passes that work on SIL even in no-debug-info modes, so your pass should be able to work as well.

2 Likes

Ok so if I have var ptr = UnsafeMutablePointer<Int>.allocate(capacity: 1), then the SIL is:

// function_ref static UnsafeMutablePointer.allocate(capacity:)
  %9 = function_ref @$sSp8allocate8capacitySpyxGSi_tFZ : $@convention(method) <τ_0_0> (Int, @thin UnsafeMutablePointer<τ_0_0>.Type) -> UnsafeMutablePointer<τ_0_0> // user: %10
  %10 = apply %9<Int>(%8, %4) : $@convention(method) <τ_0_0> (Int, @thin UnsafeMutablePointer<τ_0_0>.Type) -> UnsafeMutablePointer<τ_0_0> // user: %11
  store %10 to [trivial] %3 : $*UnsafeMutablePointer<Int> // id: %11

and the IR is:

%8 = call noalias i8* @swift_slowAlloc(i64 8, i64 %7) #1
  store i8* %8, i8** getelementptr inbounds (%TSp, %TSp* @"$s8realtime3ptrSpySiGvp", i32 0, i32 0), align 8

Seems straightforward to recognize swift_slowAlloc. I'm not sure what to do in the case of SIL. Recognizing something like UnsafeMutablePointer.allocate would mean that I would have to recognize calls outside of the Swift runtime, which would mean a potentially long (and incomplete) list of realtime-violating functions, no?

EDIT:

Ok I ran with the SIL passes (swiftc -emit-sil -O) and I get:

%15 = builtin "allocRaw"(%14 : $Builtin.Word, %12 : $Builtin.Word) : $Builtin.RawPointer

so presumably the set of builtin functions could be categorized as realtime-safe/unsafe. Would that be sufficient?

I think you have to assume any non-inlined function is realtime-unsafe (or do an analysis of any called functions), so I’m not sure this case is worse. But yeah, some SIL builtins and possibly some primitive operations might be realtime-unsafe as well.

I was made aware of this thread's existence during WWDC, and I am also interested in seeing Swift progress forwards as a more realtime-friendly (or "truly systems level") language in the future.

Unfortunately, for an AudioUnit that's written in Swift right now, any progress made via the above work will be hampered by the fact that we're going to get additional swift_allocObject calls inserted into the code that's executed in the audio unit's internalRenderBlock. More details here: [SR-9662] Subclassing an Objective-C class that returns a block unexpectedly (?) emits an unnecessary reabstraction thunk · Issue #52106 · apple/swift · GitHub

In addition to your checks that guard against allocations, we'll also want to have checks to ensure that there are no surprise locks that are taken behind the scenes in the runtime when we interact with certain reference types. Especially in the cases where someone might (as an example) accidentally try using an AVAudioPCMBuffer instance, which is backed by an Objective C implementation, and hence unsafe.

Perhaps what we need is a "runtimeless" subset (superset?) of Swift that would allow us to build out libraries/packages that don't have access to the standard library (or an alternate version/subset of it) before we could achieve something like this.

But that'd mean there's no heap-allocated types, no ARC, and I'm not sure that such a thing would even closely resemble the Swift that we know today. :smile:

Anyway, count me as a :heavy_plus_sign: for cheering on any efforts in this direction. I've got a ton of Swift-powered DSP that I use in a non-realtime context, and would love to get some more mileage out of it, and stop maintaining parallel C++ and/or Rust implementations…

8 Likes

I presented some of my investigation/progress in the AudioKit office hours:

9 Likes

Was just looking at [SR-9662] Subclassing an Objective-C class that returns a block unexpectedly (?) emits an unnecessary reabstraction thunk · Issue #52106 · apple/swift · GitHub... I wonder if using an UnsafeMutablePointer to the DSP kernel gets around the issues you were seeing? It seemed to work, if you look at the code in my presentation above.

EDIT: ok I get it. The getter is returning a function pointer to a "reabstraction thunk helper" which allocates. Fixing that would have to be part of this work. As a work-around, could we use indirection to call a @convention(Swift) render block from an ObjC block?

1 Like

Not to overwhelm you with choices, but there might also be a middle approach here—if we emitted the diagnostics during IRGen, at the point we try to emit a call to a realtime-unsafe runtime function, then we should be close enough to the SIL instruction that triggered the emission to get diagnostic location info from it at that point.

2 Likes

@liscio ... coming late to the party and not adding much except a +1 to your comment. I use swift to program microcontrollers (swift for arduino) and have experienced exactly this pain. MCUs are basically hard realtime environments. The 'microswift' I made has a super trimmed stdlib and almost no runtime, it has (almost) no heap allocated types, no ARC, no classes, no closures (except convention(c) function callbacks). It was the only way I could get it to work really. I still struggle with swift unexpectedly emitting loads of rtti type 'metadata' unwanted. That all said, it's been rewarding and it's useable... but the complaint I'm always hearing is basically a polite version of what you said... 'this doesn't even closely resemble the Swift I know!'

2 Likes

actually that's not true now, that was official Apple policy until 2016.

compare this: https://asciiwwdc.com/2015/sessions/508
to this: https://asciiwwdc.com/2016/sessions/507

i don't know if anything material change between 2015 and 2016, but my understanding is that for the last 5 years apple no longer non-recommending swift for realtime audio. (just remember not doing anything beyond reading memory, writing memory and math.)

1 Like

Creating a new AU target with the latest version of Xcode uses C++ for the DSP code.

Anyway, I think we're all in agreement that Swift isn't currently good for realtime because of its dynamic allocation behavior.

really, watch that WWDC 2016 Session 507 video if you can still find it (or ask someone to send you the relevant snapshot): i remember it very well - a square wave generator with a render proc written in pure swift. if Apple's Doug Wyatt says it's good - that sounds good to me.

also this: apple systems are not hard realtime. page fault occurs - nothing will stop audio glitch. and this can happen regardless of what language you use C or swift.

you can measure how many glitches say, per hour you are actually getting - inputProc/renderProc has timestamp parameter and if the previous timeStamp.sampleCount + numSamples doesn't match the current timeStamp.sampleCount - that's a glitch. do the two barebones implementation of, say, square wave generator, one in C, another in swift and compare the actual results on a couple of platforms. i wouldn't be surprised if you get no glitches in both implementations. or if you get a few glitches per hour - again in both implementations.

take Apple advice of 6-10 years ago with a grain of salt, especially given they stopped non recommending swift for real time audio 5 years already. obviously i am not saying it's good to use swift containers, or async dispatch or mutexes, etc... just read memory / write memory and math.

Swift would be fine for your single oscillator example which will not put much stress on the system. In my app, for example, users are often pushing it to the limit, and if some dynamic allocations snuck into the audio thread, that would increase the probability of a glitch.

so, in those bare bones tests just put additional processing to push processing to its limits:

func renderProc(...) {
	generateSquareWave()

	for i in 0...N {
		some silly no op here
	}
}

and choose N so you spend, say, 70% of allowed time, which is IOSize / sampleRate.

measure!

That's also not a good test because doing that no-op will not allocate. Part of the problem here is that it's a harder to predict in Swift what will allocate, hence the impetus to have some validation.

allocations are easy to avoid, just do not call anything but memmove and math.
and it is equally easy to "validate" by counting those "glitches per hour" if any.

one practical problem you may encounter - looks like you've already have all that massive amount of kernel code in C... yes, you can call that code from swift's input/render proc but ... is that that important?

found the link, interesting bits are around 0:38: https://devstreaming-cdn.apple.com/videos/wwdc/2016/507n0zrhzxdzmg20zcl/507/507_hd_delivering_an_exceptional_audio_experience.mp4

compare and contrast to 2015 video, around 0:49: Audio Unit Extensions - WWDC15 - Videos - Apple Developer

2 Likes

ideally there must be an ability to denote functions / closures / etc with a marker, say "@realtime". then it's no longer a guesswork or a question of following or not following wwdc guidelines - compiler will do the relevant check and either issue an error or refrain from using unsafe constructs in realtime functions. i wonder if there are such precedents in other languages.

(btw, using C or C++ doesn't automatically mean it's realtime safe. there are many things in there that are not (malloc, mutexes, semaphores, STL containers, smart pointers, etc, it's just the language itself is smaller / simpler its runtime is more simple and predictable).

2 Likes

See above my attempts to implement @realtime as a LLVM pass: Realtime threads with Swift - #34 by audulus

3 Likes

Right, and that's the point: have the compiler validate since it's harder to do it manually in Swift.

I've seen so much realtime unsafe audio code out in the field that I think having compiler validation would be very helpful.

6 Likes

yep, @realtime keyword looks a step in the right direction to me. once/if this is baked in the compiler swift can become a true realtime-safe language indeed. until this is done the situation is on par with what realtime programming is done with other languages: carefully avoid certain api calls and language constructs. in case of swift things to avoid would be classes (arc), containers, escaping closures, and many other things. and carefully check the resulting asm for anything suspicious. here is an example of a simple square wave generator and the corresponding intel asm (i used @inline(never) to avoid unwanted loop unrolling to keep the asm simple):

import Foundation
import AudioToolbox
import AVFoundation

func main() {
    let unitDesc = AudioComponentDescription(componentType: kAudioUnitType_Output, componentSubType: kAudioUnitSubType_HALOutput, componentManufacturer: kAudioUnitManufacturer_Apple, componentFlags: 0, componentFlagsMask: 0)
    let unit = try! AUAudioUnit(componentDescription: unitDesc, options: [])
    let hardwareFormat = unit.outputBusses[0].format
    let renderFormat = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: hardwareFormat.sampleRate, channels: 1, interleaved: false)!
    try! unit.inputBusses[0].setFormat(renderFormat)
    unit.outputProvider = renderProc
    try! unit.allocateRenderResources()
    try! unit.startHardware()
    sleep(30000)
    unit.stopHardware()
}

func renderProc(actionFlags: UnsafeMutablePointer<AudioUnitRenderActionFlags>, timestamp: UnsafePointer<AudioTimeStamp>, frameCount: AUAudioFrameCount, inputBusNumber: Int, inputData: UnsafeMutablePointer<AudioBufferList>) -> AUAudioUnitStatus {
    let ptr = inputData.pointee.mBuffers.mData.unsafelyUnwrapped.assumingMemoryBound(to: Int16.self)
    var i: Int = 0
    while i < frameCount {
        i = proc(ptr, i)
    }
    return 0
    
//    0x100003c30 <+0>:  pushq  %rbp
//    0x100003c31 <+1>:  movq   %rsp, %rbp
//    0x100003c34 <+4>:  pushq  %r15
//    0x100003c36 <+6>:  pushq  %r14
//    0x100003c38 <+8>:  pushq  %rbx
//    0x100003c39 <+9>:  pushq  %rax
//    0x100003c3a <+10>: movl   %edx, %r14d
//    0x100003c3d <+13>: movq   0x10(%r8), %rbx
//    0x100003c41 <+17>: movl   %edx, %r15d
//    0x100003c44 <+20>: xorl   %eax, %eax
//    0x100003c46 <+22>: jmp    0x100003c5b               ; <+43> at main.swift:21:13
//    0x100003c48 <+24>: nopl   (%rax,%rax)
//    0x100003c50 <+32>: movq   %rbx, %rdi
//    0x100003c53 <+35>: movq   %rax, %rsi
//    0x100003c56 <+38>: callq  0x100003cf0               ; audio.proc(...) -> Swift.Int at main.swift:29
//    0x100003c5b <+43>: testq  %rax, %rax
//    0x100003c5e <+46>: js     0x100003c50               ; <+32> at main.swift:22:13
//    0x100003c60 <+48>: testl  %r14d, %r14d
//    0x100003c63 <+51>: setne  %cl
//    0x100003c66 <+54>: testq  %rax, %rax
//    0x100003c69 <+57>: setne  %dl
//    0x100003c6c <+60>: cmpq   %r15, %rax
//    0x100003c6f <+63>: jge    0x100003c75               ; <+69> at main.swift:24:5
//    0x100003c71 <+65>: orb    %dl, %cl
//    0x100003c73 <+67>: jne    0x100003c50               ; <+32> at main.swift:22:13
//    0x100003c75 <+69>: xorl   %eax, %eax
//    0x100003c77 <+71>: addq   $0x8, %rsp
//    0x100003c7b <+75>: popq   %rbx
//    0x100003c7c <+76>: popq   %r14
//    0x100003c7e <+78>: popq   %r15
//    0x100003c80 <+80>: popq   %rbp
//    0x100003c81 <+81>: retq
}

@inline(never)
func proc(_ ptr: UnsafeMutablePointer<Int16>, _ i: Int) -> Int {
    let n = (((i >> 7) & 1) << 12) - 0x800
    ptr[i] = Int16(truncatingIfNeeded: n)
    return i &+ 1
    
//    0x100003cf0 <+0>:  pushq  %rbp
//    0x100003cf1 <+1>:  movq   %rsp, %rbp
//    0x100003cf4 <+4>:  movl   %esi, %eax
//    0x100003cf6 <+6>:  shll   $0x5, %eax
//    0x100003cf9 <+9>:  andl   $0x1000, %eax             ; imm = 0x1000
//    0x100003cfe <+14>: addl   $0xfffff800, %eax         ; imm = 0xFFFFF800
//    0x100003d03 <+19>: movw   %ax, (%rdi,%rsi,2)
//    0x100003d07 <+23>: leaq   0x1(%rsi), %rax
//    0x100003d0b <+27>: popq   %rbp
//    0x100003d0c <+28>: retq
}

main()
2 Likes