Currently, we have to write all our realtime audio processing code in C++. Swift can't be used because of its dynamic allocation behavior (this is Apple's policy, not just my opinion). There's quite a bit of extra boilerplate going between the two languages and it's a bit of a bummer. Plus, you're back in footgun C++ land.
I recently learned about region-based memory management and wondered if it could provide a way to use Swift on realtime threads. (There is a realtime extension for Java which uses regions)
So, in the audio case, a simple version might linearly allocate all objects for a given audio render cycle, while imposing some constraints to ensure safety. At the end of the render cycle, the region is freed.
I'm not sure how to best express this syntactically, but perhaps some sort of region block (inspired by autoreleasepool) would do it.
Has anyone thought about this? Being able to write realtime audio code in Swift would be really great.
I actually use Swift for audio processing (on AVFoundation in particular). I don't seem to get into any trouble so long as I avoid funny Collection business and ARC. So no mutating Array, Set, no classes, etc. I then use some of the unsafe APIs to deal with (preallocated) bulks memory.
They don't go into AppStore, so I don't know if they'd have any problem with it, though .
Given that Swift refused GC to achieve deterministic memory behavior for situations exactly like this, it does seem like a use case the language should address! (Otherwise where's my GC? )
Yeah. Plus, it would be nice to be able to use the expressive power of Swift on an audio thread. If you create an array, for example, it may malloc, which would preclude realtime use.
I use AUAudioUnit, so IDK, but I ported templates to another language a few times. So I'd say it shouldn't hold you back unless there's a fundamental problem with the language choice.
To a certain extend, I do get to liberally use (pure) structs since they don't really have those problems. I'd like to think that a better way is to have something like OnStackArray.
I worry that you're going to give people the impression that Swift is OK to use for audio threads. My understanding and that of practically every audio developer I know is that it isn't a good idea. I use AUAudioUnit too, and our DSP kernels are written in C++. We try to move as much as we can out to Swift, but the kernels can't be moved.
If you're writing DSP code in Swift, have you verified that it doesn't allocate?
Ok, I admit it's not great to use Swift (and my tone does sound otherwise), and the deal-breaker is the non-realtime heap allocations. Still, I don't think it'll be an attractive choice even after the zone-based allocation. Even after switching all data to structs, I'd still have to juggle with all the runtime-checks to get the performance to where I want.
(All I've said is a 5-min benchmark on a pet project years back, so take that with a chunk of salt)
I did check it with Instrument since I was also interested (as I read the same comment about avoiding allocations). Structs and (direct) enums are fine, but classes are not. Even moving classes around can incur ARC traffic, so mutating collections are out, and I needed to use preallocated UnsafeMutableBuffer. Also, passing around large structs is a real pain since they get copied again onto the stack. I ended up using withUnsafe... to pass them around as pointer instead, or even manually preallocate them with unsafe APIs.
So yea, it's far from ideal. Though I gotta say, in there, UnsafePointer is the new class.
Local (nested) functions which capture variables can also introduce allocations (and I suppose the same applies to closures).
Those captured variables need to exist in a heap-allocated box rather than the stack, in case a reference to the function escapes. Unfortunately the compiler sometimes fails to prove when it doesn’t escape, meaning innocent-looking code might be allocating boxes.
I just tracked down 2 of those issues in a project I’m working on. Moving the nested functions out led to slightly uglier code, but this code was in the hot path (that’s why I took care to avoid excessive allocations), so that minor reorganisation cut overall execution time by almost 50%.
All of which is to say: I wouldn’t use Swift for real-time programming as it is today.
Yeah, I wonder if some restrictions could be put on what you can do in a real-time scope? Perhaps you can't use closures or nested functions at all! That would be fine, since we don't use anything like that on the C++ side.
Actually I noticed that GCD works more slowly in apps downloaded from AppStore(or especially TestFlight) rather those installed from Xcode directly. I suspect that iOS somehow restricts app's CPU capabilities for downloaded apps.
Most stdlib collections are CoW, so mutating a non-unique collection incurs heap allocations. Even if it's provably unique, you'd still need to juggle with ARC, which I'm still not sure is realtime-safe. The compiler should be able to optimize out ARC on guaranteed references, but IDK.
Also I can't seem to get rid of swift_beginAccess, which I believe is runtime exclusivity check, and it heap allocate 16bytes (once). I could also be because I share the Array between realtime and non-realtime code, so I'm not exactly surprised.
While I'm at it, throwing errors also heap-allocate.
It would be nice to make an open source test suit to catch all bugs of that kind. But it in my experiences the serial queues mutate collections well. Also I have a question. Does the compiler optimization flags could spoil GСD queries?
You definitely don't want to change the array size, for one, since that definitely re-allocate the data. The onetime beginAccess 16byte is small & rare enough to make me cross my finger using Array given that the unsafe code is quite unwieldy.
It reminds me back in the day of Swift beta, where array makes a copy if the operation has the potential to change the array size. It was a confusing time, and I had the same feeling cramming Swift code into realtime env.
The way to get rid of swift_beginAccess would be to remove the need for dynamic checks — avoid reference types and mutable captures in escaping closures.
I can verify that using Swift on audio threads for recording leads to popping noise in the recorded audio. At the same time audio unit tap blocks seem to be immune to this issue and you can freely use Swift allocation there.
Yup, I managed to slip-in most of the constant and locally mutated Arrays. The compiler is doing a fine job detecting that even if arrays technically use classes .
There are times when I accidentally use shared mutable arrays, which understandably incur swift_beginAccess. I guess there aren't many ways around since I could break the exclusivity law there. Even if I access different element (which I guaranteed from synchronization), I still shouldn't be able to simultaneously write to the shared array.
I'm still somewhat scared to use Array since the ones that incur runtime check and the ones that don't look much the same, but I'm pretty sure the compiler has been getting much better.
For better or worse, I'm pretty sure tap blocks aren't executed under real-time constraints.
Could one write an LLVM IR scanner which verifies a particular Swift function is realtime safe?
I'm guessing most of the functions in the runtime aren't realtime safe (looking at swift/Runtime.md at main · apple/swift · GitHub), so the scanner would look for calls to those. It would have to follow calls as well.
.
)
.