Is this dumb? (using Integer Generic Parameter)

Using Swift
, , ,
1

I am writing some MIDI things using rtmidi C library, and I wanted to make it as static as possible. Unfortunately some MIDI messages like SysEx messages are arbitrary sized, so I had to either just use normal Array<UInt8> to make the struct representing SysEx message dynamically sized, or do this:

protocol BinaryMessage: Sendable, BitwiseCopyable {
    // ...
    var size: Int { get }
    var span: Span<UInt8> {
        @_lifetime(borrow self)
        yielding borrow
    }
    // ...
}
struct SysExMessage<let size: Int>: BinaryMessage {
    internal let _header: UInt8 = 0xF0
    internal let _content: [size of UInt8]
    internal let _end: UInt8 = 0xF7

    init(with buffer: UnsafeBufferPointer<UInt8>) { ... }
}

{ (messageBuffer: UnsafeBufferPointer<UInt8>, messageLength: Int, ...) in
    // ...
    guard messageBuffer[0] != 0xF0 else {
        let message: any BinaryMessage = switch (messageLength - 2) {
        case 1:
            SysExMessage<1>(with: messageBuffer)
        case 2:
            SysExMessage<2>(with: messageBuffer)
        case 3:
            SysExMessage<3>(with: messageBuffer)
        case 4:
            SysExMessage<4>(with: messageBuffer)
        // and so on ...
        default:
            fatalError()
        }
        
        continuation.1?.yield(message) // yielding to AsyncStream<any BinaryMessage>
        return
    }
}

which seems silly, considering it has to go into existential type (any BinaryMessage right after anyways.

I wanted to do this to make SysExMessage values have the exact same bytes as the raw MIDI message, even if it doesn't really have performance benefits. But if I want to do this I'd have to explicitly write every case (or write a macro, either ways have every case in the code), since integer generic parameters can only be literal, not any kind of value that only exists at runtime.

Normal MIDI voice messages (like note on, off, control change) can also be either 2 or 3 bytes, but I'm just always using 3 bytes there. But SysEx messages can be much longer than that so I can't do that.

I am curious how other people are dealing with dynamically-sized binary data, especially when they need to be lightweight.

I'm also kinda curious if there's any good reason why integer generic parameters must be known at runtime compile time. I understand without knowing, the compiler can't specialize the generic type, but isn't unspecialized generics already a language feature? Is it just because integer generic parameters are relatively new feature, or is it going to always be restricted like this?

1 Like
2

I think you mean known at compile time?

The feature you’re looking for is the integer generic equivalent of existentials. An “integer generic parameter existential metatype” would just be a value of integer type.

Just like an existential can be “opened” by binding the concrete type stored inside to an ordinary generic parameter, you can imagine “opening” an integer value, something like this:

func f<let n: Int>(_: n.Type) {
  let a: InlineArray<Float, n> = …
}

let x = 10
f(x)

This is not possible today, unfortunately. It might be possible in the future, and conceptually it would fit in the existing language model. (I haven’t thought deeply about the type vs metatype distinction here, don’t get too hung up on the details of the syntax above.)

In your MIDI example though, we could discuss alternate approaches to solving the problem. What would you lose by using ordinary arrays? Are you looking to gain performance, type safety, or something else? Perhaps UniqueArray would be a better fit?

6 Likes
3

This is a really interesting distinction. I think the integer-generic equivalent of opening existentials is exactly the missing piece here :)

One thing I find particularly interesting is that the restriction seems less about specialization itself and more about the current model for representing values in generic arguments. SE-0452 already allows generic code to abstract over an integer parameter but the argument still needs to originate from a literal or an enclosing integer generic parameter.

For the MIDI example, I wonder if there could eventually be a way to open a runtime integer into a generic context, allowing something like InlineArray<n, UInt8> without having to enumerate every possible size.

That said I think the question about UniqueArray is also worth exploring. If the main goal is lightweight storage and ownership rather than compile time size guarantees it might be a better fit than trying to make every SysEx message a distinct statically sized type.

1 Like
4

Oops, yes.

Yeah! I feel like while InlineArray may not benefit that much because existentials would have some performance cost, thus negating the performance gain of InlineArray, there could be many use cases for this.

Performance is important, but mostly I wanted to make it possible to just bitcast the struct and get the actual raw data. Since MIDI messages, including SysEx, are not only received but also could be sent to other devices, I liked the idea of just bitcasting the struct and getting the bytes that I can then send away. It was more of an interesting puzzle I wanted to solve, rather than some practical engineering.

But it seems like using UniqueArray or just plain old Array like you suggested might be the more reasonable solution.

Thanks!

2 Likes
5

There wouldn’t be any overhead, the integer value would be the “existential container” itself and “opening” would just entail loading the value.

From a practical standpoint (compiler bugs, weird edge cases, code readability, etc), in these situations it’s almost always better to go with the simpler solution that uses as few fancy features as possible anyway.

2 Likes
7

I'd not overcomplicate and go with a much simpler approach:

enum MIDICommand {
    case on
    case off
    case sysEx([UInt8]) // or `case sysEx(Data)`
    ...
}
3 Likes