# An API for bulk random bytes

**URL:** <https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051>\
**Category:** Pitches\
**Created:** [February 9, 2023, 5:56pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051 "2023-02-09T17:56:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![scanon](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/scanon/32/2445_2.png) [@scanon](https://forums.swift.org/u/scanon)\
**Post date:** [February 9, 2023, 5:56pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/1 "2023-02-09T17:56:02Z")

</div>

I'm [proposing](https://github.com/stephentyrone/swift-evolution/blob/nose-full-of-random/proposals/nnnn-random-fill.md) that we add the following requirement to the RandomNumberGenerator protocol:

```swift
mutating func fill(_ buffer: UnsafeMutableRawBufferPointer)

```

with a default implementation that uses the existing `next()` API to produce 64b at a time, and a customized implementation for `SystemRandomNumberGenerator` that directly requests the specified amount of random bytes from the underlying platform APIs.

On its own, this doesn't change much, but we can then adopt it within the standard library in cases where we know that we will need more than 64b of randomness (e.g. shuffling an array) to realize significant speed gains (by amortizing per-call overhead, especially for the SystemRNG). Those changes are not included in this proposal, because they are purely implementation details of the standard library.

This API _does_ make an operation available that is useful on its own in some narrow circumstances (e.g. seeding the state of a higher-throughput RNG), generating random values larger than 64b, or sampling non-uniform distributions. This has very real ergonomic and performance advantages for these use cases. A follow-on proposal once move-only types are accepted will use this API to provide an adapter that buffers the output of a provided source and conforms to the RNG protocol, making this process easier still.

---

<div class="post-metadata">

**Author:** ![xwu](https://avatars.discourse-cdn.com/v4/letter/x/b487fb/32.png) [@xwu](https://forums.swift.org/u/xwu)\
**Post date:** [February 9, 2023, 6:19pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/2 "2023-02-09T18:19:24Z")

</div>

Yes, this is fantastic.

---

<div class="post-metadata">

**Author:** ![dnadoba](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/dnadoba/32/11438_2.png) [@dnadoba](https://forums.swift.org/u/dnadoba)\
**Post date:** [February 9, 2023, 6:26pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/3 "2023-02-09T18:26:38Z")

</div>

There was already a pitch a while ago: [Pitch: Requesting larger amounts of randomness from SystemRandomNumberGenerator](https://forums.swift.org/t/pitch-requesting-larger-amounts-of-randomness-from-systemrandomnumbergenerator/27226)  
cc: @lukasa

---

<div class="post-metadata">

**Author:** ![dnadoba](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/dnadoba/32/11438_2.png) [@dnadoba](https://forums.swift.org/u/dnadoba)\
**Post date:** [February 9, 2023, 6:43pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/4 "2023-02-09T18:43:39Z")

</div>

I definitely see the need for this so big +1 from my side. However, only adding an unsafe version seems to be at odds with Swifts safety first principal. I think we should at least add a safe version of this e.g.:

```swift
func randomBytes(count: Int) -> [UInt8]

```

[as proposed by @lukasa.](https://forums.swift.org/t/pitch-requesting-larger-amounts-of-randomness-from-systemrandomnumbergenerator/27226)

This can be implemented on top of the proposed `fill(_:)` or the other way around.

Alternatively we could also add new API to `Array` e.g.:

```swift
extension Array {
    init<Generator: RandomNumberGenerator>(randomBytesCount: Int, using: inout Generator)
}

```

A version where one could choose the integer type and range would be nice too but I'm not sure if this can be implement efficiently or if there is even an actually need for that.

---

<div class="post-metadata">

**Author:** ![scanon](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/scanon/32/2445_2.png) [@scanon](https://forums.swift.org/u/scanon)\
**Post date:** [February 9, 2023, 6:44pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/5 "2023-02-09T18:44:28Z")

</div>

Allocating new storage entirely defeats the purpose of this API (to remove as much overhead as possible from bulk-randomness generation).

A follow-on proposal would provide API to produce bulk random numbers, rather than bulk random bytes, but the thing that we need as the fundamental building-block (and therefore on the protocol) is a non-allocating "fill this buffer" operation.

---

<div class="post-metadata">

**Author:** ![Nevin](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nevin/32/617_2.png) [@Nevin](https://forums.swift.org/u/Nevin)\
**Post date:** [February 9, 2023, 6:50pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/6 "2023-02-09T18:50:51Z")

</div>

+1

---

<div class="post-metadata">

**Author:** ![Sajjon](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/sajjon/32/12044_2.png) [@Sajjon](https://forums.swift.org/u/Sajjon)\
**Post date:** [February 9, 2023, 7:16pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/7 "2023-02-09T19:16:53Z")

</div>

Will this be able to replace SecRandomCopyBytes [Apple Developer Documentation](https://developer.apple.com/documentation/security/1399291-secrandomcopybytes?preferredLanguage=occ)

But is … non throwing 😀

---

<div class="post-metadata">

**Author:** ![scanon](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/scanon/32/2445_2.png) [@scanon](https://forums.swift.org/u/scanon)\
**Post date:** [February 9, 2023, 7:28pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/8 "2023-02-09T19:28:12Z")

</div>

We could certainly make it throw, though in practice these calls "never" fail (for sufficiently hand-wavy notions of "never"). Throwing could be beneficial on more niche platforms, however.

---

<div class="post-metadata">

**Author:** ![Sajjon](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/sajjon/32/12044_2.png) [@Sajjon](https://forums.swift.org/u/Sajjon)\
**Post date:** [February 9, 2023, 7:33pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/9 "2023-02-09T19:33:11Z")

</div>

Right, makes perfect sense, but least they would be called with de facto `try` (since throwing), so no need to check any semi-awkward errSecSuccess

---

<div class="post-metadata">

**Author:** ![jrose](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jrose/32/408_2.png) [@jrose](https://forums.swift.org/u/jrose)\
**Post date:** [February 9, 2023, 8:25pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/10 "2023-02-09T20:25:37Z")

</div>

Could it take a MutableCollection of bytes instead? Possibly with a check for UMRBP and/or contiguous storage as the first thing. That would solve the unsafe-API concern without losing the no-allocation guarantee.

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [February 10, 2023, 11:04am UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/11 "2023-02-10T11:04:11Z")

</div>

+1 - I also [pitched this](https://forums.swift.org/t/filling-buffers-using-randomnumbergenerator/53324) before.

I agree with @dnadoba that this shouldn't require an unsafe buffer, and my first thought was, as @jrose suggested, to use `MutableCollection` instead. So +1 to that as well.

IMO this API should scale to use-cases such as filling an `Array` with random bytes or scribbling over a `Data`, without requiring those developers to use unsafe constructs. If some random number generators can make use of contiguous mutable storage (e.g. because they are C functions provided by the system), they should do so as an implementation detail.

* * *

Also, if we're taking a look at the RNG API, I'd also suggest (once again) that we add static member syntax for `SystemRandomNumberGenerator`. It's annoying to type out, and now that we can infer generic parameters, it would make these functions a lot more ergonomic to read and write:

```swift
// We should add this:
extension RandomNumberGenerator where Self == SystemRandomNumberGenerator {
    static var system: Self { Self() }
}

// To allow code like this:
func doSomething(rng: some RandomNumberGenerator = .system) {
  ...  
}

```

---

<div class="post-metadata">

**Author:** ![dnadoba](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/dnadoba/32/11438_2.png) [@dnadoba](https://forums.swift.org/u/dnadoba)\
**Post date:** [February 10, 2023, 11:54am UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/12 "2023-02-10T11:54:20Z")

</div>

> [@Karl](#):
>
> ```swift
> // To allow code like this:
> func doSomething(rng: some RandomNumberGenerator = .system) {
> ...  
> }
> 
> ```

The problem is that this is usually not exactly the signature you want. You actually want the parameter to be `inout` and we sadly don't yet support default values for `inout` parameters.

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [February 10, 2023, 12:06pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/13 "2023-02-10T12:06:57Z")

</div>

Ahh, shame.

I should have looked at the previous pitch more carefully. You could still use it in an explicit overload if you write it like this:

```swift
extension RandomNumberGenerator where Self == SystemRandomNumberGenerator {

  static var system: Self {
    get { SystemRandomNumberGenerator() }
    set { /* No-op, throwaway instance. */ }
  }
}

// To allow:
func doSomething() {
  doSomething(rng: &.system)
}

```

But yeah... it's not as nice as just using it as a default argument. _sigh_... maybe one day...

---

<div class="post-metadata">

**Author:** ![bbrk24](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/bbrk24/32/27370_2.png) [@bbrk24](https://forums.swift.org/u/bbrk24)\
**Post date:** [February 10, 2023, 12:22pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/14 "2023-02-10T12:22:02Z")

</div>

> [@jrose](#):
>
> Could it take a MutableCollection of bytes instead?

What element type(s) should it use? The most obvious choices to me are:

- `UInt8` (i.e. `UnsafeRawBufferPointer.Element`)
- `UInt64` (i.e. the return type of `RandomNumberGenerator.next()`)
- a generic `FixedWidthInteger & UnsignedInteger` (i.e. the generic type argument to `RandomNumberGenerator.next(upperBound:)`)

---

<div class="post-metadata">

**Author:** ![1-877-547-7272](https://avatars.discourse-cdn.com/v4/letter/1/278dde/32.png) [@1-877-547-7272](https://forums.swift.org/u/1-877-547-7272)\
**Post date:** [February 10, 2023, 10:19pm UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/15 "2023-02-10T22:19:10Z")

</div>

> [@jrose](#):
>
> Could it take a MutableCollection of bytes instead? Possibly with a check for UMRBP and/or contiguous storage as the first thing.

How would this method work in cases where the collection doesn't provide contiguous storage? I can see three possibilities:

1. It would repeatedly call the random byte–generation function for each element. This can be expensive on non-Apple platforms like Linux, where `getrandom`, `getentropy`, and `/dev/urandom` all require one syscall per use amortized (`/dev/urandom` requires a call to `open` on first use). Each syscall requires an expensive userspace -\> kernel context switch.

2. It would create a buffer using `withUnsafeTemporaryAllocation`, fill the buffer with random bytes, then copy the buffer's contents to the collection. This option would require the least amount of potential random number generation–related syscalls, but it may require a heap allocation (including potential heap allocation–related syscalls) if the collection is large enough.

3. It would allocate a buffer up to a certain size such that it can always be stack-allocated. It would then repeatedly fill that buffer with random bytes, then copy the buffer's contents to the collection until the entire collection has been filled. This would never require a heap allocation, but it would require multiple syscalls to fill large collections.

All three options here create overhead which undermines the purpose of this API (which is, to quote @scanon, "to remove as much overhead as possible from bulk-randomness generation").

* * *

I think we should go with the original proposed method that accepts an `UnsafeMutableRawBufferPointer`. Then, we could build safe constructs on top of that method like the following:

> **Click to view code**
>
> ```swift
> extension Array where Element: FullWidthInteger {
> static func randomlyFilled(count: Int) -> Self {
> var generator = SystemRandomNumberGenerator()
> return randomlyFilled(count: count, using: &generator)
> }
>     
> static func randomlyFilled(count: Int, using generator: inout some RandomNumberGenerator) -> Self {
> return Self(unsafeUninitializedCapacity: count, initializingWith: { buffer, initializedCount in
> generator.fill(UnsafeMutableRawBufferPointer(buffer))
> initializedCount = buffer.count
> })
> }
> }
> 
> extension RangeReplaceableCollection where Element: FullWidthInteger {
> static func randomlyFilled(count: Int) -> Self {
> var generator = SystemRandomNumberGenerator()
> return randomlyFilled(count: count, using: &generator)
> }
>     
> static func randomlyFilled(count: Int, using generator: inout some RandomNumberGenerator) -> Self {
> // Perhaps we should consider adding a failable init(unsafeUninitializedCapacity:initializingWith:)
> // initializer to `RangeReplaceableCollecion` to avoid unnecessary overhead here
> return withUnsafeTemporaryAllocation(of: Element.self, capacity: count) { buffer in
> generator.fill(UnsafeMutableRawBufferPointer(buffer))
> return Self(buffer)
> }
> }
> }
> 
> extension MutableCollection where Element: FullWidthInteger {
> mutating func randomlyFill() {
> var generator = SystemRandomNumberGenerator()
> randomlyFill(using: &generator)
> }
>     
> mutating func randomlyFill(using generator: inout some RandomNumberGenerator) {
> let wasFilledContiguously = withContiguousMutableStorageIfAvailable { buffer in
> generator.fill(UnsafeMutableRawBufferPointer(buffer))
> } != nil
>         
> if !wasFilledContiguously {
> withUnsafeTemporaryAllocation(of: Element.self, capacity: count) { buffer in
> generator.fill(UnsafeMutableRawBufferPointer(buffer))
> var index = startIndex
> var bufferIndex = 0
> while bufferIndex != buffer.count {
> self[index] = buffer[bufferIndex]
> formIndex(after: &index)
> bufferIndex += 1
> }
> }
> }
> }
> }
> 
> /// An integer where the in-memory representation of the integer is equivalent to the value of the integer.
> protocol FullWidthInteger: FixedWidthInteger {}
> extension UInt8: FullWidthInteger {}
> extension UInt16: FullWidthInteger {}
> extension UInt32: FullWidthInteger {}
> extension UInt64: FullWidthInteger {}
> extension Int8: FullWidthInteger {}
> extension Int16: FullWidthInteger {}
> extension Int32: FullWidthInteger {}
> extension Int64: FullWidthInteger {}
> 
> ```

---

<div class="post-metadata">

**Author:** ![fclout](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/fclout/32/36155_2.png) [@fclout](https://forums.swift.org/u/fclout)\
**Post date:** [February 11, 2023, 12:41am UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/16 "2023-02-11T00:41:10Z")

</div>

Theo gave a good talk several years ago about why randomness operations shouldn't encode failure in their API, ideally should never fail, and in the un-ideal case where they _can_ fail, they should do so by trapping. The advice is mostly relevant to cryptographic uses of random number generators, but since `SystemRandomNumberGenerator` claims to be cryptographically secure, it's probably right to stick to that.

Putting on my security goggles, I'd also like to see if there's any reasonable way that this can be done without using unsafe pointers. I don't feel great about unsafe pointers in protocols because it forces implementations to use unsafe code, and that's particularly annoying for standard library protocols. Is the goal that the caller should be able to put the memory in automatic storage, and that the implementation should be able to get a pointer to contiguous storage to (as an example) call `arc4random_buf`?

---

<div class="post-metadata">

**Author:** ![Nobody1707](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nobody1707/32/2169_2.png) [@Nobody1707](https://forums.swift.org/u/Nobody1707)\
**Post date:** [February 11, 2023, 1:08am UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/17 "2023-02-11T01:08:11Z")

</div>

+1 I've wanted this for a long time. I'm also tentatively for having this API be generic instead of just for `UnsafePointer`.

---

<div class="post-metadata">

**Author:** ![xwu](https://avatars.discourse-cdn.com/v4/letter/x/b487fb/32.png) [@xwu](https://forums.swift.org/u/xwu)\
**Post date:** [February 11, 2023, 1:56am UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/18 "2023-02-11T01:56:12Z")

</div>

> [@fclout](#):
>
> The advice is mostly relevant to cryptographic uses of random number generators, but since `SystemRandomNumberGenerator` claims to be cryptographically secure, it's probably right to stick to that.

But for the protocol requirement itself, I can see why a throwing affordance might be more flexible.

---

<div class="post-metadata">

**Author:** ![fclout](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/fclout/32/36155_2.png) [@fclout](https://forums.swift.org/u/fclout)\
**Post date:** [February 11, 2023, 4:18am UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/19 "2023-02-11T04:18:35Z")

</div>

> [@xwu](#):
>
> But for the protocol requirement itself, I can see why a throwing affordance might be more flexible.

Why? There's not really any reason to implement a PRNG or a CSPRNG that can fail after it's been initialized, and even if there were, what is your client code going to do when the randomness request fails?

---

<div class="post-metadata">

**Author:** ![fclout](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/fclout/32/36155_2.png) [@fclout](https://forums.swift.org/u/fclout)\
**Post date:** [February 11, 2023, 4:37am UTC](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051/20 "2023-02-11T04:37:20Z")

</div>

The implementation you describe either calls `getrandom()` every time the user calls `next()`, or it already has a large-ish buffer that it can use.

I would also argue that it shouldn't be a goal to map this feature as closely to `getrandom()` as possible. A high-quality implementation would call it once at initialization time to seed a CSPRNG and never go to the kernel again.

[Next page](https://forums.swift.org/t/an-api-for-bulk-random-bytes/63051.md?page=2)