RandomNumberGenerator is a protocol that represents a type that can produce uniformly distributed random data on demand. It does not specify how the random data should be produced — that is left up to the implementation. While the proposed method wouldn't benefit an implementation like the one you describe, it would benefit some other implementations, including SystemRandomNumberGenerator.
Also, seeding a custom random number generator is unnecessary and wasteful on macOS, where arc4random_buf already uses a random number generator to produce high-quality random data very quickly.
That's up to the implementation. If they can produce large amounts of random data at once, they won't need to throw some away any more, regardless of contiguous storage. For those implementations which call in to a C function, and hence need a C-friendly storage buffer, they might use a temporary allocation as you describe. It's up to them.
In any case, the vastly more common situation will be contiguous storage. Especially for MutableCollection. What we're discussing is a fallback; and I think your exploration shows that we can provide high-quality fallback implementations.
The other perspective to your question is: how would a user fill a non-contiguous collection with random bytes? Basically by doing the same thing, right?
I don't find that very convincing. We've both said in different ways that calling getrandom() is slow. IMO, it's reasonable to question whether that implementation deserves design concessions.
Designing a protocol so that there is one specific way to make implementations fast is common. Random numbers are deceptively complex and I don't think that we need to be encouraging diversity. For what it's worth, I searched "RandomNumberGenerator" among Swift projects on GitHub and found only one implementation of the protocol.
The fact that seeding arc4random on macOS is not useful is a detail of the macOS implementation.
(EDIT: another thing worth saying is that the system PRNG on Linux does not ask for more than 256 bytes at a time, since beyond that point the request becomes interruptible. I'm not convinced it's the ideal implementation, but the upcoming fill method of the current implementation could easily reserve that much automatic storage without issues if the collection it gets doesn't have contiguous storage.)
disable the relevant functionality (with a relevant feedback)
ask user if they want to proceed (similar to "https is unavailable, do you want to use http?")
Although it would be much more convenient if random couldn't fail in the first place (we don't check even malloc for failures when we call it directly and we can't even attempt doing so when we call it indirectly via "let a = MyClass()").
For SystemRNG, on a platform where it can fail, we would probably make that a trap (like malloc or close failing, such errors are often unrecoverable). For other generators, it should never happen.
In a future where Swift is used in early boot in a kernel or something, this would mean that SystemRNG can’t used before the entropy pool or whatever mechanism the system uses is set up.
Unlike https/http, when these fail, you’re really never in a user-interactive setting where you can ask what to do.
Note that this does not force conforming types to use unsafe code, because a default implementation (which we can vet heavily) is provided by the standard library. Some implementations may choose to provide their own implementation as an optimization, but for conforming types other than the SystemRNG the benefits of doing so are smaller (1-2x, typically).
I understand the concern to be that to use this API requires writing unsafe code, which is understandable.
I think we can easily address the limitation by adding the protocol requirement suggested here, and then also an extension that allows constructing any RangeReplaceableCollection from random bytes something like this:
extension RangeReplaceableCollection where Element == UInt8 {
init<RNG: RandomNumberGenerator>(randomByteCount: Int, from rng: inout RNG) {
var buffer = UnsafeMutableRawBufferPointer.allocate(byteCount: randomByteCount, alignment: 1)
defer { buffer.deallocate() }
self = .init(buffer)
}
}
This doesn't have optimal performance, but we don't have any generic API that gives us the extension we'd want to enable optimal performance, and it ensures that users can avoid having to implement this themselves if they want entirely-safe projects.
I'm slightly wary of adding too many such conveniences, because this is not an API for generating bulk random numbers. This is an API for getting bulk random bits, and it's only a short jump to an attractive nuisance of "I want a bunch of random UInt8s in some range, I should use this" (you shouldn't). Those are safe APIs for bulk random number generation that should be built on top of this.
The smallest workable unit of bits in Swift is a byte, and Swift’s spelling of byte is UInt8, so in effect the phrases “bulk random bits” and “bunch of random UInt8s” are identical. Nobody has proposed adding the “…in some range” capability, and I think it’s reasonable to interpret @lukasa’s response to endorsing an extension that is constrained to Element == UInt8.
This is true (but also regrettable). UnsafeMutableRawBufferPointer has the virtue that even though it is a Collection of UInt8s, it doesn't look like one; the name reminds us that we are operating on a region of memory, not integers.
I don’t see the value in this. Swift’s collection currency type is Array, not Unsafe{Mutable,}{Raw,}BufferPointer. The Standard Library should traffic in currency types when possible.
This reasoning isn’t quite correct. Each fixed-width integer type represents both an integral value and the sequence of bits that is its bit pattern, so UInt8 does indeed represent a byte. But not solely. Every byte can be represented as a UInt8, every UInt8 value can be interpreted as a byte, but these are not identical concepts because every byte of memory is not storage for a number.
going off of this, i think the outcome of the code unit literals pitch was a pretty illustrative argument for why we should have separated "Byte" from UInt8 in the beginning.
It sounds like this new unsafe API is meant to support the SystemRNG because it currently has to do a system call (except on Apple platforms?) to get random bytes, and it's not so important for other implementations. Can we extrapolate that improving Swift's SystemRNG implementation would accomplish the same goal? Keeping around a buffer of 256 random bytes (256 chosen here to match Linux's getrandom easy case) instead of doing a system call every time you call next() would need a little bit of lock-free synchronization, but it would benefit everyone calling next() and not just people calling the new API, and if the expected speedup from a bulk API isn't worth addressing over that, then maybe we don't need it at all.
"Improving the SystemRNG" is under alternatives considered, but:
It's not just syscall overhead, even normal function-call overhead is not insignificant for these operations.
256 bytes isn't actually enough to amortize the overhead effectively, even on platforms like Linux where that's supposed to be the "fast path" (on macOS / Apple Silicon, a 256B buffer nets you only about 3x of the 16x speed improvement that you get at 4096B).
I don't think that "a little bit of lock-free synchronization" really solves the problem; you either end up with per-thread buffers (increasing dirty memory usage by a small but noticeable amount) or atomics that will still have overhead on some targets under contention.
There are potentially some security downsides to this.
The new fill() interface has significant ergonomic improvements as well; if I have a fast seeded RNG that needs, say 32 bytes of internal state to be seeded from the system RNG, or a scratch buffer for a random algorithm, taking a raw pointer to that state or buffer and passing it to fill is quite a bit cleaner and less error-prone than assembling a fill operation out of repeated calls to next( ), despite the superficial "unsafe" API.
As discussed in the proposal, while this mainly benefits the SystemRNG, it will also have benefits for a large class of seedable RNGs that can produce more than 64b of randomness at a time with higher throughput (there are plenty of 2x wins to be had in these cases, which are nothing to sneeze at).
This is also discussed in the proposal, but a follow-on will unlock the same benefits via an adapter that wraps any RNG with a buffer and vends that as a conformer to the RNG protocol (this needs some non-copyable features that don't exist yet, which is why it isn't included in this proposal).
arc4random locks on Darwin, so in today's implementation, contended SystemRNG use is serial. (For completeness, getrandom has per-CPU state instead.)
Is the security risk you mention that someone with an arbitrary read can read the next random bytes? This specific question is not creating new risk as we are already hosting the CSPRNG key material in the address space that will use it.
In general, the reason I don't like new unsafe APIs is that bit by bit, we're building an environment where we accept to compromise between performance and safety, while other projects appear to achieve both. I understand that Jordan's proposed solution (a generic method with a fast path for the case where you received an UnsafeMutableRawBufferPointer) does not give any interesting security properties to your implementation, but not teaching people that interfaces with unsafe pointers in them are good and normal has value when it comes to moving people towards better habits.
The issue is that UnsafeMutableRawBufferPointer is the type we have for working with raw untyped memory. Taking a contiguous buffer of UInt8 instead puts us in a world where if you pass an UMRBP, the access in untyped, but if you pass any other collection of UInt8, the access is typed as UInt8. To my mind, this blurs the thinking around safety much more than standardizing an API with and unsafe argument does; access to untyped memory is always unsafe, because you are poking a hole in the type system. Taking a UMRBP makes that explicit.
CC @Andrew_Trick who may have more to say on the subject.
