Just following up on this topic, as of Swift 6.4 there's still a few small holes to plug if you want to use OutputRawSpan for serializing arbitrary types into a binary representation.
For the missing Data.init(rawCapacity:building) we ended up "polyfilling" it with:
extension Data {
public init<E: Error>(
rawCapacity size: Int,
building initializer: (inout OutputRawSpan) throws(E) -> Void
) throws(E) {
precondition(size >= 0, "size must be non-negative")
guard size > 0 else { self = Data(); return }
let raw = UnsafeMutableRawBufferPointer.allocate(byteCount: size, alignment: 1)
var out = OutputRawSpan(buffer: raw, initializedCount: 0)
do {
try initializer(&out)
} catch {
// The closure throwing must not leak the allocation.
_ = out.finalize(for: raw)
raw.deallocate()
throw error
}
let written = out.finalize(for: raw)
precondition(written == size, "declared \(size) bytes but wrote \(written)")
self = Data(
bytesNoCopy: raw.baseAddress!,
count: written,
deallocator: .custom { pointer, _ in pointer.deallocate() }
)
}
}
This way you can serialize while minimizing copying memory. This is how we're using it to serialize variable (compact) integers:
let varInt = VarInt(Int.max)
let data = try Data(rawCapacity: varInt.dataSize) { out in
try varInt.encode(into: &out)
}
An alternative way of achieving something similar is just using withTemporaryAllocation() but that also needs a little extension, especially if you want to keep unsafe code contained.
extension Data {
public init(_ span: borrowing OutputRawSpan) {
self = span.withUnsafeBytes { pointer in
guard let base = pointer.baseAddress else {
return Data()
}
return Data(bytes: base, count: pointer.count)
}
}
}
Now you can do something like the following – again to serialize a variable integer:
let data = try withTemporaryAllocation(byteCount: varInt.dataSize, alignment: 1) { span in
try varInt.encode(into: &span)
return Data(span)
}
Finally, this last extension we found useful for appending Data to an OutputRawSpan when serializing:
extension OutputRawSpan {
/// Appends the raw bytes of a Data instance directly into the buffer layout.
public mutating func append<D: ContiguousBytes>(contentsOf data: D) {
data.withUnsafeBytes { sourceBuffer in
guard !sourceBuffer.isEmpty else { return }
// Gain access to the underlying storage pointer and the layout's active count
withUnsafeMutableBytes { targetBuffer, initializedCount in
// Precondition to protect against runtime buffer overflows
precondition(targetBuffer.count - initializedCount >= sourceBuffer.count, "Out of bounds: OutputRawSpan has insufficient capacity.")
// Point directly to the first uninitialized byte in the memory block
let writePointer = targetBuffer.baseAddress! + initializedCount
// Access the source data's contiguous memory buffer safely
data.withUnsafeBytes { sourceBuffer in
// Perform an optimized memcpy directly into the buffer layout
writePointer.copyMemory(from: sourceBuffer.baseAddress!, byteCount: sourceBuffer.count)
}
// Explicitly step the internal tracking counter forward
initializedCount += sourceBuffer.count
}
}
}
}
For instance we use this to serialize a Script consisting of a sequence of operations and a potential unparsable part which is just bytes:
public func encode(into out: inout OutputRawSpan) throws {
for op in operations {
try op.encode(into: &out)
}
out.append(contentsOf: unparsable)
}
Hope someone finds some of this useful. Please provide feedback if you have any, we've been struggling with binary parsing and serialization ever since we started Swift Bitcoin even implementing our own library and helpers before falling back to Swift Binary Parsing and OutputRawSpan.