Reading and decompressing a bunch of floats from a FileDescriptor, minimizing copies

Using Swift
1

I'm working on reading fairly large data from a file. It’s compressed in the file, I need to load it from a FileDescriptor, decompress it, and then interpret it as an array of some type, like UInt16 or Float.

So far, I can load and decompress the data into a Data:

let tileOffset = tileInfo.tileOffsets[inTileIdx]
let tileByteCount = tileInfo.tileByteCounts[inTileIdx]
let buf = UnsafeMutableRawBufferPointer.allocate(byteCount: Int(tileByteCount), alignment: MemoryLayout<UInt8>.alignment)
defer { buf.deallocate() }
let readCount = try self.reader.fd.read(fromAbsoluteOffset: tileOffset, into: buf)
let data = Data(bytesNoCopy: buf.baseAddress!, count: readCount, deallocator: .none)
let uncompressedData = try (data[2...] as NSData).decompressed(using: .zlib)

Using NSData’s decompress method requires me to strip off the first two bytes of the data (which is using deflate).

I feel like there’s some cool trick with memoryReboundTo or some such that would let me create a nice, managed array of Floats from uncompressedData . But I’m not sure exactly how.

I also am not sure about the deferred deallocation, which I probably need to manage more explicitly (like only do it if an exception is thrown).

I'm not sure if the data[2...] subrange creates a copy, but I don’t think so, right?

But most importantly, can I tell Swift to treat uncompressedData as an array of Float?

2

I think your best bet here would be to combine:

  • Array.init(unsafeUninitializedCapacity:initializingWith:)

  • With a decompression API that lets you target a buffer.

For the latter, if you’re working on an Apple platform then Compression framework seems like an avenue worth exploring.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

1 Like
3

I must be really dumb because I sure don't see how to do this. This is what I tried so far, and it's incomplete, but it crashes on the .subdata call with EXC_BREAKPOINT (code=1, subcode=0x1918c38fc). dataWithoutHeader is the deflate-compressed data minus the first two bytes, which cause NSData.decompressed() to fail.

		let pageSize = 1024
		let arr = try [Float](unsafeUninitializedCapacity: sampleCount)
		{ buffer, initializedCount in
			print("Count: \(initializedCount)")
			var index = 0
			let inputFilter = try InputFilter(.decompress, using: .zlib)
			{ (length: Int) -> Data? in
				let rangeLength = min(length, dataWithoutHeader.count - index)
				let subdata = dataWithoutHeader.subdata(in: index ..< index + rangeLength)
				index += rangeLength

				return subdata
			}
			
			var dd = Data()
			do
			{
				while let page = try inputFilter.readData(ofLength: pageSize)
				{
					dd.append(page)
				}
			}
			
			catch
			{
				print("Error: \(error)")
			}

            // Presumably something here with dd to initialize `buffer, but I have no idea what.
		}

The problem I see with this approach (other than I don't know how to actualy do it) is that there’s still an extra copy. There’s the copy that happend during decompression, which obviously can't be avoided. But then in the decompression loop there’s a copy as I append each decompressed page (BTW I don't know how to pick a page size).

In C, I'd just do this:

float* data = (float*) decompressedData;

Surely there’s an equivalent in Swift, I just can't figure out the right incantation of Unsafe[Mutable][Raw][Buffer][Pointer] to do it. Honestly, I hate this aspect of Swift, or at least the documentation around it. Each one is documented in isolation, and there’s no overarching document with nice images and tables to show how each Unsafe* thing relates to the other, and how to do all the typical things the API enables. The lack of example code is a significant weakness of all of Apple's documentation.

Even Array.init(unsafeUninitializedCapacity:initializingWith:) really needs an example to show how to write the closure.

But all in all, this way seems to make it virtually impossible to decompress an array of floats without extra copies.

UPDATE: The crash seems to be due to accessing a subrange of a subrange of the original compressed data, eg:

let dataWithoutHeader = compressedData[2...]

Not sure why that would crash.

UPDATE2: Ah, I see there’s a buffer-based decompression. Let me try that. The front page of the docs, under “Topics,” has “Compressing and decompressing data with input and output filters” and “Compressing and decompressing files with stream compression,” the former of which seemed more appropriate for what I was doing. (Another shortcoming of Apple’s docs: They require reading all of it rather than giving a clear path to the thing you need.)

4

After watching some videos and trying again, decompression isn't working, but I think I have the right code.

let samples = try [Float](unsafeUninitializedCapacity: sampleCount)
{ buffer, initializedCount in
	print("Count: \(initializedCount)")
	try compressedData.withUnsafeBytes<UInt8>
	{ inCompressedBytes in
		let destBufferSize = buffer.count * MemoryLayout<Float>.size
		let decompressedSize = compression_decode_buffer(buffer.baseAddress!, destBufferSize,
														inCompressedBytes, compressedData.count,
														nil, COMPRESSION_ZLIB)
		print("Actual decompressed size: \(decompressedSize), destBufferSize: \(destBufferSize)")
	}
	initializedCount = sampleCount
}

The code prints

Count: 0
Actual decompressed size: 46510, destBufferSize: 1048576

(When I decompress the same data using NSData.decompresed(), it works correctly.)

I have some issues with this code. For one, I get this warning for compressedData.withUnsafeBytes: 'withUnsafeBytes' is deprecated: use withUnsafeBytes(_: (UnsafeRawBufferPointer) throws -> R) rethrows -> R instead. I don’t know why it’s choosing the deprecated method here, and the only alternatives pass a UnsafeRawBufferPointer to the closure, which I can get by being explicit with the closure’s signature, but then compression_decode_buffer() because it wants an UnsafePointer<UInt8>, which is provided by the deprecated method.

A slightly updated version:

		let samples = try [Float](unsafeUninitializedCapacity: sampleCount)
								{ buffer, initializedCount in
									print("Count: \(initializedCount)")
									try compressedData.withUnsafeBytes<UInt8>
									{ (inCompressedBytes: UnsafeRawBufferPointer) -> Void in
										let destBufferSize = buffer.count * MemoryLayout<Float>.size
										let decompressedSize = compression_decode_buffer(buffer.baseAddress!, destBufferSize,
																						inCompressedBytes.baseAddress!, inCompressedBytes.count,
																						nil, COMPRESSION_ZLIB)
										print("Actual decompressed size: \(decompressedSize), destBufferSize: \(destBufferSize)")
									}
									initializedCount = sampleCount
								}

But it still doesn't decompress properly. destBufferSize is 1048576, but the returned decompressed size is 46510.

5

I must be really dumb …

Or, alternatively, this is quite hard. There’s a reason I didn’t include a snippet of working code in my previous post!

But it still doesn't decompress properly.

I’m not sure I followed the full story here, but I believe you’ve boiled this down to “How can I use Compression to decompress to a fixed-size buffer?” Is that right?

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

2 Likes
6

Yeah, I made another post on the Apple dev forums specifically about compression_decode_buffer(). Well, I hope that’s all that’s wrong now. Not at all sure I'm initializing Array correctly.

7

This is somewhat tangential to the ongoing discussion, but note that Data is not zero-indexed. Somewhat confusingly, the Data type uses integer indices, but is not always zero-based (i.e startIndex != 0). So, you could crash here as you are assuming indices that are not actually valid. If you want to offset by 2, you have to do compressedData[(compressedData.startIndex + 2)...], or even to be completely accurate compressedData[compressedData.index(compressedData.startIndex, offsetBy: 2)...], or compressedData.dropFirst(2) etc.

8

There are WWDC talks which explain how to use them, but yes, overall the documentation is kind of weak. I think the reason that is the case is because we're reluctant to present the unsafe APIs as a solution for this kind of problem. It's very much an experts-only tool and we don't want to advertise it like everybody should immediately reach for UnsafeBufferPointer.

Long-term, Span<T> and RawSpan (pitch) are designed to support these kinds of things in a safe, convenient way. I expect there will probably be a greater investment in documenting how to use them for processing binary data.

3 Likes
9

Here's a small example of compressing and decompressing in memory using the low-level compression_xxx_buffer operations, showing how to do it without copies and preserving data through the round-trip, which ought to be adaptable to your use case:

Summary 
import Darwin
import Compression

// Wrappers to let us use BufferPointers in the implementation.
func compress<T: BitwiseCopyable>(
    _ source: UnsafeBufferPointer<T>,
    _ destination: UnsafeMutableBufferPointer<UInt8>,
    using algorithm: compression_algorithm
) -> Int {
    compression_encode_buffer(
        destination.baseAddress!, destination.count,
        source.baseAddress!, source.count * MemoryLayout<T>.stride,
        nil, algorithm
    )
}

func expand<T: BitwiseCopyable>(
    _ source: UnsafeBufferPointer<UInt8>,
    _ destination: UnsafeMutableBufferPointer<T>,
    using algorithm: compression_algorithm
) -> Int {
    let bytes = compression_decode_buffer(
        destination.baseAddress!, destination.count * MemoryLayout<T>.stride,
        source.baseAddress!, source.count,
        nil, algorithm
    )
    // This only makes sense if we know _a priori_ how large the expanded data
    // is going to be, but generally speaking we should know that, and it should
    // be a whole number of `T`s. If we're doing something more like streaming,
    // then we would instead expand into a buffer of bytes and interpret some
    // window of those bytes as a whole number of `T`s, but that's a different
    // API.
    precondition(bytes % MemoryLayout<T>.stride == 0)
    return bytes / MemoryLayout<T>.stride
}

// Generate some data to compress (poorly, but it illustrates the point):
let input = (0 ..< 65536).map {
    sin(Float($0) * .pi / 65536)
}

// Compress the data:
let compressed = [UInt8](unsafeUninitializedCapacity: 4*65536) {
    compBuf, compressedSize in
    compressedSize = input.withUnsafeBufferPointer {
        inBuf in
        compress(inBuf, compBuf, using: COMPRESSION_ZLIB)
    }
}

// Expand the data:
let output = [Float](unsafeUninitializedCapacity: 65536) {
    outBuf, count in
    count = compressed.withUnsafeBufferPointer {
        compBuf in
        expand(compBuf, outBuf, using: COMPRESSION_ZLIB)
    }
}

// Validate:
precondition(input == output)

How was the data that you're decompressing originally compressed?

3 Likes
10

I wrote the following before I saw Steve’s response, but I decided to post it anyway because it shows that Compression framework can decode data compressed with NSData.

Summary 
import Foundation
import Compression

func compression_decode_bufferq(
    _ dst: UnsafeMutableRawBufferPointer,
    _ src: UnsafeRawBufferPointer,
    _ algorithm: compression_algorithm
) -> Int {
    compression_decode_buffer(
        dst.baseAddress!,
        dst.count,
        src.baseAddress!,
        src.count,
        nil,
        algorithm
    )
}

func main() throws {
    let input = Data("Hello Cruel World!".utf8)
    
    // First do the round trip with `NSData`.
    
    let compressed = try (input as NSData).compressed(using: .zlib) as Data
    let decompressed1 = try (compressed as NSData).decompressed(using: .zlib) as Data
    print(input == decompressed1)

    // Then do the decompression with Compression framework.
    
    var decompressed2 = Data(repeating: 0, count: input.count + 1)
    let bytesDecompressed = compressed.withUnsafeBytes { src in
        decompressed2.withUnsafeMutableBytes { dst in
            compression_decode_bufferq(dst, src, COMPRESSION_ZLIB)
        }
    }
    assert(bytesDecompressed < decompressed2.count)
    decompressed2.count = bytesDecompressed
    print(input == decompressed2)
}

try main()

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

3 Likes
11

Unfortunately, video is usually the worst way to present this kind of information. It’s not random-access, and difficult to skim.

Which makes it that much easier to use incorrectly. More information, not less, is key to safety and mitigating abuse.

Well, it's embedded in a GeoTIFF file and was generated elsehwere, but it's marked as “deflate.” The data has two bytes at the start that cause NSData.decompressed() to fail, but skipping those two bytes allows it to succeed. With or without those two bytes, compression_decode_buffer() is silently failing for me (in that it returns a byte count less than expected, but not zero and afaik there’s no error reporting mechanism. I don’t even know if I can turn on debug logging).

I’ve tried to distill it down to something others can reproduce. I’ve created a gist that has two Swift Testing tests and a link to the compressed data, as well as links to the data uncompressed by various means.

Could it be some variant of COMPRESSION_ZLIB (deflateInit2(zstream,5,Z_DEFLATED,-15,8,Z_DEFAULT_STRATEGY)) that NSData.decompressed() handles but compression_decode_buffer() does not? I guess there’s no way to create a different algorithm, is there?

2 Likes
12

Could it be some variant of COMPRESSION_ZLIB … that
NSData.decompressed() handles but
compression_decode_buffer() does not?

So, you’re able to decompress the data with NSData?

That’s interesting because NSData uses Compression under the covers. Admittedly, that’s using compression_stream_process.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

13

Yep! I have to take a subrange that drops the first two bytes, but otherwise it works. I get the same result whether or not I drop the first two bytes with compression_decode_buffer().

I'll write up feedback.

14

Feedback: FB14580588

15

In testDecomp() from your gist, you need to advance the baseAddress of the source buffer by 2 in order to skip the header.
This works:

let decompressedSize = try compressedData.withUnsafeBytes/*<UInt8>*/ {
  (inCompressedBytes: UnsafeRawBufferPointer) -> Int in
  let decompressedSize = compression_decode_buffer(
    buffer.baseAddress!, destBufferSize,
    inCompressedBytes.baseAddress! +2 /*HERE*/, inCompressedBytes.count,
    nil, COMPRESSION_ZLIB)
  return decompressedSize
}

I am not sure why the generic argument to the function didn't produce an error here. It's saying the return value will be UInt8, but the closure explicitly returns an Int.

A slightly nicer way, using dropFirst():

  let decompressedSize = try compressedData.dropFirst(2).withUnsafeBytes {
    (inCompressedBytes: UnsafeRawBufferPointer) -> Int in
    compression_decode_buffer(
      buffer.baseAddress!, destBufferSize,
      inCompressedBytes.baseAddress!, inCompressedBytes.count,
      nil, COMPRESSION_ZLIB
    )
  }
16

As I mentioned above, I tried that and it didn’t make a difference.

17

I'm getting the correct number out, running this in a playground. Is there something else that is different, then?

1 Like
18

Thanks for confirming that. I can’t check it right now, but I’ll try again when I’m able.

1 Like
19

Well, I’ll be. It’s working now. I don’t know what I was doing wrong before. I tried multiple times with and without dropping the first two bytes.

I’m sorry for the wasted time, and thank you greatly for verifying that was the issue!

2 Likes
20

Glad it's working now!

1 Like