[Proposal] Compression Library

Community Showcase
21

For now I'm leaning towards targets instead of traits since we have C targets that end users may not want to pull in, and as far as I'm aware we can't gate a target from being compiled with traits. Code might also start to pile up as we add support for different algorithms, and different targets are easier to handle.
InlineArrays can be de/compressed via the Span API.
Typed throws, yes, there's some of it going around already although I haven't decided on the final structure/hierarchy of the errors yet.
The aim is to be supported on any platform Swift runs, embedded included.
Regarding the minimum Swift version, that's TBD. As you noticed I'm using a toolchain from main otherwise the concrete Streaming{De}Compressors can't be marked ~Copyable while associating them to the CompressionAlgorithm (i.e this would not be possible

protocol CompressionAlgorithm {
    associatedtype StreamingCompressor: ~Copyable
}

) and the streaming compressor really should be ~Copyable. So the options would be to either wait for the feature or to redesign the API around not having it, but since this is a greenfield library that already uses all of the modern bells and whistles Swift now proposes (or will in the near future) I'd rather we do things well than in a rush.

1 Like
22

just FYI, you can use
.enableExperimentalFeature("SuppressedAssociatedTypes")
which seems to work with 6.3 (release version)

the details around what does and does not work with this, and how the final "official" feature will differ, are kind of hard to come by - and I certainly have no idea ; )

3 Likes
23

Emphasis mine; we can do that, as seen in my networking library's Package.swift (as I also build it for embedded):

Target.target(
        name: "Destiny",
        dependencies: [
            .product(name: "CEpoll", package: "CEpoll", condition: .when(platforms: [.linux])),
            .product(name: "Logging", package: "swift-log", condition: .when(traits: ["Logging"])),
            .product(name: "MediaTypes", package: "swift-media-types", condition: .when(traits: ["MediaTypes"])),
            .product(name: "UnwrapArithmeticOperators", package: "swift-unwrap-arithmetic-operators"),
            .product(name: "VariableLengthArray", package: "swift-variablelengtharray"),
            .product(name: "SwiftCompressionUtilities", package: "swift-compression", condition: .when(traits: ["Compression"]))
        ]
    ),

I was more interested in if we, mainly the library itself, would use inline arrays as the buffer when compressing/decompressing in native Swift code (or when communicating via shims) instead of using the heap.

Edit: I guess withUnsafeTemporaryAllocation would be more pragmatic when working with the shims.

24

I see. I've tried using InlineArray in the Deflate streaming de/compressors and so these thoughts are based on this exclusively.
I went for [UInt8] because of these reasons:

  • With InlineArray, buffer size is not really configurable: the only way we could do that is by having StreamingCompressor<let BufferSize: Int> which is just bad API design in my opinion;
  • This means we'd have to decide on a fixed size for everybody. I initially went for 32KB which is a conservative choice (matches zlib's internal window size: this means the loop rarely needs more than one iteration), but whatever we pick is baked into the type: too small and performance suffers for bigger workloads; too big and we waste memory. A runtime-configurable size is probably a better fit. We can also extend the DeflateConfiguration to accept a buffer size parameter by the user and reserveCapacity with that.
  • As you probably know, the z_stream's internal state already allocates on the heap, so we wouldn't even be reaching the goal of being heap free, and couldn't use this on embedded devices without an allocator even if we wanted to.

Having said that, I'm keeping my mind open and if you or someone with stronger opinions (than mine) makes a good case for it, we can consider switching.
I do think InlineArray can be useful for this library and I'm happy to resort to it in other cases as well, such as LZ4 where we can operate on fixed-size blocks.

Regarding the shims interaction, I've opted for SafeInteropWrappers and am going through MutableSpan which is how modern Swift should interact with C.

1 Like