Lack of general I/O stream protocols

Using Swift
,
1

Swift is lacking general-purpose byte-oriented data streams, e.g. equivalents of the venerable java.io.Stream classes, C++'s iostreams [ick], Python's StreamReader... Instead we have a hodgepodge of stuff. These are just the ones I'm aware of:

  • FoundationEssentials' InputStream and OutputStream, which are crude wrappers around older Objective-C classes. They use unsafe buffer pointers in their API, don't throw errors, and have a lot of other design problems. Honestly this API sucked even in 2001.
  • Foundation's URLSessionDataTask (another Obj-C wrapper) can stream HTTP but the API is based on callbacks scheduled on dispatch queues OperationQueues.
  • SystemPackage has FileDescriptor, which also uses unsafe pointers and is limited to file I/O.
  • BinaryParsing has a modern Span-based API. I love this library but it focuses exclusively on reading sequential in-memory data.
  • SwiftNIO of course has a ton of stream stuff, but it's a rather heavyweight library that's AFAIK oriented toward servers, so I haven't investigated it.

All of them have incompatible APIs. For complicated reasons my project is currently using four of these, and I keep having to write glue code to adapt one to the other.

It seems like a good idea to have some core protocols for data streams in the standard library, and perhaps some basic implementations, like streams to read/write in-memory buffers and a function to copy data from one stream to another. Stream implementations in other packages could then adopt these protocols, making life easier for their users.

The core requirements are pretty minimal; basically func read(into: inout RawOutputSpan) async throws for input streams, and func write(from: RawSpan) async throws -> Int for output. Do they even need close methods or will deinit do?

I'm sure nailing down these protocols would take months of bikeshedding, but it would be worth it!

PS: This is starting to sound like a pitch, but I'm not at that point yet so I'm putting this in Using Swift for now.

6 Likes
[Second review] SE-0516: `Iterable`
2

Suggestion noted and, it turns out, already acted on :grin:

We're not quite far enough along to be ready to formally pitch this yet, but here's a peek at something we've been tinkering with

One thing that will be immediately apparent is that this is very similar to the Iterable proposal currently being debated. We're watching the result of that discussion closely, since many of the same issues apply. One perspective you can take on it is that this is to AsyncSequence what Iterable is to Sequence.

The design space for streaming is deceptively complex; everyone wants different things from it: allocation-free for Embedded Swift, copy-minimizing and compatible with kernel-side buffers for high performance servers, priority donation compatible for iOS, ergonomic to use, flexible and generic enough for generality, etc etc...

26 Likes
3

I skimmed the draft and I like that it includes a survey of other language ecosystems. It isn’t included but I am curious if the recent I/O redesign in Zig’s standard library has been something the team is watching as well?

My other immediate thought is questioning whether we actually need 4 protocols for implementers to consider because couldn’t the callee owned version be implemented as a wrapper over caller owned?