Will SwiftNIO adapt to the new Combine framework?

This year's WWDC has introduced this new async framework Combine (https://developer.apple.com/documentation/combine), and wonder if NIO will adapt to it similar to last year's Network.framework? (SwiftNIO and Network.framework)

Asking because I think it'll be (potentially) particularly useful for server frameworks like Vapor which currently heavily rely on flatMaps

4 Likes

Great question!

Short version: Combine and SwiftNIO are a natural fit so yes, we will build tools to make them work together really well without much boilerplate. Starting probably with a simple ChannelHandler that will surface the values that flow through the ChannelPipeline as a Publisher. Best news is you can get started with that today!


Longer version:

The great thing is that Combine is a solution to a problem that libraries built with SwiftNIO have been struggling with: streaming data in a user-friendly way. SwiftNIO is a great way to do bi-directional streaming whilst retaining full control over more low-level network details. That makes SwiftNIO a great tool to develop many many networking protocols. At the end of the day however not every user who wants to just use a network library (for example an HTTP library) that happens to be implemented with NIO should be required to learn all about SwiftNIO's ChannelPipeline and ChannelHandlers in order to do something as common as issuing a HTTP request. That's why we always say 'SwiftNIO is a library to write networking libraries and applications'.

So far, higher-level libraries built in SwiftNIO fell into two cases:

  1. The ones where it's quite straightforward to develop a user-friendly API which are the ones that work 'one shot' (have unary requests/responses). For example imagine an HTTP request where the user wants the HTTP body back in one shot. Defining such an API is easy: func get(url: URL) -> EventLoopFuture<Data> for example.
  2. The cases where a 'one shot' call is not good enough because you need to stream the data because it might be too large to fit into memory in its entirety or might simply need to be processed bit by bit. Whilst these things are very easy to deal with in the NIO layer, finding a good user facing API for this was not. In those cases, library authors had to always invent an ad-hoc streaming abstraction for example with delegates or callbacks. Whilst that works, every single time things like the API and important back-pressure need to be reinvented. An example can be seen in the HTTP client proposal.

"Combine" does solve the 'streaming values over time' problem once and for all and that is great news for everybody, not just people who want to create a nice API for something built with SwiftNIO.

The other thing that I should mention is that the story for SwiftNIO & Combine is slightly different to SwiftNIO & Network.framework however. For Network.framework, SwiftNIO needed to add new fundamental core capabilities (like Channels, Bootstraps, and EventLoops that work with Network.framework). For Combine, everything is much simpler because Combine is a great too to surface something that's written with SwiftNIO in a good user-facing API. Fundamentally, Network.framework was a new way for SwiftNIO to use the system whilst Combine is a new way for libraries (using SwiftNIO or not) to design APIs. So in Network.framework's case the NIO team had to provide support whilst in Combine's case SwiftNIO might provide common tools but you can start NIO & Combine together today and together we can find common pieces that are used to bridge NIO & Combine and add them into SwiftNIO to save higher-level library developers for solving the same problems over and over again.

Likely the first thing I'd like to see in SwiftNIO itself is something akin to RequestResponseHandler which is a generic solution to make simple func doRequest(_ request: Request) -> EventLoopFuture<Response> kind of APIs out of a SwiftNIO ChannelPipeline. For Combine, we could start with a ChannelHandler that can do the same but not only for unary request types but for things that support streaming, like for example HTTP request/response bodies.

7 Likes

Awesome and exciting answer - thanks a lot and I'll share with the Vapor community :D

1 more question - will Combine be open source and available on Linux? Didn't see any mention about that.

11 Likes

CC @millenomi

3 Likes

I have only looked very briefly but it seems like Combine might solve some of the same problems as Reactive Streams?

If so, we would certainly want this on Linux as this kind of abstraction can be very powerful for composing asynchronous event processors.

5 Likes

W/o knowing too much about Combine specifically, reactive "UI streams" I've seen are not very well suited for server processing because they don't usually provide the means for back pressure. Ask your local Node.js streams expert ;-)

This is IMO a nice one: https://github.com/substack/stream-handbook

I'd like to see something like Node v3 streams being supported as a higher level API to NIO (i.e. eventually port Noze to NIO if I ever find the time ;-) ).

Just IMO of course ;-)

Backpressure is supported in Combine :) one of the initial use-cases that prompted the project was async IO stuff. Aside from that it's also important on iOS due to heap size limitations.

10 Likes

Sounds cool, need to look more into this I suppose! :-) (and I generally agree, stuff which makes stuff scale on the server is often equally important on small devices, e.g. streaming parsers etc).

1 Like

No doubt the SwiftNIO people are aware as they seem to be familiar with Netty, but there has been a Reactive Netty library available for some time. So hopefully, Combine and SwiftNIO will be like peanut butter and chocolate :slight_smile:

1 Like

Combine is indeed based on the Reactive Streams spec, with a few small modifications that we felt were important.

17 Likes

I am a bit disappointed that Apple copied reactive streams here instead of going beyond.
As mentioned in another thread, I would have liked to see the use of Result in the API instead of different callbacks for success, failure and finished.
Also, I hoped an Apple Framework would have deviated wrt naming things: Subject is just not approachable.

I don't quite understand this concern. Subject is not a term in the Reactive Streams API, are you thinking of Rx?

1 Like

Wouldn't a natural place to start be API to create a Publisher from an EventLoopPromise?

Great question about Result. We do use it in places where it makes sense (e.g. Future), but for Subscriber itself we felt it was the wrong shape. Result strongly implies a single value, where as the API of Subscriber reflects the higher level contract: 1 subscription, many values, 1 completion (finished/failure). The most common operation in long-running streams is going to be handling the potentially infinite amount of input received, whereas a completion only happens once. Therefore, we believe that the necessity to unwrap the result on each input was actually counter-productive to understanding the broader picture.

With respect to Subject, we went through a lot of discussion on this but, in the end, decided to follow the Swift naming guidelines and stick with the term of art. Other names like Property have other meanings in Swift, so did not seem to be as appropriate.

13 Likes

Did you consider supporting the exactly one value or error use case? This is a very common use case (network requests, database query, etc) where Result would make sense.

Since we’re talking about Combine, I am also curious why you introduced a single Scheduler protocol and then provided one concrete type ImmediateScheduler which will fatalError on many of the scheduling APIs. This design feels very much like a footgun to me. I’m sure you had reasons though. What are they?

Isn't this what https://developer.apple.com/documentation/combine/publishers/once is for?

Yes, we have Future which is a Publisher that produces exactly one value. We also have other flavors of that pattern including Optional, Once, Just, Fail, Empty, and Deferred.

With respect to ImmediateScheduler, I think we should make those fatalErrors into just... immediate scheduling. I'll file a bug for ourselves.

6 Likes

Oh, I missed these when scanning the docs. Not sure how. Do any of these suspend side effects used to produce the value until a subscription happens? Or are the all Future-like in that they represent a side effect that has already been initiated?

Why should ImmediateScheduler conform to a protocol that specified the ability to schedule after if it isn’t going to implement that requirement? That’s what I don’t understand.

Looks like Deferred waits for subscription.