These libraries do a number of similar common tasks for clients and servers
provide a higher level API compared to SwiftNIO
server: handles incoming requests, dispatching to some handler that can produce a response
client: handles submission of a request and dispatching to some way of handling the response
There are a number of different use cases for both client and server-
raw HTTP request and response
serialization/de-serialization of request/response via something like Codable
streaming/multi-part/flushable request/response
HTTP/1
HTTP/2
HTTP/2 multiplexing?
The purpose of this thread is to discuss the viability of a shared generic HTTP client/server library to provide high quality implementations that can be used across the various Swift frameworks and to prevent the need for smaller projects to re-implement the same functionality.
I think something like this is definitely viable and I'd love to see it.
While the bulk of a NIO-based HTTP implementation is well-defined at this point, there are some non-trivial decisions that must be made. One that comes to mind immediately is how to handle cookies. Making these decisions won't be easy, but I think the end result will be a great asset to SSS package makers. Reducing double work for framework maintainers gives them more time to focus on making cool stuff.
Given the importance of such a package, I think it would make sense to pitch the client and server separately--client first. There's likely to be a lot of discussion. For a first HTTP client pitch, I think defining the public API for the HTTPClient class (including any related HTTPRequest / HTTPResponse types) and example usages in real-life code would be great.
Ok. IMO the most important discussion for a client is around what HTTPRequest / HTTPResponse types look like.
In smoke-http, we are working towards these being related to codable types rather than interacting with the raw parts of the http request and response directly. So essentially our "HTTPRequest" looks like-
So the philosophical question is do we want a generic client to provide this high an abstraction (not necessarily exactly this but this level of abstraction) over raw HTTP, just a more direct representation of HTTP request and responses that such abstractions can be built on top of or both.
Hey, this is really great to see this being started!
However, I think we should look at a layered approach and start with a layer on top of NIO which handles HTTP requests in quite a general form. In other words I think we should tackle the problem bottom up.
Eventually for example it would be nice if we could use this effort to layer Foundation's URLSession on Linux on top of this whose API supports a lot of features, including body streaming.
To be able to speak HTTP in quite a general form however we need to support pretty much anything that HTTP supports which does include streaming requests and responses. Further down the line we can then add another layer that does (like smoke-http) lots of nice things around encoding/decoding messages that the user sends around.
But thinking about file uploads, downloads but also things like server-sent events, long poll, etc. we need to be able to stream both requests and responses so I don't think we can define a HTTPRequest/Response type that contains the full body, that would imply that we have enough memory to hold a full request/response in memory which is not practical for all use-cases and it would be a shame if everybody would need to drop down to NIO as soon as they need to stream.
I would think for the first SSWG HTTP client package we should collect what features are really required and implement those well and general enough for being able to layer other libraries on top.
Non-trivial features I think we should discuss are
request/response body streaming
cancellation of requests
cookie handling
redirects
I think a relatively low-level package would also be most useful to the whole SSWG as everybody can then refine those APIs in a way that fits well with their existing frameworks. I would imagine that Smoke, Kitura, Perfect and Vapor have different preferences in how exactly an API should look but I'm sure they would all love to be able to easily sent HTTP/1 & 2 requests, have the cookies dealt with and also being able to stream bodies if necessary.
I would really like to see this made to a long term goal/plan with everything NIO related. Starting from the bottom and working up. What I means is that we should eventually build out pure-swift implementations of the full OSI or IP or whatever-Internet-model layers.
This could lead to swift even being used as the backend for router/modem utilities (open-wrt or dd-wrt) in the future. It might even be used in an nginx/apache replacement in the future. Who knows!? I hope this becomes a long term plan (Even if long means years).
I agree. I think this client should aim to support as much of the HTTP spec as possible with no extra frills. That gives us freedom to build more opinionated packages on top that add support for things like Codable.
Vapor's HTTPBody type supports streaming bodies as a special case. If the request or response specifies chunked transfer encoding, HTTPBody is initialized with an HTTPChunkedStream type that can be later used to consume the streaming data. It's not a perfect solution, but it's worth considering at least.
It feels like we are heading more towards a HTTPSession that has a defined lifecycle with different capabilities in the different stages of that lifecycle. At least for HTTP1-
initialization: the ability to set the path, request headers etc
upload: the ability to submit over time zero, one or many chunks of data as part of the request body
download: the ability to retrieve response headers and over time (via a delegate/stream?) zero, one or many chunks of data from the response body
Something like this would also fairly closely align with Foundation's URLSession as @johannesweiss previously suggested would be ideal.
Totally agree, I think this could be a great approach.
If others agree (cc @server-work-group), next step would be to submit a proposal outlining key parts of the public API for HTTPSession and maybe some additional details about how it would work internally.
Thanks @kylebrowning, I haven't had any time to look into this. If you have some time to look into creating a POC to serve as a starting point for the proposal on this, feel free to go ahead!
heya, the @server-workgroup has been discussing and working towards one over the last few weeks and [a client] was just pitched in Generic HTTP Client Library pitch. i'd love to see us combine the efforts
