HTTP API Sketch v2


(Johannes Weiss) #1

Hi,

First of all, thanks very much to Helge for carefully analysing (and implementing) the first API sketch. As promised, I reworked it a bit.

Changes:
- extracted HTTPResponse as its own value type (so it could also be used for a client)
- added Helge's suggestions:
  * proper backpressure support
  * being able to ignore further parts of the HTTP body (stop parameter)

If I forgot something important, please let me know. I can't promise that I'll address it before the meeting as I'm in another office tomorrow for meetings and probably won't be able to attend either.

Please find everything below...

Cheers,
  Johannes

--- SNIP ---
/* a web app is a function that gets a HTTPRequest and a HTTPResponseWriter and returns a function which processes the HTTP request body in chunks as they arrive */

public typealias WebApp = (HTTPRequest, HTTPResponseWriter) -> HTTPBodyProcessing

public struct HTTPRequest {
  public var method : HTTPMethod
  public var target : String /* e.g. "/foo/bar?buz=qux" */
  public var httpVersion : HTTPVersion
  public var headers : HTTPHeaders
}

public struct HTTPResponse {
  public var httpVersion : HTTPVersion
  public var status: HTTPResponseStatus
  public var transferEncoding: HTTPTransferEncoding
  public var headers: HTTPHeaders
}

public protocol HTTPResponseWriter: class {
  func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send an HTTP `100 Continue` */

  func writeResponse(_ response: HTTPResponse)

  func writeTrailer(key: String, value: String)

  func writeBody(data: DispatchData) /* convenience */
  func writeBody(data: Data) /* convenience */
  func writeBody(data: DispatchData, completion: @escaping (Result<POSIXError, ()>) -> Void)
  func writeBody(data: Data, completion: @escaping (Result<POSIXError, ()>) -> Void)

  func done() /* convenience */
  func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
  func abort()
}

public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void /* the Bool can be set to true when we don't want to process anything further */

public enum HTTPBodyProcessing {
   case discardBody /* if you're not interested in the body */
   case processBody(handler: HTTPBodyHandler)
}

public enum HTTPBodyChunk {
  case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a new bit of the HTTP request body has arrived, finishedProcessing() must be called when done with that chunk */
  case failed(error: HTTPParserError) /* error while streaming the HTTP request body, eg. connection closed */
  case trailer(key: String, value: String) /* trailer has arrived (this we actually haven't implemented yet) */
  case end /* body and trailers finished */
}

public struct HTTPHeaders {
  var storage: [String:[String]] /* lower cased keys */
  var original: [(String, String)] /* original casing */
  var description: String

  subscript(key: String) -> [String]
  func makeIterator() -> IndexingIterator<Array<(String, String)>>

  init(_ headers: [(String, String)] = [])
}

public typealias HTTPVersion = (Int, Int)

public enum HTTPTransferEncoding {
  case identity(contentLength: UInt)
  case chunked
}

public enum HTTPResponseStatus {
  /* use custom if you want to use a non-standard response code or
     have it available in a (UInt, String) pair from a higher-level web framework. */
  case custom(code: UInt, reasonPhrase: String)

  /* all the codes from http://www.iana.org/assignments/http-status-codes */
  case `continue`
  case switchingProtocols
  case processing
  case ok
  case created
  case accepted
  case nonAuthoritativeInformation
  case noContent
  case resetContent
  case partialContent
  case multiStatus
  case alreadyReported
  case imUsed
  case multipleChoices
  case movedPermanently
  case found
  case seeOther
  case notModified
  case useProxy
  case temporaryRedirect
  case permanentRedirect
  case badRequest
  case unauthorized
  case paymentRequired
  case forbidden
  case notFound
  case methodNotAllowed
  case notAcceptable
  case proxyAuthenticationRequired
  case requestTimeout
  case conflict
  case gone
  case lengthRequired
  case preconditionFailed
  case payloadTooLarge
  case uriTooLong
  case unsupportedMediaType
  case rangeNotSatisfiable
  case expectationFailed
  case misdirectedRequest
  case unprocessableEntity
  case locked
  case failedDependency
  case upgradeRequired
  case preconditionRequired
  case tooManyRequests
  case requestHeaderFieldsTooLarge
  case unavailableForLegalReasons
  case internalServerError
  case notImplemented
  case badGateway
  case serviceUnavailable
  case gatewayTimeout
  case httpVersionNotSupported
  case variantAlsoNegotiates
  case insufficientStorage
  case loopDetected
  case notExtended
  case networkAuthenticationRequired
}

public enum HTTPMethod {
  case custom(method: String)

  /* everything that http_parser.[ch] supports */
  case DELETE
  case GET
  case HEAD
  case POST
  case PUT
  case CONNECT
  case OPTIONS
  case TRACE
  case COPY
  case LOCK
  case MKCOL
  case MOVE
  case PROPFIND
  case PROPPATCH
  case SEARCH
  case UNLOCK
  case BIND
  case REBIND
  case UNBIND
  case ACL
  case REPORT
  case MKACTIVITY
  case CHECKOUT
  case MERGE
  case MSEARCH
  case NOTIFY
  case SUBSCRIBE
  case UNSUBSCRIBE
  case PATCH
  case PURGE
  case MKCALENDAR
  case LINK
  case UNLINK
}
--- SNAP ---

Here's the demo code for a simple echo server

--- SNIP ---
serve { (req, res) in
   if req.target == "/echo" {
       guard req.httpVersion == (1, 1) else {
           /* HTTP/1.0 doesn't support chunked encoding */
           res.writeResponse(HTTPResponse(version: req.version,
                                          status: .httpVersionNotSupported,
                                          transferEncoding: .identity(contentLength: 0)))
           res.done()
           return .discardBody
       }
       res.writeResponse(HTTPResponse(version: req.version,
                                      status: .ok,
                                      transferEncoding: .chunked,
                                      headers: SomeConcreteHTTPHeaders([("X-foo": "bar")])))
       return .processBody { (chunk, stop) in
           switch chunk {
               case .chunk(let data, let finishedProcessing):
                   res.writeBody(data: data) { _ in
                       finishedProcessing()
                   }
               case .end:
                   res.done()
               default:
                   stop = true /* don't call us anymore */
                   res.abort()
           }
       }
   } else { ... }
}
--- SNAP ---


(Colin Barrett) #2

Hello all,

Is writing compositional middleware a goal for this iteration of the API? I
noticed that the HTTPResponseWriter doesn't appear to allow for, e.g.
writing a single header and then passing the writer on (as one would do in,
say, an anti-CSRF middleware). Possible I'm missing something though!

-Colin

···

On Wed, Apr 5, 2017 at 1:36 PM Johannes Weiß via swift-server-dev < swift-server-dev@swift.org> wrote:

Hi,

First of all, thanks very much to Helge for carefully analysing (and
implementing) the first API sketch. As promised, I reworked it a bit.

Changes:
- extracted HTTPResponse as its own value type (so it could also be used
for a client)
- added Helge's suggestions:
  * proper backpressure support
  * being able to ignore further parts of the HTTP body (stop parameter)

If I forgot something important, please let me know. I can't promise that
I'll address it before the meeting as I'm in another office tomorrow for
meetings and probably won't be able to attend either.

Please find everything below...

Cheers,
  Johannes

--- SNIP ---
/* a web app is a function that gets a HTTPRequest and a
HTTPResponseWriter and returns a function which processes the HTTP request
body in chunks as they arrive */

public typealias WebApp = (HTTPRequest, HTTPResponseWriter) ->
HTTPBodyProcessing

public struct HTTPRequest {
  public var method : HTTPMethod
  public var target : String /* e.g. "/foo/bar?buz=qux" */
  public var httpVersion : HTTPVersion
  public var headers : HTTPHeaders
}

public struct HTTPResponse {
  public var httpVersion : HTTPVersion
  public var status: HTTPResponseStatus
  public var transferEncoding: HTTPTransferEncoding
  public var headers: HTTPHeaders
}

public protocol HTTPResponseWriter: class {
  func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send an
HTTP `100 Continue` */

  func writeResponse(_ response: HTTPResponse)

  func writeTrailer(key: String, value: String)

  func writeBody(data: DispatchData) /* convenience */
  func writeBody(data: Data) /* convenience */
  func writeBody(data: DispatchData, completion: @escaping
(Result<POSIXError, ()>) -> Void)
  func writeBody(data: Data, completion: @escaping (Result<POSIXError,
()>) -> Void)

  func done() /* convenience */
  func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
  func abort()
}

public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void /*
the Bool can be set to true when we don't want to process anything further
*/

public enum HTTPBodyProcessing {
   case discardBody /* if you're not interested in the body */
   case processBody(handler: HTTPBodyHandler)
}

public enum HTTPBodyChunk {
  case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a new
bit of the HTTP request body has arrived, finishedProcessing() must be
called when done with that chunk */
  case failed(error: HTTPParserError) /* error while streaming the HTTP
request body, eg. connection closed */
  case trailer(key: String, value: String) /* trailer has arrived (this we
actually haven't implemented yet) */
  case end /* body and trailers finished */
}

public struct HTTPHeaders {
  var storage: [String:[String]] /* lower cased keys */
  var original: [(String, String)] /* original casing */
  var description: String

  subscript(key: String) -> [String]
  func makeIterator() -> IndexingIterator<Array<(String, String)>>

  init(_ headers: [(String, String)] = [])
}

public typealias HTTPVersion = (Int, Int)

public enum HTTPTransferEncoding {
  case identity(contentLength: UInt)
  case chunked
}

public enum HTTPResponseStatus {
  /* use custom if you want to use a non-standard response code or
     have it available in a (UInt, String) pair from a higher-level web
framework. */
  case custom(code: UInt, reasonPhrase: String)

  /* all the codes from http://www.iana.org/assignments/http-status-codes
*/
  case `continue`
  case switchingProtocols
  case processing
  case ok
  case created
  case accepted
  case nonAuthoritativeInformation
  case noContent
  case resetContent
  case partialContent
  case multiStatus
  case alreadyReported
  case imUsed
  case multipleChoices
  case movedPermanently
  case found
  case seeOther
  case notModified
  case useProxy
  case temporaryRedirect
  case permanentRedirect
  case badRequest
  case unauthorized
  case paymentRequired
  case forbidden
  case notFound
  case methodNotAllowed
  case notAcceptable
  case proxyAuthenticationRequired
  case requestTimeout
  case conflict
  case gone
  case lengthRequired
  case preconditionFailed
  case payloadTooLarge
  case uriTooLong
  case unsupportedMediaType
  case rangeNotSatisfiable
  case expectationFailed
  case misdirectedRequest
  case unprocessableEntity
  case locked
  case failedDependency
  case upgradeRequired
  case preconditionRequired
  case tooManyRequests
  case requestHeaderFieldsTooLarge
  case unavailableForLegalReasons
  case internalServerError
  case notImplemented
  case badGateway
  case serviceUnavailable
  case gatewayTimeout
  case httpVersionNotSupported
  case variantAlsoNegotiates
  case insufficientStorage
  case loopDetected
  case notExtended
  case networkAuthenticationRequired
}

public enum HTTPMethod {
  case custom(method: String)

  /* everything that http_parser.[ch] supports */
  case DELETE
  case GET
  case HEAD
  case POST
  case PUT
  case CONNECT
  case OPTIONS
  case TRACE
  case COPY
  case LOCK
  case MKCOL
  case MOVE
  case PROPFIND
  case PROPPATCH
  case SEARCH
  case UNLOCK
  case BIND
  case REBIND
  case UNBIND
  case ACL
  case REPORT
  case MKACTIVITY
  case CHECKOUT
  case MERGE
  case MSEARCH
  case NOTIFY
  case SUBSCRIBE
  case UNSUBSCRIBE
  case PATCH
  case PURGE
  case MKCALENDAR
  case LINK
  case UNLINK
}
--- SNAP ---

Here's the demo code for a simple echo server

--- SNIP ---
serve { (req, res) in
   if req.target == "/echo" {
       guard req.httpVersion == (1, 1) else {
           /* HTTP/1.0 doesn't support chunked encoding */
           res.writeResponse(HTTPResponse(version: req.version,
                                          status: .httpVersionNotSupported,
                                          transferEncoding:
.identity(contentLength: 0)))
           res.done()
           return .discardBody
       }
       res.writeResponse(HTTPResponse(version: req.version,
                                      status: .ok,
                                      transferEncoding: .chunked,
                                      headers:
SomeConcreteHTTPHeaders([("X-foo": "bar")])))
       return .processBody { (chunk, stop) in
           switch chunk {
               case .chunk(let data, let finishedProcessing):
                   res.writeBody(data: data) { _ in
                       finishedProcessing()
                   }
               case .end:
                   res.done()
               default:
                   stop = true /* don't call us anymore */
                   res.abort()
           }
       }
   } else { ... }
}
--- SNAP ---

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Helge Heß) #3

Hi Johannes,

my notes on it:

- concrete types instead of protocols for HTTPRequest
  - bad for layering this on top of alternative
    implementations
  - bad because it prohibits future changes in the
    implementation (e.g. I still think it is a bad idea
    to convert everything to Strings in advance)
  - it can still come with a default imp
  - also: you could make NSURLRequest/Response support it

- maybe HTTPRequest should be called HTTPRequestHead,
  because in this case it really is what it is, it
  doesn’t even have a stream reference attached to it.

- you should probably not add ‘convenience’ to this, that
  will be done by frameworks using it in whatever they
  consider ‘convenient’
  - or it other words: does providing a `write` w/o a
    completion handler give performance benefits and
    is part of the API because of this, or is it just
    `write(..) { _ in }`. I could see that it can be a
    perf advantage (no setup of an escaping closure
    necessary), but then it should not be marked as
    ‘convenience’
  - same for Data vs DispatchData. Decide on just one?

- maybe such:

     func writeTrailer(key: String, value: String)

  should be

     func writeTrailer(key: UnsafePointer<CChar>,
                       value: UnsafePointer<CChar>)

  this also works with strings out of the box while
  allowing applications not using Strings for protocol
  data ;->

- I guess all ‘write’ functions need a completion
  callback? (including writeTrailer, writeResponse)

- I’m still a little worried about using `enum`s for
  an evolving protocol. That will look nice now but
  will fall apart in the future as we can’t extend
  them w/o breaking apps

- Talking enums. You use them for status and method,
  why not for header keys? I see _big_ performance
  advantages in that.

- Having the stream support ‘HTTPResponse’ looks like
  too high level for your specific design. You can keep
  that object and add a convenience method to put such
  on the stream.
  - also: HTTPResponse vs HTTPResponseHead[er?]

That is it for now :slight_smile:

hh

···

On 5. Apr 2017, at 19:35, Johannes Weiß <johannesweiss@apple.com> wrote:

Hi,

First of all, thanks very much to Helge for carefully analysing (and implementing) the first API sketch. As promised, I reworked it a bit.

Changes:
- extracted HTTPResponse as its own value type (so it could also be used for a client)
- added Helge's suggestions:
* proper backpressure support
* being able to ignore further parts of the HTTP body (stop parameter)

If I forgot something important, please let me know. I can't promise that I'll address it before the meeting as I'm in another office tomorrow for meetings and probably won't be able to attend either.

Please find everything below...

Cheers,
Johannes

--- SNIP ---
/* a web app is a function that gets a HTTPRequest and a HTTPResponseWriter and returns a function which processes the HTTP request body in chunks as they arrive */

public typealias WebApp = (HTTPRequest, HTTPResponseWriter) -> HTTPBodyProcessing

public struct HTTPRequest {
public var method : HTTPMethod
public var target : String /* e.g. "/foo/bar?buz=qux" */
public var httpVersion : HTTPVersion
public var headers : HTTPHeaders
}

public struct HTTPResponse {
public var httpVersion : HTTPVersion
public var status: HTTPResponseStatus
public var transferEncoding: HTTPTransferEncoding
public var headers: HTTPHeaders
}

public protocol HTTPResponseWriter: class {
func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send an HTTP `100 Continue` */

func writeResponse(_ response: HTTPResponse)

func writeTrailer(key: String, value: String)

func writeBody(data: DispatchData) /* convenience */
func writeBody(data: Data) /* convenience */
func writeBody(data: DispatchData, completion: @escaping (Result<POSIXError, ()>) -> Void)
func writeBody(data: Data, completion: @escaping (Result<POSIXError, ()>) -> Void)

func done() /* convenience */
func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
func abort()
}

public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void /* the Bool can be set to true when we don't want to process anything further */

public enum HTTPBodyProcessing {
  case discardBody /* if you're not interested in the body */
  case processBody(handler: HTTPBodyHandler)
}

public enum HTTPBodyChunk {
case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a new bit of the HTTP request body has arrived, finishedProcessing() must be called when done with that chunk */
case failed(error: HTTPParserError) /* error while streaming the HTTP request body, eg. connection closed */
case trailer(key: String, value: String) /* trailer has arrived (this we actually haven't implemented yet) */
case end /* body and trailers finished */
}

public struct HTTPHeaders {
var storage: [String:[String]] /* lower cased keys */
var original: [(String, String)] /* original casing */
var description: String

subscript(key: String) -> [String]
func makeIterator() -> IndexingIterator<Array<(String, String)>>

init(_ headers: [(String, String)] = [])
}

public typealias HTTPVersion = (Int, Int)

public enum HTTPTransferEncoding {
case identity(contentLength: UInt)
case chunked
}

public enum HTTPResponseStatus {
/* use custom if you want to use a non-standard response code or
    have it available in a (UInt, String) pair from a higher-level web framework. */
case custom(code: UInt, reasonPhrase: String)

/* all the codes from http://www.iana.org/assignments/http-status-codes */
case `continue`
case switchingProtocols
case processing
case ok
case created
case accepted
case nonAuthoritativeInformation
case noContent
case resetContent
case partialContent
case multiStatus
case alreadyReported
case imUsed
case multipleChoices
case movedPermanently
case found
case seeOther
case notModified
case useProxy
case temporaryRedirect
case permanentRedirect
case badRequest
case unauthorized
case paymentRequired
case forbidden
case notFound
case methodNotAllowed
case notAcceptable
case proxyAuthenticationRequired
case requestTimeout
case conflict
case gone
case lengthRequired
case preconditionFailed
case payloadTooLarge
case uriTooLong
case unsupportedMediaType
case rangeNotSatisfiable
case expectationFailed
case misdirectedRequest
case unprocessableEntity
case locked
case failedDependency
case upgradeRequired
case preconditionRequired
case tooManyRequests
case requestHeaderFieldsTooLarge
case unavailableForLegalReasons
case internalServerError
case notImplemented
case badGateway
case serviceUnavailable
case gatewayTimeout
case httpVersionNotSupported
case variantAlsoNegotiates
case insufficientStorage
case loopDetected
case notExtended
case networkAuthenticationRequired
}

public enum HTTPMethod {
case custom(method: String)

/* everything that http_parser.[ch] supports */
case DELETE
case GET
case HEAD
case POST
case PUT
case CONNECT
case OPTIONS
case TRACE
case COPY
case LOCK
case MKCOL
case MOVE
case PROPFIND
case PROPPATCH
case SEARCH
case UNLOCK
case BIND
case REBIND
case UNBIND
case ACL
case REPORT
case MKACTIVITY
case CHECKOUT
case MERGE
case MSEARCH
case NOTIFY
case SUBSCRIBE
case UNSUBSCRIBE
case PATCH
case PURGE
case MKCALENDAR
case LINK
case UNLINK
}
--- SNAP ---

Here's the demo code for a simple echo server

--- SNIP ---
serve { (req, res) in
  if req.target == "/echo" {
      guard req.httpVersion == (1, 1) else {
          /* HTTP/1.0 doesn't support chunked encoding */
          res.writeResponse(HTTPResponse(version: req.version,
                                         status: .httpVersionNotSupported,
                                         transferEncoding: .identity(contentLength: 0)))
          res.done()
          return .discardBody
      }
      res.writeResponse(HTTPResponse(version: req.version,
                                     status: .ok,
                                     transferEncoding: .chunked,
                                     headers: SomeConcreteHTTPHeaders([("X-foo": "bar")])))
      return .processBody { (chunk, stop) in
          switch chunk {
              case .chunk(let data, let finishedProcessing):
                  res.writeBody(data: data) { _ in
                      finishedProcessing()
                  }
              case .end:
                  res.done()
              default:
                  stop = true /* don't call us anymore */
                  res.abort()
          }
      }
  } else { ... }
}
--- SNAP ---


(Johannes Weiss) #4

Hi Colin,

Is writing compositional middleware a goal for this iteration of the API? I noticed that the HTTPResponseWriter doesn't appear to allow for, e.g. writing a single header and then passing the writer on (as one would do in, say, an anti-CSRF middleware). Possible I'm missing something though!

Good point! The version 1 actually only had writeHeader(key:value:) and for the sake of having a HTTPResponse type (that could then be used by an HTTP client) I got rid of that. But maybe that was over the top and we should go back to what I had before.

···

--
  Johannes

-Colin

On Wed, Apr 5, 2017 at 1:36 PM Johannes Weiß via swift-server-dev <swift-server-dev@swift.org> wrote:
Hi,

First of all, thanks very much to Helge for carefully analysing (and implementing) the first API sketch. As promised, I reworked it a bit.

Changes:
- extracted HTTPResponse as its own value type (so it could also be used for a client)
- added Helge's suggestions:
  * proper backpressure support
  * being able to ignore further parts of the HTTP body (stop parameter)

If I forgot something important, please let me know. I can't promise that I'll address it before the meeting as I'm in another office tomorrow for meetings and probably won't be able to attend either.

Please find everything below...

Cheers,
  Johannes

--- SNIP ---
/* a web app is a function that gets a HTTPRequest and a HTTPResponseWriter and returns a function which processes the HTTP request body in chunks as they arrive */

public typealias WebApp = (HTTPRequest, HTTPResponseWriter) -> HTTPBodyProcessing

public struct HTTPRequest {
  public var method : HTTPMethod
  public var target : String /* e.g. "/foo/bar?buz=qux" */
  public var httpVersion : HTTPVersion
  public var headers : HTTPHeaders
}

public struct HTTPResponse {
  public var httpVersion : HTTPVersion
  public var status: HTTPResponseStatus
  public var transferEncoding: HTTPTransferEncoding
  public var headers: HTTPHeaders
}

public protocol HTTPResponseWriter: class {
  func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send an HTTP `100 Continue` */

  func writeResponse(_ response: HTTPResponse)

  func writeTrailer(key: String, value: String)

  func writeBody(data: DispatchData) /* convenience */
  func writeBody(data: Data) /* convenience */
  func writeBody(data: DispatchData, completion: @escaping (Result<POSIXError, ()>) -> Void)
  func writeBody(data: Data, completion: @escaping (Result<POSIXError, ()>) -> Void)

  func done() /* convenience */
  func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
  func abort()
}

public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void /* the Bool can be set to true when we don't want to process anything further */

public enum HTTPBodyProcessing {
   case discardBody /* if you're not interested in the body */
   case processBody(handler: HTTPBodyHandler)
}

public enum HTTPBodyChunk {
  case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a new bit of the HTTP request body has arrived, finishedProcessing() must be called when done with that chunk */
  case failed(error: HTTPParserError) /* error while streaming the HTTP request body, eg. connection closed */
  case trailer(key: String, value: String) /* trailer has arrived (this we actually haven't implemented yet) */
  case end /* body and trailers finished */
}

public struct HTTPHeaders {
  var storage: [String:[String]] /* lower cased keys */
  var original: [(String, String)] /* original casing */
  var description: String

  subscript(key: String) -> [String]
  func makeIterator() -> IndexingIterator<Array<(String, String)>>

  init(_ headers: [(String, String)] = [])
}

public typealias HTTPVersion = (Int, Int)

public enum HTTPTransferEncoding {
  case identity(contentLength: UInt)
  case chunked
}

public enum HTTPResponseStatus {
  /* use custom if you want to use a non-standard response code or
     have it available in a (UInt, String) pair from a higher-level web framework. */
  case custom(code: UInt, reasonPhrase: String)

  /* all the codes from http://www.iana.org/assignments/http-status-codes */
  case `continue`
  case switchingProtocols
  case processing
  case ok
  case created
  case accepted
  case nonAuthoritativeInformation
  case noContent
  case resetContent
  case partialContent
  case multiStatus
  case alreadyReported
  case imUsed
  case multipleChoices
  case movedPermanently
  case found
  case seeOther
  case notModified
  case useProxy
  case temporaryRedirect
  case permanentRedirect
  case badRequest
  case unauthorized
  case paymentRequired
  case forbidden
  case notFound
  case methodNotAllowed
  case notAcceptable
  case proxyAuthenticationRequired
  case requestTimeout
  case conflict
  case gone
  case lengthRequired
  case preconditionFailed
  case payloadTooLarge
  case uriTooLong
  case unsupportedMediaType
  case rangeNotSatisfiable
  case expectationFailed
  case misdirectedRequest
  case unprocessableEntity
  case locked
  case failedDependency
  case upgradeRequired
  case preconditionRequired
  case tooManyRequests
  case requestHeaderFieldsTooLarge
  case unavailableForLegalReasons
  case internalServerError
  case notImplemented
  case badGateway
  case serviceUnavailable
  case gatewayTimeout
  case httpVersionNotSupported
  case variantAlsoNegotiates
  case insufficientStorage
  case loopDetected
  case notExtended
  case networkAuthenticationRequired
}

public enum HTTPMethod {
  case custom(method: String)

  /* everything that http_parser.[ch] supports */
  case DELETE
  case GET
  case HEAD
  case POST
  case PUT
  case CONNECT
  case OPTIONS
  case TRACE
  case COPY
  case LOCK
  case MKCOL
  case MOVE
  case PROPFIND
  case PROPPATCH
  case SEARCH
  case UNLOCK
  case BIND
  case REBIND
  case UNBIND
  case ACL
  case REPORT
  case MKACTIVITY
  case CHECKOUT
  case MERGE
  case MSEARCH
  case NOTIFY
  case SUBSCRIBE
  case UNSUBSCRIBE
  case PATCH
  case PURGE
  case MKCALENDAR
  case LINK
  case UNLINK
}
--- SNAP ---

Here's the demo code for a simple echo server

--- SNIP ---
serve { (req, res) in
   if req.target == "/echo" {
       guard req.httpVersion == (1, 1) else {
           /* HTTP/1.0 doesn't support chunked encoding */
           res.writeResponse(HTTPResponse(version: req.version,
                                          status: .httpVersionNotSupported,
                                          transferEncoding: .identity(contentLength: 0)))
           res.done()
           return .discardBody
       }
       res.writeResponse(HTTPResponse(version: req.version,
                                      status: .ok,
                                      transferEncoding: .chunked,
                                      headers: SomeConcreteHTTPHeaders([("X-foo": "bar")])))
       return .processBody { (chunk, stop) in
           switch chunk {
               case .chunk(let data, let finishedProcessing):
                   res.writeBody(data: data) { _ in
                       finishedProcessing()
                   }
               case .end:
                   res.done()
               default:
                   stop = true /* don't call us anymore */
                   res.abort()
           }
       }
   } else { ... }
}
--- SNAP ---

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Helge Heß) #5

My understanding is that anything middleware is not a goal at all, except for permitting a middleware framework to be built on top of this.

This is just the low level http output stream, is there anything which you think prohibits building that on top?

The writer can be passed around, it is a ref object until done is called?

hh

···

On 6. Apr 2017, at 02:20, Colin Barrett via swift-server-dev <swift-server-dev@swift.org <mailto:swift-server-dev@swift.org>> wrote:

Hello all,

Is writing compositional middleware a goal for this iteration of the API? I noticed that the HTTPResponseWriter doesn't appear to allow for, e.g. writing a single header and then passing the writer on (as one would do in, say, an anti-CSRF middleware). Possible I'm missing something though!

-Colin

On Wed, Apr 5, 2017 at 1:36 PM Johannes Weiß via swift-server-dev <swift-server-dev@swift.org <mailto:swift-server-dev@swift.org>> wrote:
Hi,

First of all, thanks very much to Helge for carefully analysing (and implementing) the first API sketch. As promised, I reworked it a bit.

Changes:
- extracted HTTPResponse as its own value type (so it could also be used for a client)
- added Helge's suggestions:
  * proper backpressure support
  * being able to ignore further parts of the HTTP body (stop parameter)

If I forgot something important, please let me know. I can't promise that I'll address it before the meeting as I'm in another office tomorrow for meetings and probably won't be able to attend either.

Please find everything below...

Cheers,
  Johannes

--- SNIP ---
/* a web app is a function that gets a HTTPRequest and a HTTPResponseWriter and returns a function which processes the HTTP request body in chunks as they arrive */

public typealias WebApp = (HTTPRequest, HTTPResponseWriter) -> HTTPBodyProcessing

public struct HTTPRequest {
  public var method : HTTPMethod
  public var target : String /* e.g. "/foo/bar?buz=qux" */
  public var httpVersion : HTTPVersion
  public var headers : HTTPHeaders
}

public struct HTTPResponse {
  public var httpVersion : HTTPVersion
  public var status: HTTPResponseStatus
  public var transferEncoding: HTTPTransferEncoding
  public var headers: HTTPHeaders
}

public protocol HTTPResponseWriter: class {
  func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send an HTTP `100 Continue` */

  func writeResponse(_ response: HTTPResponse)

  func writeTrailer(key: String, value: String)

  func writeBody(data: DispatchData) /* convenience */
  func writeBody(data: Data) /* convenience */
  func writeBody(data: DispatchData, completion: @escaping (Result<POSIXError, ()>) -> Void)
  func writeBody(data: Data, completion: @escaping (Result<POSIXError, ()>) -> Void)

  func done() /* convenience */
  func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
  func abort()
}

public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void /* the Bool can be set to true when we don't want to process anything further */

public enum HTTPBodyProcessing {
   case discardBody /* if you're not interested in the body */
   case processBody(handler: HTTPBodyHandler)
}

public enum HTTPBodyChunk {
  case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a new bit of the HTTP request body has arrived, finishedProcessing() must be called when done with that chunk */
  case failed(error: HTTPParserError) /* error while streaming the HTTP request body, eg. connection closed */
  case trailer(key: String, value: String) /* trailer has arrived (this we actually haven't implemented yet) */
  case end /* body and trailers finished */
}

public struct HTTPHeaders {
  var storage: [String:[String]] /* lower cased keys */
  var original: [(String, String)] /* original casing */
  var description: String

  subscript(key: String) -> [String]
  func makeIterator() -> IndexingIterator<Array<(String, String)>>

  init(_ headers: [(String, String)] = [])
}

public typealias HTTPVersion = (Int, Int)

public enum HTTPTransferEncoding {
  case identity(contentLength: UInt)
  case chunked
}

public enum HTTPResponseStatus {
  /* use custom if you want to use a non-standard response code or
     have it available in a (UInt, String) pair from a higher-level web framework. */
  case custom(code: UInt, reasonPhrase: String)

  /* all the codes from http://www.iana.org/assignments/http-status-codes */
  case `continue`
  case switchingProtocols
  case processing
  case ok
  case created
  case accepted
  case nonAuthoritativeInformation
  case noContent
  case resetContent
  case partialContent
  case multiStatus
  case alreadyReported
  case imUsed
  case multipleChoices
  case movedPermanently
  case found
  case seeOther
  case notModified
  case useProxy
  case temporaryRedirect
  case permanentRedirect
  case badRequest
  case unauthorized
  case paymentRequired
  case forbidden
  case notFound
  case methodNotAllowed
  case notAcceptable
  case proxyAuthenticationRequired
  case requestTimeout
  case conflict
  case gone
  case lengthRequired
  case preconditionFailed
  case payloadTooLarge
  case uriTooLong
  case unsupportedMediaType
  case rangeNotSatisfiable
  case expectationFailed
  case misdirectedRequest
  case unprocessableEntity
  case locked
  case failedDependency
  case upgradeRequired
  case preconditionRequired
  case tooManyRequests
  case requestHeaderFieldsTooLarge
  case unavailableForLegalReasons
  case internalServerError
  case notImplemented
  case badGateway
  case serviceUnavailable
  case gatewayTimeout
  case httpVersionNotSupported
  case variantAlsoNegotiates
  case insufficientStorage
  case loopDetected
  case notExtended
  case networkAuthenticationRequired
}

public enum HTTPMethod {
  case custom(method: String)

  /* everything that http_parser.[ch] supports */
  case DELETE
  case GET
  case HEAD
  case POST
  case PUT
  case CONNECT
  case OPTIONS
  case TRACE
  case COPY
  case LOCK
  case MKCOL
  case MOVE
  case PROPFIND
  case PROPPATCH
  case SEARCH
  case UNLOCK
  case BIND
  case REBIND
  case UNBIND
  case ACL
  case REPORT
  case MKACTIVITY
  case CHECKOUT
  case MERGE
  case MSEARCH
  case NOTIFY
  case SUBSCRIBE
  case UNSUBSCRIBE
  case PATCH
  case PURGE
  case MKCALENDAR
  case LINK
  case UNLINK
}
--- SNAP ---

Here's the demo code for a simple echo server

--- SNIP ---
serve { (req, res) in
   if req.target == "/echo" {
       guard req.httpVersion == (1, 1) else {
           /* HTTP/1.0 doesn't support chunked encoding */
           res.writeResponse(HTTPResponse(version: req.version,
                                          status: .httpVersionNotSupported,
                                          transferEncoding: .identity(contentLength: 0)))
           res.done()
           return .discardBody
       }
       res.writeResponse(HTTPResponse(version: req.version,
                                      status: .ok,
                                      transferEncoding: .chunked,
                                      headers: SomeConcreteHTTPHeaders([("X-foo": "bar")])))
       return .processBody { (chunk, stop) in
           switch chunk {
               case .chunk(let data, let finishedProcessing):
                   res.writeBody(data: data) { _ in
                       finishedProcessing()
                   }
               case .end:
                   res.done()
               default:
                   stop = true /* don't call us anymore */
                   res.abort()
           }
       }
   } else { ... }
}
--- SNAP ---

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org <mailto:swift-server-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-server-dev
_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org <mailto:swift-server-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Johannes Weiss) #6

Hi Johannes,

my notes on it:

- concrete types instead of protocols for HTTPRequest
- bad for layering this on top of alternative
   implementations

ok, let's discuss this separately on the list.

- bad because it prohibits future changes in the
   implementation (e.g. I still think it is a bad idea
   to convert everything to Strings in advance)
- it can still come with a default imp

ok, maybe we should offer both types of API?

- also: you could make NSURLRequest/Response support it

I don't think they're very useful in their current state. For example:

- headers is a NSDictionary<NSString *,NSString *> which isn't expressive enough
- many properties that don't universally make sense like
  - @property(readonly) BOOL allowsCellularAccess;
  - @property(readonly) NSURLRequestCachePolicy cachePolicy;
  - @property(readonly) NSTimeInterval timeoutInterval;
  - @property(readonly) NSURLRequestNetworkServiceType networkServiceType;
  - @property(readonly, copy) NSURL *URL; (we only know the target, not the whole URL at this point)
  - basically they were designed for NSURLConnection I believe
- other things like HTTP version are missing
- also NSURLRequest is also a concrete type so your points from above also apply there, right?

- maybe HTTPRequest should be called HTTPRequestHead,
because in this case it really is what it is, it
doesn’t even have a stream reference attached to it.

I'm open to that, not really fussed

- you should probably not add ‘convenience’ to this, that
will be done by frameworks using it in whatever they
consider ‘convenient’

that was more for the benefit of the readers of this mailing list. But yes, in the actual API, I'd leave that comment out.

- or it other words: does providing a `write` w/o a
   completion handler give performance benefits and
   is part of the API because of this, or is it just
   `write(..) { _ in }`. I could see that it can be a
   perf advantage (no setup of an escaping closure
   necessary), but then it should not be marked as
   ‘convenience’
- same for Data vs DispatchData. Decide on just one?

hmm, if you happen to have a DispatchData it does give you performance benefits to hand that down. So in our implementation I started off with only DispatchData but it then turned out that it's convenient to also being able to write data directly. Especially before we had our higher level web frameworks on top of that. But maybe that's not needed.

- maybe such:

    func writeTrailer(key: String, value: String)

should be

    func writeTrailer(key: UnsafePointer<CChar>,
                      value: UnsafePointer<CChar>)

this also works with strings out of the box while
allowing applications not using Strings for protocol
data ;->

will work with string _literals_ not Strings, maybe we should have both?

- I guess all ‘write’ functions need a completion
callback? (including writeTrailer, writeResponse)

in my experience having that on the done() handler is enough. The amount of data written in trailers/response (headers)/... is small enough that you can do it regardless.

- I’m still a little worried about using `enum`s for
an evolving protocol. That will look nice now but
will fall apart in the future as we can’t extend
them w/o breaking apps

which of the enums will be necessary to evolve? Both have a `custom` case now for everything else. And most use-cases would use higher-level frameworks anyway.

- Talking enums. You use them for status and method,
why not for header keys? I see _big_ performance
advantages in that.

that's surprising, what exactly is giving you the big advantages and how much is big? Is it the storage or what exactly?

- Having the stream support ‘HTTPResponse’ looks like
too high level for your specific design. You can keep
that object and add a convenience method to put such
on the stream.

as all the others above, I think very good discussion points. Anyone else having opinions?

- also: HTTPResponse vs HTTPResponseHead[er?]

sure, why not? :wink:

Thanks,
  -- Johannes

···

On 6 Apr 2017, at 09:28, Helge Heß <me@helgehess.eu> wrote:

That is it for now :slight_smile:

hh

On 5. Apr 2017, at 19:35, Johannes Weiß <johannesweiss@apple.com> wrote:

Hi,

First of all, thanks very much to Helge for carefully analysing (and implementing) the first API sketch. As promised, I reworked it a bit.

Changes:
- extracted HTTPResponse as its own value type (so it could also be used for a client)
- added Helge's suggestions:
* proper backpressure support
* being able to ignore further parts of the HTTP body (stop parameter)

If I forgot something important, please let me know. I can't promise that I'll address it before the meeting as I'm in another office tomorrow for meetings and probably won't be able to attend either.

Please find everything below...

Cheers,
Johannes

--- SNIP ---
/* a web app is a function that gets a HTTPRequest and a HTTPResponseWriter and returns a function which processes the HTTP request body in chunks as they arrive */

public typealias WebApp = (HTTPRequest, HTTPResponseWriter) -> HTTPBodyProcessing

public struct HTTPRequest {
public var method : HTTPMethod
public var target : String /* e.g. "/foo/bar?buz=qux" */
public var httpVersion : HTTPVersion
public var headers : HTTPHeaders
}

public struct HTTPResponse {
public var httpVersion : HTTPVersion
public var status: HTTPResponseStatus
public var transferEncoding: HTTPTransferEncoding
public var headers: HTTPHeaders
}

public protocol HTTPResponseWriter: class {
func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send an HTTP `100 Continue` */

func writeResponse(_ response: HTTPResponse)

func writeTrailer(key: String, value: String)

func writeBody(data: DispatchData) /* convenience */
func writeBody(data: Data) /* convenience */
func writeBody(data: DispatchData, completion: @escaping (Result<POSIXError, ()>) -> Void)
func writeBody(data: Data, completion: @escaping (Result<POSIXError, ()>) -> Void)

func done() /* convenience */
func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
func abort()
}

public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void /* the Bool can be set to true when we don't want to process anything further */

public enum HTTPBodyProcessing {
case discardBody /* if you're not interested in the body */
case processBody(handler: HTTPBodyHandler)
}

public enum HTTPBodyChunk {
case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a new bit of the HTTP request body has arrived, finishedProcessing() must be called when done with that chunk */
case failed(error: HTTPParserError) /* error while streaming the HTTP request body, eg. connection closed */
case trailer(key: String, value: String) /* trailer has arrived (this we actually haven't implemented yet) */
case end /* body and trailers finished */
}

public struct HTTPHeaders {
var storage: [String:[String]] /* lower cased keys */
var original: [(String, String)] /* original casing */
var description: String

subscript(key: String) -> [String]
func makeIterator() -> IndexingIterator<Array<(String, String)>>

init(_ headers: [(String, String)] = [])
}

public typealias HTTPVersion = (Int, Int)

public enum HTTPTransferEncoding {
case identity(contentLength: UInt)
case chunked
}

public enum HTTPResponseStatus {
/* use custom if you want to use a non-standard response code or
   have it available in a (UInt, String) pair from a higher-level web framework. */
case custom(code: UInt, reasonPhrase: String)

/* all the codes from http://www.iana.org/assignments/http-status-codes */
case `continue`
case switchingProtocols
case processing
case ok
case created
case accepted
case nonAuthoritativeInformation
case noContent
case resetContent
case partialContent
case multiStatus
case alreadyReported
case imUsed
case multipleChoices
case movedPermanently
case found
case seeOther
case notModified
case useProxy
case temporaryRedirect
case permanentRedirect
case badRequest
case unauthorized
case paymentRequired
case forbidden
case notFound
case methodNotAllowed
case notAcceptable
case proxyAuthenticationRequired
case requestTimeout
case conflict
case gone
case lengthRequired
case preconditionFailed
case payloadTooLarge
case uriTooLong
case unsupportedMediaType
case rangeNotSatisfiable
case expectationFailed
case misdirectedRequest
case unprocessableEntity
case locked
case failedDependency
case upgradeRequired
case preconditionRequired
case tooManyRequests
case requestHeaderFieldsTooLarge
case unavailableForLegalReasons
case internalServerError
case notImplemented
case badGateway
case serviceUnavailable
case gatewayTimeout
case httpVersionNotSupported
case variantAlsoNegotiates
case insufficientStorage
case loopDetected
case notExtended
case networkAuthenticationRequired
}

public enum HTTPMethod {
case custom(method: String)

/* everything that http_parser.[ch] supports */
case DELETE
case GET
case HEAD
case POST
case PUT
case CONNECT
case OPTIONS
case TRACE
case COPY
case LOCK
case MKCOL
case MOVE
case PROPFIND
case PROPPATCH
case SEARCH
case UNLOCK
case BIND
case REBIND
case UNBIND
case ACL
case REPORT
case MKACTIVITY
case CHECKOUT
case MERGE
case MSEARCH
case NOTIFY
case SUBSCRIBE
case UNSUBSCRIBE
case PATCH
case PURGE
case MKCALENDAR
case LINK
case UNLINK
}
--- SNAP ---

Here's the demo code for a simple echo server

--- SNIP ---
serve { (req, res) in
if req.target == "/echo" {
     guard req.httpVersion == (1, 1) else {
         /* HTTP/1.0 doesn't support chunked encoding */
         res.writeResponse(HTTPResponse(version: req.version,
                                        status: .httpVersionNotSupported,
                                        transferEncoding: .identity(contentLength: 0)))
         res.done()
         return .discardBody
     }
     res.writeResponse(HTTPResponse(version: req.version,
                                    status: .ok,
                                    transferEncoding: .chunked,
                                    headers: SomeConcreteHTTPHeaders([("X-foo": "bar")])))
     return .processBody { (chunk, stop) in
         switch chunk {
             case .chunk(let data, let finishedProcessing):
                 res.writeBody(data: data) { _ in
                     finishedProcessing()
                 }
             case .end:
                 res.done()
             default:
                 stop = true /* don't call us anymore */
                 res.abort()
         }
     }
} else { ... }
}
--- SNAP ---


(Colin Barrett) #7

Hi Colin,

Is writing compositional middleware a goal for this iteration of the API? I
noticed that the HTTPResponseWriter doesn't appear to allow for, e.g.
writing a single header and then passing the writer on (as one would do in,
say, an anti-CSRF middleware). Possible I'm missing something though!

Good point! The version 1 actually only had writeHeader(key:value:) and for
the sake of having a HTTPResponse type (that could then be used by an HTTP
client) I got rid of that. But maybe that was over the top and we should go
back to what I had before.

I think having the HTTPResponse is nice. I would suggest these changes:

writeResponse(..., finishHeaders = true)
writeHeader(...)
finishHeaders() [-> Bool] // It is an error to call writeHeader after
finishHeaders. Calling finishHeaders twice is a no-op. (Return value could
differentiates those cases?)
writeBody(...) // Calls finishHeaders()

Sorry for the brevity of my sketch, let me know if that's unclear in any
way.

Thanks,
-Colin

···

On Thu, Apr 6, 2017 at 3:53 AM Johannes Weiß <johannesweiss@apple.com> wrote:

--
  Johannes

-Colin

On Wed, Apr 5, 2017 at 1:36 PM Johannes Weiß via swift-server-dev < swift-server-dev@swift.org> wrote:

Hi,

First of all, thanks very much to Helge for carefully analysing (and
implementing) the first API sketch. As promised, I reworked it a bit.

Changes:
- extracted HTTPResponse as its own value type (so it could also be used
for a client)
- added Helge's suggestions:
  * proper backpressure support
  * being able to ignore further parts of the HTTP body (stop parameter)

If I forgot something important, please let me know. I can't promise that
I'll address it before the meeting as I'm in another office tomorrow for
meetings and probably won't be able to attend either.

Please find everything below...

Cheers,
  Johannes

--- SNIP ---
/* a web app is a function that gets a HTTPRequest and a HTTPResponseWriter
and returns a function which processes the HTTP request body in chunks as
they arrive */

public typealias WebApp = (HTTPRequest, HTTPResponseWriter) ->
HTTPBodyProcessing

public struct HTTPRequest {
  public var method : HTTPMethod
  public var target : String /* e.g. "/foo/bar?buz=qux" */
  public var httpVersion : HTTPVersion
  public var headers : HTTPHeaders
}

public struct HTTPResponse {
  public var httpVersion : HTTPVersion
  public var status: HTTPResponseStatus
  public var transferEncoding: HTTPTransferEncoding
  public var headers: HTTPHeaders
}

public protocol HTTPResponseWriter: class {
  func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send an
HTTP `100 Continue` */

  func writeResponse(_ response: HTTPResponse)

  func writeTrailer(key: String, value: String)

  func writeBody(data: DispatchData) /* convenience */
  func writeBody(data: Data) /* convenience */
  func writeBody(data: DispatchData, completion: @escaping
(Result<POSIXError, ()>) -> Void)
  func writeBody(data: Data, completion: @escaping (Result<POSIXError, ()>)
-> Void)

  func done() /* convenience */
  func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
  func abort()
}

public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void /*
the Bool can be set to true when we don't want to process anything further
*/

public enum HTTPBodyProcessing {
   case discardBody /* if you're not interested in the body */
   case processBody(handler: HTTPBodyHandler)
}

public enum HTTPBodyChunk {
  case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a new
bit of the HTTP request body has arrived, finishedProcessing() must be
called when done with that chunk */
  case failed(error: HTTPParserError) /* error while streaming the HTTP
request body, eg. connection closed */
  case trailer(key: String, value: String) /* trailer has arrived (this we
actually haven't implemented yet) */
  case end /* body and trailers finished */
}

public struct HTTPHeaders {
  var storage: [String:[String]] /* lower cased keys */
  var original: [(String, String)] /* original casing */
  var description: String

  subscript(key: String) -> [String]
  func makeIterator() -> IndexingIterator<Array<(String, String)>>

  init(_ headers: [(String, String)] = [])
}

public typealias HTTPVersion = (Int, Int)

public enum HTTPTransferEncoding {
  case identity(contentLength: UInt)
  case chunked
}

public enum HTTPResponseStatus {
  /* use custom if you want to use a non-standard response code or
     have it available in a (UInt, String) pair from a higher-level web
framework. */
  case custom(code: UInt, reasonPhrase: String)

  /* all the codes from http://www.iana.org/assignments/http-status-codes */
  case `continue`
  case switchingProtocols
  case processing
  case ok
  case created
  case accepted
  case nonAuthoritativeInformation
  case noContent
  case resetContent
  case partialContent
  case multiStatus
  case alreadyReported
  case imUsed
  case multipleChoices
  case movedPermanently
  case found
  case seeOther
  case notModified
  case useProxy
  case temporaryRedirect
  case permanentRedirect
  case badRequest
  case unauthorized
  case paymentRequired
  case forbidden
  case notFound
  case methodNotAllowed
  case notAcceptable
  case proxyAuthenticationRequired
  case requestTimeout
  case conflict
  case gone
  case lengthRequired
  case preconditionFailed
  case payloadTooLarge
  case uriTooLong
  case unsupportedMediaType
  case rangeNotSatisfiable
  case expectationFailed
  case misdirectedRequest
  case unprocessableEntity
  case locked
  case failedDependency
  case upgradeRequired
  case preconditionRequired
  case tooManyRequests
  case requestHeaderFieldsTooLarge
  case unavailableForLegalReasons
  case internalServerError
  case notImplemented
  case badGateway
  case serviceUnavailable
  case gatewayTimeout
  case httpVersionNotSupported
  case variantAlsoNegotiates
  case insufficientStorage
  case loopDetected
  case notExtended
  case networkAuthenticationRequired
}

public enum HTTPMethod {
  case custom(method: String)

  /* everything that http_parser.[ch] supports */
  case DELETE
  case GET
  case HEAD
  case POST
  case PUT
  case CONNECT
  case OPTIONS
  case TRACE
  case COPY
  case LOCK
  case MKCOL
  case MOVE
  case PROPFIND
  case PROPPATCH
  case SEARCH
  case UNLOCK
  case BIND
  case REBIND
  case UNBIND
  case ACL
  case REPORT
  case MKACTIVITY
  case CHECKOUT
  case MERGE
  case MSEARCH
  case NOTIFY
  case SUBSCRIBE
  case UNSUBSCRIBE
  case PATCH
  case PURGE
  case MKCALENDAR
  case LINK
  case UNLINK
}
--- SNAP ---

Here's the demo code for a simple echo server

--- SNIP ---
serve { (req, res) in
   if req.target == "/echo" {
       guard req.httpVersion == (1, 1) else {
           /* HTTP/1.0 doesn't support chunked encoding */
           res.writeResponse(HTTPResponse(version: req.version,
                                          status: .httpVersionNotSupported,
                                          transferEncoding:
.identity(contentLength: 0)))
           res.done()
           return .discardBody
       }
       res.writeResponse(HTTPResponse(version: req.version,
                                      status: .ok,
                                      transferEncoding: .chunked,
                                      headers:
SomeConcreteHTTPHeaders([("X-foo": "bar")])))
       return .processBody { (chunk, stop) in
           switch chunk {
               case .chunk(let data, let finishedProcessing):
                   res.writeBody(data: data) { _ in
                       finishedProcessing()
                   }
               case .end:
                   res.done()
               default:
                   stop = true /* don't call us anymore */
                   res.abort()
           }
       }
   } else { ... }
}
--- SNAP ---

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Helge Heß) #8

- concrete types instead of protocols for HTTPRequest
- bad for layering this on top of alternative
  implementations
- it can still come with a default imp

ok, maybe we should offer both types of API?

That is my thinking, yes. A protocol and then a concrete default implementation to get people going.

- also: you could make NSURLRequest/Response support it

I don't think they're very useful in their current state. For example:

- headers is a NSDictionary<NSString *,NSString *> which isn't expressive enough
- many properties that don't universally make sense like

...

- also NSURLRequest is also a concrete type so your points from above also apply there, right?

You misunderstood me. I’m saying that `HTTPRequest` should be a protocol and that Foundation.URLRequest could be one implementation conforming to it. And subsequently work with all APIs using the new S3 `HTTPRequest`.

- you should probably not add ‘convenience’ to this, that
will be done by frameworks using it in whatever they
consider ‘convenient’

that was more for the benefit of the readers of this mailing list. But yes, in the actual API, I'd leave that comment out.

If it is a protocol I’d leave the whole method out! :slight_smile: More like:

  protocol Writer {
    func writeBody(data: DispatchData)
  }
  extension Writer { // convenience
    func writeBody(data: Data)

  }

- or it other words: does providing a `write` w/o a
  completion handler give performance benefits and
  is part of the API because of this, or is it just
  `write(..) { _ in }`. I could see that it can be a
  perf advantage (no setup of an escaping closure
  necessary), but then it should not be marked as
  ‘convenience’
- same for Data vs DispatchData. Decide on just one?

hmm, if you happen to have a DispatchData it does give you performance benefits to hand that down. So in our implementation I started off with only DispatchData but it then turned out that it's convenient to also being able to write data directly. Especially before we had our higher level web frameworks on top of that. But maybe that's not needed.

If Data doesn’t replace DispatchData eventually, I think the Foundation team should provide cheap ways to convert between the two. It is a little weird that there are two of those …

- maybe such:

   func writeTrailer(key: String, value: String)

should be

   func writeTrailer(key: UnsafePointer<CChar>,
                     value: UnsafePointer<CChar>)

this also works with strings out of the box while
allowing applications not using Strings for protocol
data ;->

will work with string _literals_ not Strings, maybe we should have both?

It works with any String:

    func printIt(_ key: UnsafePointer<CChar>) {
      let s = String(cString: key)
      print("key is \(s)”)
    }
    var abc = “hello”
    abc += “ world”
    printIt(abc)

- I guess all ‘write’ functions need a completion
callback? (including writeTrailer, writeResponse)

in my experience having that on the done() handler is enough. The amount of data written in trailers/response (headers)/... is small enough that you can do it regardless.

Maybe. Not sure, depends a little on the async framework in use. A header can certainly be big enough to overflow the socket buffer and block.
Nginx uses 40 bytes per connection? I’m sure they wouldn’t want to be forced to buffer a 700 byte header block :slight_smile:

Not sure, I guess you are right.

- I’m still a little worried about using `enum`s for
an evolving protocol. That will look nice now but
will fall apart in the future as we can’t extend
them w/o breaking apps

which of the enums will be necessary to evolve?

All of them. E.g. PATCH is pretty new, or MKCOLLECTION. If a new method or HTTP status becomes ubiquitous, it will quickly get weird, all code will look like:

  switch method {
    case .get: …
    case .put: ...
    case .delete: …
    case custom(“BATCH”): // wtf! so ugly
  }

Makes your pretty Swift code really ugly :slight_smile: Int’s plus constants are better for stuff that can change.

Both have a `custom` case now for everything else. And most use-cases would use higher-level frameworks anyway.

Well, ideally a framework could reuse common stuff like the HTTPMethod type. It looks a little stupid to wrap such :slight_smile:

- Talking enums. You use them for status and method,
why not for header keys? I see _big_ performance
advantages in that.

that's surprising, what exactly is giving you the big advantages and how much is big? Is it the storage or what exactly?

- storage (essentially a byte vs the full String thing)

- storage reuse (are you going to reconstruct the String
  every time you parse a header? or have a thread safe
  intern map?) No issue with a byte.

- ARC (potentially, much more likely than for Ints :wink:

- comparison speed - compare two bytes (a single
  assembly instruction) vs comparing two Strings,
  potentially unique ones with all the checks involved.
  Order of magnitude more assembly instructions (I guess :wink:

That is also a reason why I’m in favour of supporting value
types instead of just string. Converting your content-length
into a String is a little stupid. That can be done at the
C level below much faster.

As I said, I suppose people that need performance may be doing their own thing anyways. But it would be cool if it would be great out of the box :slight_smile:
BTW: I’m not necessarily thinking high-scale cloud servers, I’m thinking more about stuff running on Rasπ’s.

- Having the stream support ‘HTTPResponse’ looks like
too high level for your specific design. You can keep
that object and add a convenience method to put such
on the stream.

as all the others above, I think very good discussion points. Anyone else having opinions?

- also: HTTPResponse vs HTTPResponseHead[er?]

sure, why not? :wink:

I’d like to mention again that in HTTP/2 methods are just headers. Maybe the stuff should be modelled after that?

  request[.method] == .put

hh

···

On 07 Apr 2017, at 12:24, Johannes Weiß <johannesweiss@apple.com> wrote:

Thanks,
-- Johannes

That is it for now :slight_smile:

hh

On 5. Apr 2017, at 19:35, Johannes Weiß <johannesweiss@apple.com> wrote:

Hi,

First of all, thanks very much to Helge for carefully analysing (and implementing) the first API sketch. As promised, I reworked it a bit.

Changes:
- extracted HTTPResponse as its own value type (so it could also be used for a client)
- added Helge's suggestions:
* proper backpressure support
* being able to ignore further parts of the HTTP body (stop parameter)

If I forgot something important, please let me know. I can't promise that I'll address it before the meeting as I'm in another office tomorrow for meetings and probably won't be able to attend either.

Please find everything below...

Cheers,
Johannes

--- SNIP ---
/* a web app is a function that gets a HTTPRequest and a HTTPResponseWriter and returns a function which processes the HTTP request body in chunks as they arrive */

public typealias WebApp = (HTTPRequest, HTTPResponseWriter) -> HTTPBodyProcessing

public struct HTTPRequest {
public var method : HTTPMethod
public var target : String /* e.g. "/foo/bar?buz=qux" */
public var httpVersion : HTTPVersion
public var headers : HTTPHeaders
}

public struct HTTPResponse {
public var httpVersion : HTTPVersion
public var status: HTTPResponseStatus
public var transferEncoding: HTTPTransferEncoding
public var headers: HTTPHeaders
}

public protocol HTTPResponseWriter: class {
func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send an HTTP `100 Continue` */

func writeResponse(_ response: HTTPResponse)

func writeTrailer(key: String, value: String)

func writeBody(data: DispatchData) /* convenience */
func writeBody(data: Data) /* convenience */
func writeBody(data: DispatchData, completion: @escaping (Result<POSIXError, ()>) -> Void)
func writeBody(data: Data, completion: @escaping (Result<POSIXError, ()>) -> Void)

func done() /* convenience */
func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
func abort()
}

public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void /* the Bool can be set to true when we don't want to process anything further */

public enum HTTPBodyProcessing {
case discardBody /* if you're not interested in the body */
case processBody(handler: HTTPBodyHandler)
}

public enum HTTPBodyChunk {
case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a new bit of the HTTP request body has arrived, finishedProcessing() must be called when done with that chunk */
case failed(error: HTTPParserError) /* error while streaming the HTTP request body, eg. connection closed */
case trailer(key: String, value: String) /* trailer has arrived (this we actually haven't implemented yet) */
case end /* body and trailers finished */
}

public struct HTTPHeaders {
var storage: [String:[String]] /* lower cased keys */
var original: [(String, String)] /* original casing */
var description: String

subscript(key: String) -> [String]
func makeIterator() -> IndexingIterator<Array<(String, String)>>

init(_ headers: [(String, String)] = [])
}

public typealias HTTPVersion = (Int, Int)

public enum HTTPTransferEncoding {
case identity(contentLength: UInt)
case chunked
}

public enum HTTPResponseStatus {
/* use custom if you want to use a non-standard response code or
  have it available in a (UInt, String) pair from a higher-level web framework. */
case custom(code: UInt, reasonPhrase: String)

/* all the codes from http://www.iana.org/assignments/http-status-codes */
case `continue`
case switchingProtocols
case processing
case ok
case created
case accepted
case nonAuthoritativeInformation
case noContent
case resetContent
case partialContent
case multiStatus
case alreadyReported
case imUsed
case multipleChoices
case movedPermanently
case found
case seeOther
case notModified
case useProxy
case temporaryRedirect
case permanentRedirect
case badRequest
case unauthorized
case paymentRequired
case forbidden
case notFound
case methodNotAllowed
case notAcceptable
case proxyAuthenticationRequired
case requestTimeout
case conflict
case gone
case lengthRequired
case preconditionFailed
case payloadTooLarge
case uriTooLong
case unsupportedMediaType
case rangeNotSatisfiable
case expectationFailed
case misdirectedRequest
case unprocessableEntity
case locked
case failedDependency
case upgradeRequired
case preconditionRequired
case tooManyRequests
case requestHeaderFieldsTooLarge
case unavailableForLegalReasons
case internalServerError
case notImplemented
case badGateway
case serviceUnavailable
case gatewayTimeout
case httpVersionNotSupported
case variantAlsoNegotiates
case insufficientStorage
case loopDetected
case notExtended
case networkAuthenticationRequired
}

public enum HTTPMethod {
case custom(method: String)

/* everything that http_parser.[ch] supports */
case DELETE
case GET
case HEAD
case POST
case PUT
case CONNECT
case OPTIONS
case TRACE
case COPY
case LOCK
case MKCOL
case MOVE
case PROPFIND
case PROPPATCH
case SEARCH
case UNLOCK
case BIND
case REBIND
case UNBIND
case ACL
case REPORT
case MKACTIVITY
case CHECKOUT
case MERGE
case MSEARCH
case NOTIFY
case SUBSCRIBE
case UNSUBSCRIBE
case PATCH
case PURGE
case MKCALENDAR
case LINK
case UNLINK
}
--- SNAP ---

Here's the demo code for a simple echo server

--- SNIP ---
serve { (req, res) in
if req.target == "/echo" {
    guard req.httpVersion == (1, 1) else {
        /* HTTP/1.0 doesn't support chunked encoding */
        res.writeResponse(HTTPResponse(version: req.version,
                                       status: .httpVersionNotSupported,
                                       transferEncoding: .identity(contentLength: 0)))
        res.done()
        return .discardBody
    }
    res.writeResponse(HTTPResponse(version: req.version,
                                   status: .ok,
                                   transferEncoding: .chunked,
                                   headers: SomeConcreteHTTPHeaders([("X-foo": "bar")])))
    return .processBody { (chunk, stop) in
        switch chunk {
            case .chunk(let data, let finishedProcessing):
                res.writeBody(data: data) { _ in
                    finishedProcessing()
                }
            case .end:
                res.done()
            default:
                stop = true /* don't call us anymore */
                res.abort()
        }
    }
} else { ... }
}
--- SNAP ---


(Johannes Weiss) #9

Hi Colin,

Hi Colin,

Is writing compositional middleware a goal for this iteration of the API? I noticed that the HTTPResponseWriter doesn't appear to allow for, e.g. writing a single header and then passing the writer on (as one would do in, say, an anti-CSRF middleware). Possible I'm missing something though!

Good point! The version 1 actually only had writeHeader(key:value:) and for the sake of having a HTTPResponse type (that could then be used by an HTTP client) I got rid of that. But maybe that was over the top and we should go back to what I had before.

I think having the HTTPResponse is nice. I would suggest these changes:

writeResponse(..., finishHeaders = true)
writeHeader(...)
finishHeaders() [-> Bool] // It is an error to call writeHeader after finishHeaders. Calling finishHeaders twice is a no-op. (Return value could differentiates those cases?)
writeBody(...) // Calls finishHeaders()

Sorry for the brevity of my sketch, let me know if that's unclear in any way.

Thanks, that makes sense! I literally just started an API Sketch v3 document where I put these in so I won't forget.

Just one question about the finishHeaders(): You see this as being mandatory to call, right? At the moment our implementation doesn't have that and it's handled implicitly. Basically if writeBody/writeTrailer/done are called we finish the headers if that hasn't been done yet. In other words, there's a state machine in HTTPResponseWriter which has a headers finished state but it's managed internally.

Do you see any advantages of exposing that explicitly?

Cheers,
  Johannes

···

On 6 Apr 2017, at 22:20, Colin Barrett <colin@springsandstruts.com> wrote:
On Thu, Apr 6, 2017 at 3:53 AM Johannes Weiß <johannesweiss@apple.com> wrote:

Thanks,
-Colin

--
  Johannes

-Colin

On Wed, Apr 5, 2017 at 1:36 PM Johannes Weiß via swift-server-dev <swift-server-dev@swift.org> wrote:
Hi,

First of all, thanks very much to Helge for carefully analysing (and implementing) the first API sketch. As promised, I reworked it a bit.

Changes:
- extracted HTTPResponse as its own value type (so it could also be used for a client)
- added Helge's suggestions:
  * proper backpressure support
  * being able to ignore further parts of the HTTP body (stop parameter)

If I forgot something important, please let me know. I can't promise that I'll address it before the meeting as I'm in another office tomorrow for meetings and probably won't be able to attend either.

Please find everything below...

Cheers,
  Johannes

--- SNIP ---
/* a web app is a function that gets a HTTPRequest and a HTTPResponseWriter and returns a function which processes the HTTP request body in chunks as they arrive */

public typealias WebApp = (HTTPRequest, HTTPResponseWriter) -> HTTPBodyProcessing

public struct HTTPRequest {
  public var method : HTTPMethod
  public var target : String /* e.g. "/foo/bar?buz=qux" */
  public var httpVersion : HTTPVersion
  public var headers : HTTPHeaders
}

public struct HTTPResponse {
  public var httpVersion : HTTPVersion
  public var status: HTTPResponseStatus
  public var transferEncoding: HTTPTransferEncoding
  public var headers: HTTPHeaders
}

public protocol HTTPResponseWriter: class {
  func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send an HTTP `100 Continue` */

  func writeResponse(_ response: HTTPResponse)

  func writeTrailer(key: String, value: String)

  func writeBody(data: DispatchData) /* convenience */
  func writeBody(data: Data) /* convenience */
  func writeBody(data: DispatchData, completion: @escaping (Result<POSIXError, ()>) -> Void)
  func writeBody(data: Data, completion: @escaping (Result<POSIXError, ()>) -> Void)

  func done() /* convenience */
  func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
  func abort()
}

public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void /* the Bool can be set to true when we don't want to process anything further */

public enum HTTPBodyProcessing {
   case discardBody /* if you're not interested in the body */
   case processBody(handler: HTTPBodyHandler)
}

public enum HTTPBodyChunk {
  case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a new bit of the HTTP request body has arrived, finishedProcessing() must be called when done with that chunk */
  case failed(error: HTTPParserError) /* error while streaming the HTTP request body, eg. connection closed */
  case trailer(key: String, value: String) /* trailer has arrived (this we actually haven't implemented yet) */
  case end /* body and trailers finished */
}

public struct HTTPHeaders {
  var storage: [String:[String]] /* lower cased keys */
  var original: [(String, String)] /* original casing */
  var description: String

  subscript(key: String) -> [String]
  func makeIterator() -> IndexingIterator<Array<(String, String)>>

  init(_ headers: [(String, String)] = [])
}

public typealias HTTPVersion = (Int, Int)

public enum HTTPTransferEncoding {
  case identity(contentLength: UInt)
  case chunked
}

public enum HTTPResponseStatus {
  /* use custom if you want to use a non-standard response code or
     have it available in a (UInt, String) pair from a higher-level web framework. */
  case custom(code: UInt, reasonPhrase: String)

  /* all the codes from http://www.iana.org/assignments/http-status-codes */
  case `continue`
  case switchingProtocols
  case processing
  case ok
  case created
  case accepted
  case nonAuthoritativeInformation
  case noContent
  case resetContent
  case partialContent
  case multiStatus
  case alreadyReported
  case imUsed
  case multipleChoices
  case movedPermanently
  case found
  case seeOther
  case notModified
  case useProxy
  case temporaryRedirect
  case permanentRedirect
  case badRequest
  case unauthorized
  case paymentRequired
  case forbidden
  case notFound
  case methodNotAllowed
  case notAcceptable
  case proxyAuthenticationRequired
  case requestTimeout
  case conflict
  case gone
  case lengthRequired
  case preconditionFailed
  case payloadTooLarge
  case uriTooLong
  case unsupportedMediaType
  case rangeNotSatisfiable
  case expectationFailed
  case misdirectedRequest
  case unprocessableEntity
  case locked
  case failedDependency
  case upgradeRequired
  case preconditionRequired
  case tooManyRequests
  case requestHeaderFieldsTooLarge
  case unavailableForLegalReasons
  case internalServerError
  case notImplemented
  case badGateway
  case serviceUnavailable
  case gatewayTimeout
  case httpVersionNotSupported
  case variantAlsoNegotiates
  case insufficientStorage
  case loopDetected
  case notExtended
  case networkAuthenticationRequired
}

public enum HTTPMethod {
  case custom(method: String)

  /* everything that http_parser.[ch] supports */
  case DELETE
  case GET
  case HEAD
  case POST
  case PUT
  case CONNECT
  case OPTIONS
  case TRACE
  case COPY
  case LOCK
  case MKCOL
  case MOVE
  case PROPFIND
  case PROPPATCH
  case SEARCH
  case UNLOCK
  case BIND
  case REBIND
  case UNBIND
  case ACL
  case REPORT
  case MKACTIVITY
  case CHECKOUT
  case MERGE
  case MSEARCH
  case NOTIFY
  case SUBSCRIBE
  case UNSUBSCRIBE
  case PATCH
  case PURGE
  case MKCALENDAR
  case LINK
  case UNLINK
}
--- SNAP ---

Here's the demo code for a simple echo server

--- SNIP ---
serve { (req, res) in
   if req.target == "/echo" {
       guard req.httpVersion == (1, 1) else {
           /* HTTP/1.0 doesn't support chunked encoding */
           res.writeResponse(HTTPResponse(version: req.version,
                                          status: .httpVersionNotSupported,
                                          transferEncoding: .identity(contentLength: 0)))
           res.done()
           return .discardBody
       }
       res.writeResponse(HTTPResponse(version: req.version,
                                      status: .ok,
                                      transferEncoding: .chunked,
                                      headers: SomeConcreteHTTPHeaders([("X-foo": "bar")])))
       return .processBody { (chunk, stop) in
           switch chunk {
               case .chunk(let data, let finishedProcessing):
                   res.writeBody(data: data) { _ in
                       finishedProcessing()
                   }
               case .end:
                   res.done()
               default:
                   stop = true /* don't call us anymore */
                   res.abort()
           }
       }
   } else { ... }
}
--- SNAP ---

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Helge Heß) #10

Well, an advantage is that there have been a lot of discussions on how such objects should look like.
By avoiding it, the original proposal worked around that issue quite nicely (essentially providing a lower level so that framework implementors can do what they want for the response object).

hh

···

On 06 Apr 2017, at 23:20, Colin Barrett via swift-server-dev <swift-server-dev@swift.org> wrote:

I think having the HTTPResponse is nice


(Johannes Weiss) #11

Hi,

- concrete types instead of protocols for HTTPRequest
- bad for layering this on top of alternative
implementations
- it can still come with a default imp

ok, maybe we should offer both types of API?

That is my thinking, yes. A protocol and then a concrete default implementation to get people going.

- also: you could make NSURLRequest/Response support it

I don't think they're very useful in their current state. For example:

- headers is a NSDictionary<NSString *,NSString *> which isn't expressive enough
- many properties that don't universally make sense like

...

- also NSURLRequest is also a concrete type so your points from above also apply there, right?

You misunderstood me. I’m saying that `HTTPRequest` should be a protocol and that Foundation.URLRequest could be one implementation conforming to it. And subsequently work with all APIs using the new S3 `HTTPRequest`.

I see, that makes sense. I think we should discuss the value type vs. protocol types separately though.

- you should probably not add ‘convenience’ to this, that
will be done by frameworks using it in whatever they
consider ‘convenient’

that was more for the benefit of the readers of this mailing list. But yes, in the actual API, I'd leave that comment out.

If it is a protocol I’d leave the whole method out! :slight_smile: More like:

protocol Writer {
   func writeBody(data: DispatchData)
}
extension Writer { // convenience
   func writeBody(data: Data)

}

makes sense.

- or it other words: does providing a `write` w/o a
completion handler give performance benefits and
is part of the API because of this, or is it just
`write(..) { _ in }`. I could see that it can be a
perf advantage (no setup of an escaping closure
necessary), but then it should not be marked as
‘convenience’
- same for Data vs DispatchData. Decide on just one?

hmm, if you happen to have a DispatchData it does give you performance benefits to hand that down. So in our implementation I started off with only DispatchData but it then turned out that it's convenient to also being able to write data directly. Especially before we had our higher level web frameworks on top of that. But maybe that's not needed.

If Data doesn’t replace DispatchData eventually, I think the Foundation team should provide cheap ways to convert between the two. It is a little weird that there are two of those …

- maybe such:

  func writeTrailer(key: String, value: String)

should be

  func writeTrailer(key: UnsafePointer<CChar>,
                    value: UnsafePointer<CChar>)

this also works with strings out of the box while
allowing applications not using Strings for protocol
data ;->

will work with string _literals_ not Strings, maybe we should have both?

It works with any String:

   func printIt(_ key: UnsafePointer<CChar>) {
     let s = String(cString: key)
     print("key is \(s)”)
   }
   var abc = “hello”
   abc += “ world”
   printIt(abc)

you're right, my bad! However that's not cheap because it will need to copy the String. A C string is \0 terminated contiguous memory and a Swift string might not be \0 terminated in memory so it'll need to make a copy which is not ideal.

- I guess all ‘write’ functions need a completion
callback? (including writeTrailer, writeResponse)

in my experience having that on the done() handler is enough. The amount of data written in trailers/response (headers)/... is small enough that you can do it regardless.

Maybe. Not sure, depends a little on the async framework in use. A header can certainly be big enough to overflow the socket buffer and block.
Nginx uses 40 bytes per connection? I’m sure they wouldn’t want to be forced to buffer a 700 byte header block :slight_smile:

Not sure, I guess you are right.

- I’m still a little worried about using `enum`s for
an evolving protocol. That will look nice now but
will fall apart in the future as we can’t extend
them w/o breaking apps

which of the enums will be necessary to evolve?

All of them. E.g. PATCH is pretty new, or MKCOLLECTION. If a new method or HTTP status becomes ubiquitous, it will quickly get weird, all code will look like:

switch method {
   case .get: …
   case .put: ...
   case .delete: …
   case custom(“BATCH”): // wtf! so ugly
}

Makes your pretty Swift code really ugly :slight_smile: Int’s plus constants are better for stuff that can change.

we're talking about quite a low level HTTP library here. Not many people would actually use it directly I suppose. URL target matching is also really ugly. You'd certainly want to support http://example.com//foo//bar.html so the target would be "//foo//bar.html" so you can't use string comparison.

So I feel it's good enough for a low-level framework.

Both have a `custom` case now for everything else. And most use-cases would use higher-level frameworks anyway.

Well, ideally a framework could reuse common stuff like the HTTPMethod type. It looks a little stupid to wrap such :slight_smile:

hmm, I'd wrap it to not export low-level implementation details.

- Talking enums. You use them for status and method,
why not for header keys? I see _big_ performance
advantages in that.

that's surprising, what exactly is giving you the big advantages and how much is big? Is it the storage or what exactly?

- storage (essentially a byte vs the full String thing)

- storage reuse (are you going to reconstruct the String
every time you parse a header? or have a thread safe
intern map?) No issue with a byte.

- ARC (potentially, much more likely than for Ints :wink:
- comparison speed - compare two bytes (a single
assembly instruction) vs comparing two Strings,
potentially unique ones with all the checks involved.
Order of magnitude more assembly instructions (I guess :wink:

that can all be implemented in HTTPHeaders as an implementation detail, right? When constructing it'd see that it's for example the 'Host' header and map it to some internal constant.

That is also a reason why I’m in favour of supporting value
types instead of just string. Converting your content-length
into a String is a little stupid. That can be done at the
C level below much faster.

As I said, I suppose people that need performance may be doing their own thing anyways. But it would be cool if it would be great out of the box :slight_smile:
BTW: I’m not necessarily thinking high-scale cloud servers, I’m thinking more about stuff running on Rasπ’s.

- Having the stream support ‘HTTPResponse’ looks like
too high level for your specific design. You can keep
that object and add a convenience method to put such
on the stream.

as all the others above, I think very good discussion points. Anyone else having opinions?

- also: HTTPResponse vs HTTPResponseHead[er?]

sure, why not? :wink:

I’d like to mention again that in HTTP/2 methods are just headers. Maybe the stuff should be modelled after that?

request[.method] == .put

I do believe that people expect a request method field for HTTP/1.

-- Johannes

···

On 7 Apr 2017, at 12:18, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:
On 07 Apr 2017, at 12:24, Johannes Weiß <johannesweiss@apple.com> wrote:


(Helge Heß) #12

- maybe such:

func writeTrailer(key: String, value: String)

should be

func writeTrailer(key: UnsafePointer<CChar>,
                   value: UnsafePointer<CChar>)

this also works with strings out of the box while
allowing applications not using Strings for protocol
data ;->

will work with string _literals_ not Strings, maybe we should have both?

It works with any String:

  func printIt(_ key: UnsafePointer<CChar>) {
    let s = String(cString: key)
    print("key is \(s)”)
  }
  var abc = “hello”
  abc += “ world”
  printIt(abc)

you're right, my bad! However that's not cheap because it will need to copy the String. A C string is \0 terminated contiguous memory and a Swift string might not be \0 terminated in memory so it'll need to make a copy which is not ideal.

I can’t follow you here. Eventually the String has to be converted to a socket buffer. Since String doesn’t have an API to walk the segments (right?) you always end up doing the thing above. (unless you propose filling the socket buffer by walking the utf-8 sequence :wink:

For practical purposes most header values are very likely cstring-backed in the first place, right?

Both have a `custom` case now for everything else. And most use-cases would use higher-level frameworks anyway.

Well, ideally a framework could reuse common stuff like the HTTPMethod type. It looks a little stupid to wrap such :slight_smile:

hmm, I'd wrap it to not export low-level implementation details.

OK, one last try :-): The more types (especially so low level ones) can be successfully shared between higher level libraries,
a) the more consistent the S3 UX is for users and
b) 3rd party package authors which can restrict themselves to this protocol set, only need to build a single implementation.

- Talking enums. You use them for status and method,
why not for header keys? I see _big_ performance
advantages in that.

that's surprising, what exactly is giving you the big advantages and how much is big? Is it the storage or what exactly?

- storage (essentially a byte vs the full String thing)

- storage reuse (are you going to reconstruct the String
every time you parse a header? or have a thread safe
intern map?) No issue with a byte.

- ARC (potentially, much more likely than for Ints :wink:
- comparison speed - compare two bytes (a single
assembly instruction) vs comparing two Strings,
potentially unique ones with all the checks involved.
Order of magnitude more assembly instructions (I guess :wink:

that can all be implemented in HTTPHeaders as an implementation detail, right?

Hm, no? If you have to construct the Strings for the HTTPHeaders you gain nothing. It actually is worse because do the work twice :slight_smile:

The parser could already match the headers (at the C level, via strcmp() ...) and pass up only enums. Or even better, the state machine of the http_parser could be changed to cover more standard headers.

Also: Why not use Strings for method then? Would be more consistent and would also avoid the protocol-changes issue ;->

I’d like to mention again that in HTTP/2 methods are just headers. Maybe the stuff should be modelled after that?

request[.method] == .put

I do believe that people expect a request method field for HTTP/1.

The end user? He also usually expects a contentType field. Here you go:

  extension HTTPRequest {
    var method : HTTPMethod { return request[.method] }
    var contentType : MIMEType { return request[.contentType] }
  }

But I don’t care too much, just an idea to line things up with the future.

hh

···

On 07 Apr 2017, at 13:34, Johannes Weiß <johannesweiss@apple.com> wrote:


(Johannes Weiss) #13

Hi,

- maybe such:

func writeTrailer(key: String, value: String)

should be

func writeTrailer(key: UnsafePointer<CChar>,
                  value: UnsafePointer<CChar>)

this also works with strings out of the box while
allowing applications not using Strings for protocol
data ;->

will work with string _literals_ not Strings, maybe we should have both?

It works with any String:

func printIt(_ key: UnsafePointer<CChar>) {
   let s = String(cString: key)
   print("key is \(s)”)
}
var abc = “hello”
abc += “ world”
printIt(abc)

you're right, my bad! However that's not cheap because it will need to copy the String. A C string is \0 terminated contiguous memory and a Swift string might not be \0 terminated in memory so it'll need to make a copy which is not ideal.

I can’t follow you here. Eventually the String has to be converted to a socket buffer. Since String doesn’t have an API to walk the segments (right?) you always end up doing the thing above. (unless you propose filling the socket buffer by walking the utf-8 sequence :wink:

well, AFAIK String stores a backing buffer plus length in bytes. So there's not necessarily a trailing \0 stored there. But to expose it as a CString it will need to 'add' that and hence probably copy it.

For practical purposes most header values are very likely cstring-backed in the first place, right?

No, from http_parser.c you'll get back a char * pointer and length, there won't be any \0 bytes unless you make an extra copy.

Both have a `custom` case now for everything else. And most use-cases would use higher-level frameworks anyway.

Well, ideally a framework could reuse common stuff like the HTTPMethod type. It looks a little stupid to wrap such :slight_smile:

hmm, I'd wrap it to not export low-level implementation details.

OK, one last try :-): The more types (especially so low level ones) can be successfully shared between higher level libraries,
a) the more consistent the S3 UX is for users and
b) 3rd party package authors which can restrict themselves to this protocol set, only need to build a single implementation.

- Talking enums. You use them for status and method,
why not for header keys? I see _big_ performance
advantages in that.

that's surprising, what exactly is giving you the big advantages and how much is big? Is it the storage or what exactly?

- storage (essentially a byte vs the full String thing)

- storage reuse (are you going to reconstruct the String
every time you parse a header? or have a thread safe
intern map?) No issue with a byte.

- ARC (potentially, much more likely than for Ints :wink:
- comparison speed - compare two bytes (a single
assembly instruction) vs comparing two Strings,
potentially unique ones with all the checks involved.
Order of magnitude more assembly instructions (I guess :wink:

that can all be implemented in HTTPHeaders as an implementation detail, right?

Hm, no? If you have to construct the Strings for the HTTPHeaders you gain nothing. It actually is worse because do the work twice :slight_smile:

The parser could already match the headers (at the C level, via strcmp() ...) and pass up only enums. Or even better, the state machine of the http_parser could be changed to cover more standard headers.

well, there could be a `mutating func addHeader(key: UnsafeBufferPointer<CChar>) method on HTTPHeaders which compares the key using strcmp and if it sees that it's 'Host' for example it reuses a shared "Host": String for all instances or even an enum or so.

Also: Why not use Strings for method then? Would be more consistent and would also avoid the protocol-changes issue ;->

could do

I’d like to mention again that in HTTP/2 methods are just headers. Maybe the stuff should be modelled after that?

request[.method] == .put

I do believe that people expect a request method field for HTTP/1.

The end user? He also usually expects a contentType field. Here you go:

extension HTTPRequest {
   var method : HTTPMethod { return request[.method] }
   var contentType : MIMEType { return request[.contentType] }
}

But I don’t care too much, just an idea to line things up with the future.

that makes things less type-safe than they can be but I'm also not too fussed.

-- Johannes

···

On 7 Apr 2017, at 13:18, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:
On 07 Apr 2017, at 13:34, Johannes Weiß <johannesweiss@apple.com> wrote:


(Colin Barrett) #14

Hi Colin,

>
> Hi Colin,
>
>> Is writing compositional middleware a goal for this iteration of the
API? I noticed that the HTTPResponseWriter doesn't appear to allow for,
e.g. writing a single header and then passing the writer on (as one would
do in, say, an anti-CSRF middleware). Possible I'm missing something though!
>
> Good point! The version 1 actually only had writeHeader(key:value:) and
for the sake of having a HTTPResponse type (that could then be used by an
HTTP client) I got rid of that. But maybe that was over the top and we
should go back to what I had before.
>
> I think having the HTTPResponse is nice. I would suggest these changes:
>
> writeResponse(..., finishHeaders = true)
> writeHeader(...)
> finishHeaders() [-> Bool] // It is an error to call writeHeader after
finishHeaders. Calling finishHeaders twice is a no-op. (Return value could
differentiates those cases?)
> writeBody(...) // Calls finishHeaders()
>
> Sorry for the brevity of my sketch, let me know if that's unclear in any
way.

Thanks, that makes sense! I literally just started an API Sketch v3
document where I put these in so I won't forget.

Just one question about the finishHeaders(): You see this as being
mandatory to call, right? At the moment our implementation doesn't have
that and it's handled implicitly. Basically if writeBody/writeTrailer/done
are called we finish the headers if that hasn't been done yet. In other
words, there's a state machine in HTTPResponseWriter which has a headers
finished state but it's managed internally.

Do you see any advantages of exposing that explicitly?

Mostly just for smell reasons. Calls to writeHeader are going to fail after
writeBody. This can be seen as an additional side effect of writeBody,
besides actually emitting the body. Having two entangled effects like this
usually leads regret, or people abusing the API.

Just to be clear, its fine for one method (writeBody) to invoke the other
(finishHeaders), but users should be able to control the timing of each
effect should they so choose (for example to help enforce invariants in
their own code—"past this point I promise to write no more headers").

It's something that could be done in a wrapper class, so it's not the end
of the world if you decide to leave it out.

Cheers,

···

On Fri, Apr 7, 2017 at 6:01 AM Johannes Weiß <johannesweiss@apple.com> wrote:

> On 6 Apr 2017, at 22:20, Colin Barrett <colin@springsandstruts.com> > wrote:
> On Thu, Apr 6, 2017 at 3:53 AM Johannes Weiß <johannesweiss@apple.com> > wrote:
  Johannes

>
> Thanks,
> -Colin
>
>
> --
> Johannes
>
>>
>> -Colin
>>
>> On Wed, Apr 5, 2017 at 1:36 PM Johannes Weiß via swift-server-dev < > swift-server-dev@swift.org> wrote:
>> Hi,
>>
>> First of all, thanks very much to Helge for carefully analysing (and
implementing) the first API sketch. As promised, I reworked it a bit.
>>
>> Changes:
>> - extracted HTTPResponse as its own value type (so it could also be
used for a client)
>> - added Helge's suggestions:
>> * proper backpressure support
>> * being able to ignore further parts of the HTTP body (stop parameter)
>>
>> If I forgot something important, please let me know. I can't promise
that I'll address it before the meeting as I'm in another office tomorrow
for meetings and probably won't be able to attend either.
>>
>> Please find everything below...
>>
>> Cheers,
>> Johannes
>>
>> --- SNIP ---
>> /* a web app is a function that gets a HTTPRequest and a
HTTPResponseWriter and returns a function which processes the HTTP request
body in chunks as they arrive */
>>
>> public typealias WebApp = (HTTPRequest, HTTPResponseWriter) ->
HTTPBodyProcessing
>>
>> public struct HTTPRequest {
>> public var method : HTTPMethod
>> public var target : String /* e.g. "/foo/bar?buz=qux" */
>> public var httpVersion : HTTPVersion
>> public var headers : HTTPHeaders
>> }
>>
>> public struct HTTPResponse {
>> public var httpVersion : HTTPVersion
>> public var status: HTTPResponseStatus
>> public var transferEncoding: HTTPTransferEncoding
>> public var headers: HTTPHeaders
>> }
>>
>> public protocol HTTPResponseWriter: class {
>> func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send
an HTTP `100 Continue` */
>>
>> func writeResponse(_ response: HTTPResponse)
>>
>> func writeTrailer(key: String, value: String)
>>
>> func writeBody(data: DispatchData) /* convenience */
>> func writeBody(data: Data) /* convenience */
>> func writeBody(data: DispatchData, completion: @escaping
(Result<POSIXError, ()>) -> Void)
>> func writeBody(data: Data, completion: @escaping (Result<POSIXError,
()>) -> Void)
>>
>> func done() /* convenience */
>> func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
>> func abort()
>> }
>>
>> public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void
/* the Bool can be set to true when we don't want to process anything
further */
>>
>> public enum HTTPBodyProcessing {
>> case discardBody /* if you're not interested in the body */
>> case processBody(handler: HTTPBodyHandler)
>> }
>>
>> public enum HTTPBodyChunk {
>> case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a
new bit of the HTTP request body has arrived, finishedProcessing() must be
called when done with that chunk */
>> case failed(error: HTTPParserError) /* error while streaming the HTTP
request body, eg. connection closed */
>> case trailer(key: String, value: String) /* trailer has arrived (this
we actually haven't implemented yet) */
>> case end /* body and trailers finished */
>> }
>>
>> public struct HTTPHeaders {
>> var storage: [String:[String]] /* lower cased keys */
>> var original: [(String, String)] /* original casing */
>> var description: String
>>
>> subscript(key: String) -> [String]
>> func makeIterator() -> IndexingIterator<Array<(String, String)>>
>>
>> init(_ headers: [(String, String)] = [])
>> }
>>
>> public typealias HTTPVersion = (Int, Int)
>>
>> public enum HTTPTransferEncoding {
>> case identity(contentLength: UInt)
>> case chunked
>> }
>>
>> public enum HTTPResponseStatus {
>> /* use custom if you want to use a non-standard response code or
>> have it available in a (UInt, String) pair from a higher-level web
framework. */
>> case custom(code: UInt, reasonPhrase: String)
>>
>> /* all the codes from
http://www.iana.org/assignments/http-status-codes */
>> case `continue`
>> case switchingProtocols
>> case processing
>> case ok
>> case created
>> case accepted
>> case nonAuthoritativeInformation
>> case noContent
>> case resetContent
>> case partialContent
>> case multiStatus
>> case alreadyReported
>> case imUsed
>> case multipleChoices
>> case movedPermanently
>> case found
>> case seeOther
>> case notModified
>> case useProxy
>> case temporaryRedirect
>> case permanentRedirect
>> case badRequest
>> case unauthorized
>> case paymentRequired
>> case forbidden
>> case notFound
>> case methodNotAllowed
>> case notAcceptable
>> case proxyAuthenticationRequired
>> case requestTimeout
>> case conflict
>> case gone
>> case lengthRequired
>> case preconditionFailed
>> case payloadTooLarge
>> case uriTooLong
>> case unsupportedMediaType
>> case rangeNotSatisfiable
>> case expectationFailed
>> case misdirectedRequest
>> case unprocessableEntity
>> case locked
>> case failedDependency
>> case upgradeRequired
>> case preconditionRequired
>> case tooManyRequests
>> case requestHeaderFieldsTooLarge
>> case unavailableForLegalReasons
>> case internalServerError
>> case notImplemented
>> case badGateway
>> case serviceUnavailable
>> case gatewayTimeout
>> case httpVersionNotSupported
>> case variantAlsoNegotiates
>> case insufficientStorage
>> case loopDetected
>> case notExtended
>> case networkAuthenticationRequired
>> }
>>
>> public enum HTTPMethod {
>> case custom(method: String)
>>
>> /* everything that http_parser.[ch] supports */
>> case DELETE
>> case GET
>> case HEAD
>> case POST
>> case PUT
>> case CONNECT
>> case OPTIONS
>> case TRACE
>> case COPY
>> case LOCK
>> case MKCOL
>> case MOVE
>> case PROPFIND
>> case PROPPATCH
>> case SEARCH
>> case UNLOCK
>> case BIND
>> case REBIND
>> case UNBIND
>> case ACL
>> case REPORT
>> case MKACTIVITY
>> case CHECKOUT
>> case MERGE
>> case MSEARCH
>> case NOTIFY
>> case SUBSCRIBE
>> case UNSUBSCRIBE
>> case PATCH
>> case PURGE
>> case MKCALENDAR
>> case LINK
>> case UNLINK
>> }
>> --- SNAP ---
>>
>> Here's the demo code for a simple echo server
>>
>> --- SNIP ---
>> serve { (req, res) in
>> if req.target == "/echo" {
>> guard req.httpVersion == (1, 1) else {
>> /* HTTP/1.0 doesn't support chunked encoding */
>> res.writeResponse(HTTPResponse(version: req.version,
>> status:
.httpVersionNotSupported,
>> transferEncoding:
.identity(contentLength: 0)))
>> res.done()
>> return .discardBody
>> }
>> res.writeResponse(HTTPResponse(version: req.version,
>> status: .ok,
>> transferEncoding: .chunked,
>> headers:
SomeConcreteHTTPHeaders([("X-foo": "bar")])))
>> return .processBody { (chunk, stop) in
>> switch chunk {
>> case .chunk(let data, let finishedProcessing):
>> res.writeBody(data: data) { _ in
>> finishedProcessing()
>> }
>> case .end:
>> res.done()
>> default:
>> stop = true /* don't call us anymore */
>> res.abort()
>> }
>> }
>> } else { ... }
>> }
>> --- SNAP ---
>>
>> _______________________________________________
>> swift-server-dev mailing list
>> swift-server-dev@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-server-dev


(George) #15

Hello everyone,

I’ve been following this conversation and think we missed something. One of Swift’s main benefits is it’s type system, which is much more expressive than other popular server-side languages (I’m looking at you Go). Now, there are certainly benefits to having a standard “HTTPRequest” type, whether it be a reference or value type. What is more powerful, however, is to allow the user to determine what they expect the structure of a particular Request or Response to be.
Earlier in this thread, I think someone dismissed protocols/generics because “at some point, we need to implement a concrete type”. I don’t agree with this. I think we can leverage the Swift type system to have something much better.

I recommend having protocols for HTTP{Request,Response}{Encodable,Decoder} types, which are driven by a encoder or decoder in the standard library.
For sending requests, you could have:
protocol HTTPRequestEncodable {
  var method: String { get }
  var path: String { get }
  var queryParameters: [String:String] { get }
  …etc
}

When writing to the socket, and ONLY when writing to the socket, these methods will be called to serialize the data. The body may have a slightly more complex API to support streaming bodies (the main issue would be notifying the new driver when new data is available and differentiating between repayable and not-replayable requests).

On the server side, you can have (this is a rough sketch):
protocol HTTPRequestDecoder {
  mutating func processMethod(string)
  mutating func processPathComponent(string)
  mutating fund processHeader(string, string)
  … and so on
}
When reading off the socket, these methods can be called on the actual data which is in the socket’s buffer, with the user deciding how to represent the data. That way, if they only care about “GET”/“POST”, that can be represented as a 2-case enum, or if they care about more exotic methods, they can represent those as well without passing the cost of those exotic methods onto users who just want to use the simple case.

The types implementing these protocols would, of course be in a partial state for some of the decoding part, but that can be remedied with a validated() method (I welcome better names) which is called after processing is done and returns a validated type if all the fields are what the user expects them to be.

I can flesh out the APIs more if people are interested, but I think this is sufficient to communicate the idea, and I’d love to hear more feedback.

There is a performance overhead to using protocols/generics, but I believe these can be effectively mediated by the compiler. Also, this does not preclude the standard library from providing a default HTTPRequest type which can leverage full module optimization to get the same performance as a simple concrete type.

- George


(Johannes Weiss) #16

Hi Colin,

Hi Colin,

>
> Hi Colin,
>
>> Is writing compositional middleware a goal for this iteration of the API? I noticed that the HTTPResponseWriter doesn't appear to allow for, e.g. writing a single header and then passing the writer on (as one would do in, say, an anti-CSRF middleware). Possible I'm missing something though!
>
> Good point! The version 1 actually only had writeHeader(key:value:) and for the sake of having a HTTPResponse type (that could then be used by an HTTP client) I got rid of that. But maybe that was over the top and we should go back to what I had before.
>
> I think having the HTTPResponse is nice. I would suggest these changes:
>
> writeResponse(..., finishHeaders = true)
> writeHeader(...)
> finishHeaders() [-> Bool] // It is an error to call writeHeader after finishHeaders. Calling finishHeaders twice is a no-op. (Return value could differentiates those cases?)
> writeBody(...) // Calls finishHeaders()
>
> Sorry for the brevity of my sketch, let me know if that's unclear in any way.

Thanks, that makes sense! I literally just started an API Sketch v3 document where I put these in so I won't forget.

Just one question about the finishHeaders(): You see this as being mandatory to call, right? At the moment our implementation doesn't have that and it's handled implicitly. Basically if writeBody/writeTrailer/done are called we finish the headers if that hasn't been done yet. In other words, there's a state machine in HTTPResponseWriter which has a headers finished state but it's managed internally.

Do you see any advantages of exposing that explicitly?

Mostly just for smell reasons. Calls to writeHeader are going to fail after writeBody. This can be seen as an additional side effect of writeBody, besides actually emitting the body. Having two entangled effects like this usually leads regret, or people abusing the API.

cool, added to my v3 draft :).

Cheers,
  J

···

On 7 Apr 2017, at 21:54, Colin Barrett <colin@springsandstruts.com> wrote:
On Fri, Apr 7, 2017 at 6:01 AM Johannes Weiß <johannesweiss@apple.com> wrote:
> On 6 Apr 2017, at 22:20, Colin Barrett <colin@springsandstruts.com> wrote:
> On Thu, Apr 6, 2017 at 3:53 AM Johannes Weiß <johannesweiss@apple.com> wrote:

Just to be clear, its fine for one method (writeBody) to invoke the other (finishHeaders), but users should be able to control the timing of each effect should they so choose (for example to help enforce invariants in their own code—"past this point I promise to write no more headers").

It's something that could be done in a wrapper class, so it's not the end of the world if you decide to leave it out.

Cheers,
  Johannes

>
> Thanks,
> -Colin
>
>
> --
> Johannes
>
>>
>> -Colin
>>
>> On Wed, Apr 5, 2017 at 1:36 PM Johannes Weiß via swift-server-dev <swift-server-dev@swift.org> wrote:
>> Hi,
>>
>> First of all, thanks very much to Helge for carefully analysing (and implementing) the first API sketch. As promised, I reworked it a bit.
>>
>> Changes:
>> - extracted HTTPResponse as its own value type (so it could also be used for a client)
>> - added Helge's suggestions:
>> * proper backpressure support
>> * being able to ignore further parts of the HTTP body (stop parameter)
>>
>> If I forgot something important, please let me know. I can't promise that I'll address it before the meeting as I'm in another office tomorrow for meetings and probably won't be able to attend either.
>>
>> Please find everything below...
>>
>> Cheers,
>> Johannes
>>
>> --- SNIP ---
>> /* a web app is a function that gets a HTTPRequest and a HTTPResponseWriter and returns a function which processes the HTTP request body in chunks as they arrive */
>>
>> public typealias WebApp = (HTTPRequest, HTTPResponseWriter) -> HTTPBodyProcessing
>>
>> public struct HTTPRequest {
>> public var method : HTTPMethod
>> public var target : String /* e.g. "/foo/bar?buz=qux" */
>> public var httpVersion : HTTPVersion
>> public var headers : HTTPHeaders
>> }
>>
>> public struct HTTPResponse {
>> public var httpVersion : HTTPVersion
>> public var status: HTTPResponseStatus
>> public var transferEncoding: HTTPTransferEncoding
>> public var headers: HTTPHeaders
>> }
>>
>> public protocol HTTPResponseWriter: class {
>> func writeContinue(headers: HTTPHeaders = HTTPHeaders()) /* to send an HTTP `100 Continue` */
>>
>> func writeResponse(_ response: HTTPResponse)
>>
>> func writeTrailer(key: String, value: String)
>>
>> func writeBody(data: DispatchData) /* convenience */
>> func writeBody(data: Data) /* convenience */
>> func writeBody(data: DispatchData, completion: @escaping (Result<POSIXError, ()>) -> Void)
>> func writeBody(data: Data, completion: @escaping (Result<POSIXError, ()>) -> Void)
>>
>> func done() /* convenience */
>> func done(completion: @escaping (Result<POSIXError, ()>) -> Void)
>> func abort()
>> }
>>
>> public typealias HTTPBodyHandler = (HTTPBodyChunk, inout Bool) -> Void /* the Bool can be set to true when we don't want to process anything further */
>>
>> public enum HTTPBodyProcessing {
>> case discardBody /* if you're not interested in the body */
>> case processBody(handler: HTTPBodyHandler)
>> }
>>
>> public enum HTTPBodyChunk {
>> case chunk(data: DispatchData, finishedProcessing: () -> Void) /* a new bit of the HTTP request body has arrived, finishedProcessing() must be called when done with that chunk */
>> case failed(error: HTTPParserError) /* error while streaming the HTTP request body, eg. connection closed */
>> case trailer(key: String, value: String) /* trailer has arrived (this we actually haven't implemented yet) */
>> case end /* body and trailers finished */
>> }
>>
>> public struct HTTPHeaders {
>> var storage: [String:[String]] /* lower cased keys */
>> var original: [(String, String)] /* original casing */
>> var description: String
>>
>> subscript(key: String) -> [String]
>> func makeIterator() -> IndexingIterator<Array<(String, String)>>
>>
>> init(_ headers: [(String, String)] = [])
>> }
>>
>> public typealias HTTPVersion = (Int, Int)
>>
>> public enum HTTPTransferEncoding {
>> case identity(contentLength: UInt)
>> case chunked
>> }
>>
>> public enum HTTPResponseStatus {
>> /* use custom if you want to use a non-standard response code or
>> have it available in a (UInt, String) pair from a higher-level web framework. */
>> case custom(code: UInt, reasonPhrase: String)
>>
>> /* all the codes from http://www.iana.org/assignments/http-status-codes */
>> case `continue`
>> case switchingProtocols
>> case processing
>> case ok
>> case created
>> case accepted
>> case nonAuthoritativeInformation
>> case noContent
>> case resetContent
>> case partialContent
>> case multiStatus
>> case alreadyReported
>> case imUsed
>> case multipleChoices
>> case movedPermanently
>> case found
>> case seeOther
>> case notModified
>> case useProxy
>> case temporaryRedirect
>> case permanentRedirect
>> case badRequest
>> case unauthorized
>> case paymentRequired
>> case forbidden
>> case notFound
>> case methodNotAllowed
>> case notAcceptable
>> case proxyAuthenticationRequired
>> case requestTimeout
>> case conflict
>> case gone
>> case lengthRequired
>> case preconditionFailed
>> case payloadTooLarge
>> case uriTooLong
>> case unsupportedMediaType
>> case rangeNotSatisfiable
>> case expectationFailed
>> case misdirectedRequest
>> case unprocessableEntity
>> case locked
>> case failedDependency
>> case upgradeRequired
>> case preconditionRequired
>> case tooManyRequests
>> case requestHeaderFieldsTooLarge
>> case unavailableForLegalReasons
>> case internalServerError
>> case notImplemented
>> case badGateway
>> case serviceUnavailable
>> case gatewayTimeout
>> case httpVersionNotSupported
>> case variantAlsoNegotiates
>> case insufficientStorage
>> case loopDetected
>> case notExtended
>> case networkAuthenticationRequired
>> }
>>
>> public enum HTTPMethod {
>> case custom(method: String)
>>
>> /* everything that http_parser.[ch] supports */
>> case DELETE
>> case GET
>> case HEAD
>> case POST
>> case PUT
>> case CONNECT
>> case OPTIONS
>> case TRACE
>> case COPY
>> case LOCK
>> case MKCOL
>> case MOVE
>> case PROPFIND
>> case PROPPATCH
>> case SEARCH
>> case UNLOCK
>> case BIND
>> case REBIND
>> case UNBIND
>> case ACL
>> case REPORT
>> case MKACTIVITY
>> case CHECKOUT
>> case MERGE
>> case MSEARCH
>> case NOTIFY
>> case SUBSCRIBE
>> case UNSUBSCRIBE
>> case PATCH
>> case PURGE
>> case MKCALENDAR
>> case LINK
>> case UNLINK
>> }
>> --- SNAP ---
>>
>> Here's the demo code for a simple echo server
>>
>> --- SNIP ---
>> serve { (req, res) in
>> if req.target == "/echo" {
>> guard req.httpVersion == (1, 1) else {
>> /* HTTP/1.0 doesn't support chunked encoding */
>> res.writeResponse(HTTPResponse(version: req.version,
>> status: .httpVersionNotSupported,
>> transferEncoding: .identity(contentLength: 0)))
>> res.done()
>> return .discardBody
>> }
>> res.writeResponse(HTTPResponse(version: req.version,
>> status: .ok,
>> transferEncoding: .chunked,
>> headers: SomeConcreteHTTPHeaders([("X-foo": "bar")])))
>> return .processBody { (chunk, stop) in
>> switch chunk {
>> case .chunk(let data, let finishedProcessing):
>> res.writeBody(data: data) { _ in
>> finishedProcessing()
>> }
>> case .end:
>> res.done()
>> default:
>> stop = true /* don't call us anymore */
>> res.abort()
>> }
>> }
>> } else { ... }
>> }
>> --- SNAP ---
>>
>> _______________________________________________
>> swift-server-dev mailing list
>> swift-server-dev@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-server-dev


(Helge Heß) #17

Hi,

Earlier in this thread, I think someone dismissed protocols/generics because “at some point, we need to implement a concrete type”.

I don’t think anything was ‘dismissed’. The sole meaning was that there will be a Swift Server module and that this should/will come with a concrete type (so that people can have a minimal running http server out of the box by just using that module). You can still have protocols or whatever you chose.

I think we can leverage the Swift type system to have something much better.

I recommend having protocols for HTTP{Request,Response}{Encodable,Decoder} types

<snip> protocols <snip>

Your `HTTPRequestEncodable` is in a way the reverse of Johannes’ HTTPResponseWriter. The advantage of the latter is that it also doesn’t enforce a specific response type and neither requires a type at all. W/ your approach there always has to be a type implementing the protocol.

I’m not entirely sure why you would call it `HTTPRequestEncodable`. I would just call it `HTTPRequest`? To me an `Encodable` has a method which provides another value (the encoded representation). Like this:

  protocol HTTPRequestEncodable {
    func encodeAsHTTPRequest() -> HTTPRequest
  }

which may still be useful. But extra indirections :slight_smile:

Your `HTTPRequestDecoder` is like a parser delegate, kinda like the peer to the `HTTPResponseWriter`. Could be done like this.
Though if you provide an “Encodable” I suppose the assumption would be that you also provide a “Decodable” too ...

I can flesh out the APIs more if people are interested, but I think this is sufficient to communicate the idea, and I’d love to hear more feedback.

A complete proposal like Johannes’ would be more useful. Or maybe even better a revision of Johannes’ proposal with your ideas in it.

There is a performance overhead to using protocols/generics, but I believe these can be effectively mediated by the compiler.

I don’t see how you would eliminate the protocol overhead cross-module. But I think it should be small enough (after all the proposal also uses Strings ;-).

hh

···

On 9. Apr 2017, at 04:12, George Leontiev via swift-server-dev <swift-server-dev@swift.org> wrote:


(George) #18

I’ve fleshed out my earlier suggestion.

For responses coming back from a server or requests going to a server I like what Helge referred to as a “parser delegate” (naming suggestions welcome) because it allows us to parse the request or response exactly as it comes of the wire, and make no assumptions about how the user wants that data. Also, this API can be used with both an async API (built on top of dispatch) but could also be leveraged to create a synchronous API.

In terms of utility, I like that clients can provide their own HTTPResponseDecoders which can decode to the specific type they are expecting from that endpoint. Servers, on the other hand, can perform their routing logic while processing that path and use the results of that to more appropriately decode the rest of the request.

As I and Helge mentioned, there is overhead to using a protocol cross-module, but the benefit is that we no longer have to have a catch-all type to represent a decoded request or response; this allows users who care about performance to encode their concrete types as efficiently as they can. We can also have a catch-all type provided by the standard library which leaves all the fields as strings (indeed, it can be backed by the actual request data and just point to ranges) and would benefit from full-module optimization. Even better, we can make an opinionated concrete implementation (which can use enums for stata and headers and enforce things like the ContentType header value should always be a MediaType, etc.) but allow specialized applications which need more exotic semantics to fall back to implementing their own type (instead of having `.custom(…)` enum cases)

Note: I specifically omitted the body implementation and we may need something like Johannes’ *Writer API to support sending chunked things.

- George

Code:
// MARK: - Stubs

public protocol HTTPBody {
    // API omitted.
    // There are decisions about how to handle streaming bodies which are outside the scope of this example.
}

// Definition omitted
public typealias HTTPHeader = Void

public struct PathIterator: IteratorProtocol {
    
    func next() -> String {
        fatalError()
    }
    
}

// MARK: - Body Decoder

public protocol HTTPBodyDecoder {
    
    mutating func decodeTrailingHeader(_ key: String, value: String) throws
    
    mutating func decodeBodyChunk(_ data: Data) throws
    
}

// MARK: - Server API

// A type which can be created from an HTTP Request as it is coming off the wire. Suggestions on naming are welcome!
public protocol HTTPRequestDecoder: HTTPBodyDecoder {
    
    associatedtype Request
    
    // Thrown exceptions terminate decoding
    
    // Called before any other methods
    mutating func decodeMethod(_ value: String) throws
    
    // Called after `decodeMethod` and before any other methods. Types conforming to `HTTPRequestDecoder` can use this method to walk the requested path and prepare to handle the rest of the request appropriately. Implementation does not need to exhaust the iterator.
    mutating func decodePath(_ iterator: inout PathIterator) throws
    
    // Called after all of the above methods, and before any methods below.
    mutating func decodeVersion(_ value: String) throws
    
    // Called after all of the above methods and before any methods below. May be called multiple times
    mutating func decodeHeader(_ key: String, value: String) throws
    
    // HTTPBodyDecoder methods conceptually appear here
    
    // Returns an object to be used to represent the request
    func decoded() throws -> Request
    
}

public extension HTTPRequestDecoder {
    
    mutating func decodeVersion(_ value: String) throws {
        // Implementing decodeVersion is not necessary
    }
    
}

/**
A type which can be represented as an HTTP Response
*/
public protocol HTTPResponse {
    var statusCode: UInt { get }
    var headers: [HTTPHeader] { get }
    var body: HTTPBody { get }
}

// MARK: - Client API

/**
A type which can be represented as an HTTP Request
*/
public protocol HTTPRequest {
    var method: String { get }
    
    // Note that the target host a part of an HTTPRequest
    var path: String { get }
    var headers: [HTTPHeader] { get }
    var body: HTTPBody { get }
}

public protocol HTTPResponseDecoder: HTTPBodyDecoder {
    
    associatedtype Response
    
    // Thrown exceptions terminate decoding
    
    // Called before any other methods
    mutating func decodeVersion(_ value: String) throws
    
    // Called after `decodeVersion` and before any other methods
    mutating func decodeStatus(_ value: UInt) throws
    
    // Called after `decodeVersion` and before any methods below
    mutating func decodeStatusText(_ value: String) throws
    
    // Called after all of the above methods and before any methods below. May be called multiple times
    mutating func decodeHeader(_ key: String, value: String) throws
    
    // HTTPBodyDecoder methods conceptually appear here
    
    // Returns an object to be used to represent the response
    func decoded() throws -> Response
    
}

// MARK: - Implementing concrete types.

// We can provide come concrete implementations to handle *most* cases. For instance, a concrete type could only support IANA registered stata (as an enum) while a separate concrete type can allow for custom stata.

// Here is an example for a very simple server

public enum GeorgeMethod: String {
    case get, post
}

public struct GeorgeHTTPRequest {
    
    let status: GeorgeStatus
    
    // Implementation omitted
    
}

// We can provide convenience types/protocols for generating decoders from types like my simple enum type above
public struct GeorgeHTTPRequestDecoder: HTTPRequestDecoder {
    
    // Implementation omitted
    
    func decode() throws -> GeorgeHTTPRequest {
        fatalError()
    }
    
}

···

On Apr 9, 2017, at 4:50 AM, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi,

On 9. Apr 2017, at 04:12, George Leontiev via swift-server-dev <swift-server-dev@swift.org> wrote:

Earlier in this thread, I think someone dismissed protocols/generics because “at some point, we need to implement a concrete type”.

I don’t think anything was ‘dismissed’. The sole meaning was that there will be a Swift Server module and that this should/will come with a concrete type (so that people can have a minimal running http server out of the box by just using that module). You can still have protocols or whatever you chose.

I think we can leverage the Swift type system to have something much better.

I recommend having protocols for HTTP{Request,Response}{Encodable,Decoder} types

<snip> protocols <snip>

Your `HTTPRequestEncodable` is in a way the reverse of Johannes’ HTTPResponseWriter. The advantage of the latter is that it also doesn’t enforce a specific response type and neither requires a type at all. W/ your approach there always has to be a type implementing the protocol.

I’m not entirely sure why you would call it `HTTPRequestEncodable`. I would just call it `HTTPRequest`? To me an `Encodable` has a method which provides another value (the encoded representation). Like this:

protocol HTTPRequestEncodable {
   func encodeAsHTTPRequest() -> HTTPRequest
}

which may still be useful. But extra indirections :slight_smile:

Your `HTTPRequestDecoder` is like a parser delegate, kinda like the peer to the `HTTPResponseWriter`. Could be done like this.
Though if you provide an “Encodable” I suppose the assumption would be that you also provide a “Decodable” too ...

I can flesh out the APIs more if people are interested, but I think this is sufficient to communicate the idea, and I’d love to hear more feedback.

A complete proposal like Johannes’ would be more useful. Or maybe even better a revision of Johannes’ proposal with your ideas in it.

There is a performance overhead to using protocols/generics, but I believe these can be effectively mediated by the compiler.

I don’t see how you would eliminate the protocol overhead cross-module. But I think it should be small enough (after all the proposal also uses Strings ;-).

hh

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(Rien) #19

Hello All,

New on the list, so …

What is the purpose of this HTTP API? why is it proposed?

Was this discussed earlier? I tried the archives, but a pointer as to “when” would be very welcome.

Regards,
Rien

Site: http://balancingrock.nl
Blog: http://swiftrien.blogspot.com
Github: http://github.com/Balancingrock
Project: http://swiftfire.nl - A server for websites build in Swift

···

On 10 Apr 2017, at 03:23, George Leontiev via swift-server-dev <swift-server-dev@swift.org> wrote:

I’ve fleshed out my earlier suggestion.

For responses coming back from a server or requests going to a server I like what Helge referred to as a “parser delegate” (naming suggestions welcome) because it allows us to parse the request or response exactly as it comes of the wire, and make no assumptions about how the user wants that data. Also, this API can be used with both an async API (built on top of dispatch) but could also be leveraged to create a synchronous API.

In terms of utility, I like that clients can provide their own HTTPResponseDecoders which can decode to the specific type they are expecting from that endpoint. Servers, on the other hand, can perform their routing logic while processing that path and use the results of that to more appropriately decode the rest of the request.

As I and Helge mentioned, there is overhead to using a protocol cross-module, but the benefit is that we no longer have to have a catch-all type to represent a decoded request or response; this allows users who care about performance to encode their concrete types as efficiently as they can. We can also have a catch-all type provided by the standard library which leaves all the fields as strings (indeed, it can be backed by the actual request data and just point to ranges) and would benefit from full-module optimization. Even better, we can make an opinionated concrete implementation (which can use enums for stata and headers and enforce things like the ContentType header value should always be a MediaType, etc.) but allow specialized applications which need more exotic semantics to fall back to implementing their own type (instead of having `.custom(…)` enum cases)

Note: I specifically omitted the body implementation and we may need something like Johannes’ *Writer API to support sending chunked things.

- George

Code:
// MARK: - Stubs

public protocol HTTPBody {
    // API omitted.
    // There are decisions about how to handle streaming bodies which are outside the scope of this example.
}

// Definition omitted
public typealias HTTPHeader = Void

public struct PathIterator: IteratorProtocol {
    
    func next() -> String {
        fatalError()
    }
    
}

// MARK: - Body Decoder

public protocol HTTPBodyDecoder {
    
    mutating func decodeTrailingHeader(_ key: String, value: String) throws
    
    mutating func decodeBodyChunk(_ data: Data) throws
    
}

// MARK: - Server API

// A type which can be created from an HTTP Request as it is coming off the wire. Suggestions on naming are welcome!
public protocol HTTPRequestDecoder: HTTPBodyDecoder {
    
    associatedtype Request
    
    // Thrown exceptions terminate decoding
    
    // Called before any other methods
    mutating func decodeMethod(_ value: String) throws
    
    // Called after `decodeMethod` and before any other methods. Types conforming to `HTTPRequestDecoder` can use this method to walk the requested path and prepare to handle the rest of the request appropriately. Implementation does not need to exhaust the iterator.
    mutating func decodePath(_ iterator: inout PathIterator) throws
    
    // Called after all of the above methods, and before any methods below.
    mutating func decodeVersion(_ value: String) throws
    
    // Called after all of the above methods and before any methods below. May be called multiple times
    mutating func decodeHeader(_ key: String, value: String) throws
    
    // HTTPBodyDecoder methods conceptually appear here
    
    // Returns an object to be used to represent the request
    func decoded() throws -> Request
    
}

public extension HTTPRequestDecoder {
    
    mutating func decodeVersion(_ value: String) throws {
        // Implementing decodeVersion is not necessary
    }
    
}

/**
A type which can be represented as an HTTP Response
*/
public protocol HTTPResponse {
    var statusCode: UInt { get }
    var headers: [HTTPHeader] { get }
    var body: HTTPBody { get }
}

// MARK: - Client API

/**
A type which can be represented as an HTTP Request
*/
public protocol HTTPRequest {
    var method: String { get }
    
    // Note that the target host a part of an HTTPRequest
    var path: String { get }
    var headers: [HTTPHeader] { get }
    var body: HTTPBody { get }
}

public protocol HTTPResponseDecoder: HTTPBodyDecoder {
    
    associatedtype Response
    
    // Thrown exceptions terminate decoding
    
    // Called before any other methods
    mutating func decodeVersion(_ value: String) throws
    
    // Called after `decodeVersion` and before any other methods
    mutating func decodeStatus(_ value: UInt) throws
    
    // Called after `decodeVersion` and before any methods below
    mutating func decodeStatusText(_ value: String) throws
    
    // Called after all of the above methods and before any methods below. May be called multiple times
    mutating func decodeHeader(_ key: String, value: String) throws
    
    // HTTPBodyDecoder methods conceptually appear here
    
    // Returns an object to be used to represent the response
    func decoded() throws -> Response
    
}

// MARK: - Implementing concrete types.

// We can provide come concrete implementations to handle *most* cases. For instance, a concrete type could only support IANA registered stata (as an enum) while a separate concrete type can allow for custom stata.

// Here is an example for a very simple server

public enum GeorgeMethod: String {
    case get, post
}

public struct GeorgeHTTPRequest {
    
    let status: GeorgeStatus
    
    // Implementation omitted
    
}

// We can provide convenience types/protocols for generating decoders from types like my simple enum type above
public struct GeorgeHTTPRequestDecoder: HTTPRequestDecoder {
    
    // Implementation omitted
    
    func decode() throws -> GeorgeHTTPRequest {
        fatalError()
    }
    
}

On Apr 9, 2017, at 4:50 AM, Helge Heß via swift-server-dev <swift-server-dev@swift.org> wrote:

Hi,

On 9. Apr 2017, at 04:12, George Leontiev via swift-server-dev <swift-server-dev@swift.org> wrote:

Earlier in this thread, I think someone dismissed protocols/generics because “at some point, we need to implement a concrete type”.

I don’t think anything was ‘dismissed’. The sole meaning was that there will be a Swift Server module and that this should/will come with a concrete type (so that people can have a minimal running http server out of the box by just using that module). You can still have protocols or whatever you chose.

I think we can leverage the Swift type system to have something much better.

I recommend having protocols for HTTP{Request,Response}{Encodable,Decoder} types

<snip> protocols <snip>

Your `HTTPRequestEncodable` is in a way the reverse of Johannes’ HTTPResponseWriter. The advantage of the latter is that it also doesn’t enforce a specific response type and neither requires a type at all. W/ your approach there always has to be a type implementing the protocol.

I’m not entirely sure why you would call it `HTTPRequestEncodable`. I would just call it `HTTPRequest`? To me an `Encodable` has a method which provides another value (the encoded representation). Like this:

protocol HTTPRequestEncodable {
   func encodeAsHTTPRequest() -> HTTPRequest
}

which may still be useful. But extra indirections :slight_smile:

Your `HTTPRequestDecoder` is like a parser delegate, kinda like the peer to the `HTTPResponseWriter`. Could be done like this.
Though if you provide an “Encodable” I suppose the assumption would be that you also provide a “Decodable” too ...

I can flesh out the APIs more if people are interested, but I think this is sufficient to communicate the idea, and I’d love to hear more feedback.

A complete proposal like Johannes’ would be more useful. Or maybe even better a revision of Johannes’ proposal with your ideas in it.

There is a performance overhead to using protocols/generics, but I believe these can be effectively mediated by the compiler.

I don’t see how you would eliminate the protocol overhead cross-module. But I think it should be small enough (after all the proposal also uses Strings ;-).

hh

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev


(James Lei) #20

I would like to know what if we could use bytes to accept input instead of
UInt and String for performance?

···

On Monday, April 10, 2017, George Leontiev via swift-server-dev < swift-server-dev@swift.org <javascript:_e(%7B%7D,'cvml','swift-server-dev@swift.org');>> wrote:

I’ve fleshed out my earlier suggestion.

For responses coming back from a server or requests going to a server I
like what Helge referred to as a “parser delegate” (naming suggestions
welcome) because it allows us to parse the request or response exactly as
it comes of the wire, and make no assumptions about how the user wants that
data. Also, this API can be used with both an async API (built on top of
dispatch) but could also be leveraged to create a synchronous API.

In terms of utility, I like that clients can provide their own
HTTPResponseDecoders which can decode to the specific type they are
expecting from that endpoint. Servers, on the other hand, can perform their
routing logic while processing that path and use the results of that to
more appropriately decode the rest of the request.

As I and Helge mentioned, there is overhead to using a protocol
cross-module, but the benefit is that we no longer have to have a catch-all
type to represent a decoded request or response; this allows users who care
about performance to encode their concrete types as efficiently as they
can. We can also have a catch-all type provided by the standard library
which leaves all the fields as strings (indeed, it can be backed by the
actual request data and just point to ranges) and would benefit from
full-module optimization. Even better, we can make an opinionated concrete
implementation (which can use enums for stata and headers and enforce
things like the ContentType header value should always be a MediaType,
etc.) but allow specialized applications which need more exotic semantics
to fall back to implementing their own type (instead of having `.custom(…)`
enum cases)

Note: I specifically omitted the body implementation and we may need
something like Johannes’ *Writer API to support sending chunked things.

- George

Code:
// MARK: - Stubs

public protocol HTTPBody {
    // API omitted.
    // There are decisions about how to handle streaming bodies which are
outside the scope of this example.
}

// Definition omitted
public typealias HTTPHeader = Void

public struct PathIterator: IteratorProtocol {

    func next() -> String {
        fatalError()
    }

}

// MARK: - Body Decoder

public protocol HTTPBodyDecoder {

    mutating func decodeTrailingHeader(_ key: String, value: String)
throws

    mutating func decodeBodyChunk(_ data: Data) throws

}

// MARK: - Server API

// A type which can be created from an HTTP Request as it is coming off
the wire. Suggestions on naming are welcome!
public protocol HTTPRequestDecoder: HTTPBodyDecoder {

    associatedtype Request

    // Thrown exceptions terminate decoding

    // Called before any other methods
    mutating func decodeMethod(_ value: String) throws

    // Called after `decodeMethod` and before any other methods. Types
conforming to `HTTPRequestDecoder` can use this method to walk the
requested path and prepare to handle the rest of the request appropriately.
Implementation does not need to exhaust the iterator.
    mutating func decodePath(_ iterator: inout PathIterator) throws

    // Called after all of the above methods, and before any methods
below.
    mutating func decodeVersion(_ value: String) throws

    // Called after all of the above methods and before any methods
below. May be called multiple times
    mutating func decodeHeader(_ key: String, value: String) throws

    // HTTPBodyDecoder methods conceptually appear here

    // Returns an object to be used to represent the request
    func decoded() throws -> Request

}

public extension HTTPRequestDecoder {

    mutating func decodeVersion(_ value: String) throws {
        // Implementing decodeVersion is not necessary
    }

}

/**
A type which can be represented as an HTTP Response
*/
public protocol HTTPResponse {
    var statusCode: UInt { get }
    var headers: [HTTPHeader] { get }
    var body: HTTPBody { get }
}

// MARK: - Client API

/**
A type which can be represented as an HTTP Request
*/
public protocol HTTPRequest {
    var method: String { get }

    // Note that the target host a part of an HTTPRequest
    var path: String { get }
    var headers: [HTTPHeader] { get }
    var body: HTTPBody { get }
}

public protocol HTTPResponseDecoder: HTTPBodyDecoder {

    associatedtype Response

    // Thrown exceptions terminate decoding

    // Called before any other methods
    mutating func decodeVersion(_ value: String) throws

    // Called after `decodeVersion` and before any other methods
    mutating func decodeStatus(_ value: UInt) throws

    // Called after `decodeVersion` and before any methods below
    mutating func decodeStatusText(_ value: String) throws

    // Called after all of the above methods and before any methods
below. May be called multiple times
    mutating func decodeHeader(_ key: String, value: String) throws

    // HTTPBodyDecoder methods conceptually appear here

    // Returns an object to be used to represent the response
    func decoded() throws -> Response

}

// MARK: - Implementing concrete types.

// We can provide come concrete implementations to handle *most* cases.
For instance, a concrete type could only support IANA registered stata (as
an enum) while a separate concrete type can allow for custom stata.

// Here is an example for a very simple server

public enum GeorgeMethod: String {
    case get, post
}

public struct GeorgeHTTPRequest {

    let status: GeorgeStatus

    // Implementation omitted

}

// We can provide convenience types/protocols for generating decoders from
types like my simple enum type above
public struct GeorgeHTTPRequestDecoder: HTTPRequestDecoder {

    // Implementation omitted

    func decode() throws -> GeorgeHTTPRequest {
        fatalError()
    }

}

On Apr 9, 2017, at 4:50 AM, Helge Heß via swift-server-dev < > swift-server-dev@swift.org> wrote:

Hi,

On 9. Apr 2017, at 04:12, George Leontiev via swift-server-dev < > swift-server-dev@swift.org> wrote:

Earlier in this thread, I think someone dismissed protocols/generics
because “at some point, we need to implement a concrete type”.

I don’t think anything was ‘dismissed’. The sole meaning was that there
will be a Swift Server module and that this should/will come with a
concrete type (so that people can have a minimal running http server out of
the box by just using that module). You can still have protocols or
whatever you chose.

I think we can leverage the Swift type system to have something much
better.

I recommend having protocols for HTTP{Request,Response}{Encodable,Decoder}
types

<snip> protocols <snip>

Your `HTTPRequestEncodable` is in a way the reverse of Johannes’
HTTPResponseWriter. The advantage of the latter is that it also doesn’t
enforce a specific response type and neither requires a type at all. W/
your approach there always has to be a type implementing the protocol.

I’m not entirely sure why you would call it `HTTPRequestEncodable`. I
would just call it `HTTPRequest`? To me an `Encodable` has a method which
provides another value (the encoded representation). Like this:

protocol HTTPRequestEncodable {
   func encodeAsHTTPRequest() -> HTTPRequest
}

which may still be useful. But extra indirections :slight_smile:

Your `HTTPRequestDecoder` is like a parser delegate, kinda like the peer
to the `HTTPResponseWriter`. Could be done like this.
Though if you provide an “Encodable” I suppose the assumption would be
that you also provide a “Decodable” too ...

I can flesh out the APIs more if people are interested, but I think this
is sufficient to communicate the idea, and I’d love to hear more feedback.

A complete proposal like Johannes’ would be more useful. Or maybe even
better a revision of Johannes’ proposal with your ideas in it.

There is a performance overhead to using protocols/generics, but I believe
these can be effectively mediated by the compiler.

I don’t see how you would eliminate the protocol overhead cross-module.
But I think it should be small enough (after all the proposal also uses
Strings ;-).

hh

_______________________________________________
swift-server-dev mailing list
swift-server-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-server-dev