Announcing Swift HTTP Types

[Karl]

I appreciate that you guys also don't want the CF dependency; I'm sure that was nobody's first choice.

I've filed a GitHub issue to track it: Drop CF dependency · Issue #10 · apple/swift-http-types · GitHub

@Guoye

This is not the case. CFURLCreateAbsoluteURLWithBytes and CFURLGetBytes are escape hatches which allow clients (e.g. WebKit) to sneak strings which aren't considered valid by URL past URL.

Wow! TIL. That's very subtle, and I don't think it is mentioned in the documentation.

For construction, it seems that we have an NSURL initialiser which forwards to CFURLCAUWB (documentation unfortunately is empty). NSURL is available on all platforms.

NSURL(absoluteURLWithDataRepresentation: Data("https://example.com?q=1|2".utf8), relativeTo: nil) as URL
$R34: Foundation.URL = "https://example.com?q=1|2"
                                               ^

As for the component getters, it seems the behaviour can vary - .host does not perform any escaping but .path and .query do.

let url = (NSURL(absoluteURLWithDataRepresentation: Data("https://ex  ample.com?q=1 2".utf8), relativeTo: nil) as URL)

url.host(percentEncoded: true) 
$R54: String? = "ex  ample.com"
                   ^^
url.query(percentEncoded: true) 
$R55: String? = "q=1%202"
                    ^^^

When I suggested that the Foundation team add these APIs, it was always the intention that they give the raw "unadulterated" components; I never imagined the getter would actually add escaping and it is possible that it was unintentional.

@Karl

I think the biggest, easiest improvements would be to:

1. Add .percentEncodedX properties to URL, so that people don't need to go through URLComponents to avoid URL's automatic percent-decoding. Automatic decoding is really, really, really bad and currently there is no way around it (even converting URL -> URLComponents will automatically percent-decode under certain circumstances, and can corrupt the path).

2. Add a method which provides the URL's string buffer, with ranges of all of the components. This would be seriously useful for URL -> WebURL conversion -- currently, because there are differences between the standards (as well as bugs in URL), we need to request each component individually to ensure the converted URL points to the same location, and each component potentially allocates a String.

I would suggest that either:

1. We implement that behaviour in SCF (where percentEncoded: true returns the raw component), and Darwin changes its implementation to do the same. That means we can use official, public APIs everywhere

   Since "normal" users don't sneak incorrectly-escaped URLs in to Foundation.URL, they should not be affected by this change. We should urgently document that just because a user passed percentEncoded: true, it doesn't mean the resulting string is escaped, and it MUST be sanitised before it is written to the network -- but as I showed above with .host(percentEncoded:), that is already true today; a hostname with unescaped newlines can be trivially exploited if written as part of an HTTP header.

2. We add some kind of SPI which exposes these CF methods, whether via an "official" @_spi or just some underscored methods. We don't need this on Darwin Foundation, which can fall back to CF, so it can be limited to the open-source SCF.

Anyway, we can continue this discussion on the GitHub issue, if you prefer.