WebURL KeyValuePairs API

Related Projects WebURL
,
1

Hi!

In order to make it easier to read/write key-value pairs from URL components, WebURL 0.5.0 will include a new KeyValuePairs type. The current formParams view will be deprecated and removed in the next minor release.

Key-value pairs in URLs ("query parameters") are more complex than most people realise. I've put together a design document which explores the issues and limitations of current APIs, and explains how WebURL.KeyValuePairs attempts to improve on them.

There is a PR which you can try right now (branch name: new-formparams). There are still one or two minor things I'm thinking of tweaking, but broadly-speaking this is what I'm planning for key-value pairs, and why it is that way, and I invite and appreciate your feedback.

The new API is limited to Swift 5.7+, basically because of the new generics syntax. It could be back-ported, but expressing things like some Collection<(some StringProtocol, some StringProtocol)> in the old syntax is so incredibly awkward that I'm hoping to avoid it :sweat_smile:. There's no special runtime requirements, so it's not limited by OS version and can deploy everywhere; it just needs a recent compiler to understand the syntax.

Examples

The design document is a real deep-dive; we go back to first principles to really try understand the problems and how to solve them, and how we can empower Swift applications with better tools than are available anywhere else.

But I think the result is still really easy to use. I've collected some examples from the document to give you an idea of how things are on the user's side.

Basic Reading and Writing

// Example: Reading query parameters.
// You can batch-read up to 4 keys at once (hopefully more once variadic generics arrives)

let url = WebURL("http://example/search?q=quick+recipes&start=10&limit=20")!

let (searchQuery, start, limit) = url.queryParams["q", "start", "limit"]
// ("quick recipes", "10", "20")

// Example: Modifying query parameters.

var url = WebURL("http://example/search")!

url.queryParams["q"] = "quick recipes"
// "http://example/search?q=quick+recipes"
//                       ^^^^^^^^^^^^^^^^

url.queryParams += [
  ("start", "10"),
  ("limit", "20")
]
// "http://example/search?q=quick+recipes&start=10&limit=20"
//                                       ^^^^^^^^^^^^^^^^^^

The KeyValuePairs API scales beyond the query and includes other opaque components, such as the fragment. Here, we parse and modify PDF fragment identifiers using the same rich API as query parameters get:

// Example: Reading and modifying PDF fragment identifiers.

var url = WebURL("http://example.com/example.pdf#page=105&zoom=200")!
let kvps = url.keyValuePairs(in: .fragment, schema: .percentEncoded)

for pair in kvps {
  print(pair.key, " -- ", pair.value)
  // page -- 105
  // zoom -- 200
}

print(kvps["page"])
// "105"

url.withMutableKeyValuePairs(in: .fragment, schema: .percentEncoded) { kvps in
  kvps["zoom"] = "350"
}
// "http://example.com/example.pdf#page=105&zoom=350"
//                                          ^^^^^^^^

Advanced Writing

// Example: Inserting a sort field at a particular location.

var url = WebURL("http://example/students?class=12&sort=name")!

let sortIdx = url.queryParams.firstIndex(where: { $0.key == "sort" }) ?? url.queryParams.endIndex
url.queryParams.insert(key: "sort", value: "age", at: sortIdx)

// "http://example/students?class=12&sort=age&sort=name"
//                                   ^^^^^^^^

// Example: Faceted search.
// For those unfamiliar - this is like toggling a filter on a search UI.
// Like "toggle this brand of shoes" or whatever.

extension WebURL.KeyValuePairs {
  mutating func toggleFacet(name: String, value: String) {
    if let i = firstIndex(where: { $0.key == name && $0.value == value }) {
      removeAll(in: i..., where: { $0.key == name && $0.value == value })
    } else {
      append(key: name, value: value)
    }
  }
}

var url = WebURL("http://example.com/products?brand=A&brand=B")!

url.queryParams.toggleFacet(name: "brand", value: "C")
// "http://example.com/products?brand=A&brand=B&brand=C"
//                                              ^^^^^^^

url.queryParams.toggleFacet(name: "brand", value: "B")
// "http://example.com/products?brand=A&brand=C"
//                                    ^^^

// Example: Filtering tracking parameters.

let trackingKeys: Set<String> = [
  "utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content", /* ... */
]
  
var url = WebURL("http://example/p?sort=new&utm_source=swift.org&utm_campaign=example&version=2")!

url.queryParams.removeAll { trackingKeys.contains($0.key) }
// "http://example/p?sort=new&version=2"

The next example is my personal favorite.

// Example: Take some data from a document (e.g. a configuration file)
// and convert it to URL query parameters -- in one line.

let document = """
- foo: bar
- baz: quxes & quxettes
- formula: 1 + 1 = 2
"""

var url = WebURL("http://example/p")!
url.queryParams += document.matches(of: /- (?<key>.+): (?<value>.+)/).lazy.map { ($0.key, $0.value) }

// "http://example/p?foo=bar&baz=quxes%20%26%20quxettes&formula=1%20%2B%201%20%3D%202"

The reason I really like this example is not because I think it's an especially common thing to do, but because it really shows the advantages of using generics:

String.matches returns an Array of Regex matches, - but that's no problem, because the KeyValuePairs API accepts generic Collections, so we can lazy-map it, dig out the parts of the regex match we need (the captures), and feed that directly to the += operator. And those regex captures? Those are exposed using Substring (not String), but that's also not a problem, because the API accepts generic Collections of generic StringProtocol values.

Other than the lazy map, we don't need to perform any conversions. This code is very efficient - but more importantly, it's really easy to read.

To illustrate that point, let's compare to the equivalent using Foundation's URLComponents:

import Foundation

var url = URLComponents(string: "http://example/p")!
url.queryItems = document.matches(of: /- (?<key>.+): (?<value>.+)/).map {
//                                                                 ^ - copy
  URLQueryItem(name: String($0.key), value: String($0.value))
//                   ^^^^^^                 ^^^^^^ - copies
}
// "http://example/p?foo=bar&baz=quxes%20%26%20quxettes&formula=1%20+%201%20%3D%202"

Foundation's URLQueryItem only works with String, so the key and value from each match needs to be copied in to an individual allocation. Furthermore, URLComponents.queryItems only works with Array, so we also need to make an eager copy of the entire list of matches. Even writing this code is awkward, involving several cycles of the compiler giving us errors and us figuring out how to solve them.

WebURL's version is more intuitive to read and write, and more efficient, and the way it encodes values doesn't have the same ambiguity that Foundation or JavaScript have, so it produces a more robust, interoperable result. And if you want to create, say, a fragment in this way, or customise the encoding, you can do all of that too and it stays readable. I think it's a big improvement across the board.

Higher-Level APIs

This type provides features to support higher-level libraries. For example, it might be nice to encode/decode objects in a Codable-like fashion. That kind of thing is not in-scope for WebURL itself, because it isn't part of the URL Standard (and as mentioned in the design document, there is huge variation and some popular conventions from JavaScript libraries are straight-up invalid), but I do want to provide APIs which allows for those kinds of libraries to be built. Not all applications are expressible in those terms, but for those that are, it can be nice.

Codable itself is not really ideal for it, though. Maybe macros will help us replace it with something simpler. We'll see.

7 Likes