[Pitch] URI Templating

Related Projects Foundation
1

Hi everyone,

Here’s a pitch to add RFC 6570 URI Templating support to the Foundation URL type.

Looking forward to feedback, and thanks for reading.

URI Templating

  • Proposal: SF-NNNNN
  • Authors: Daniel Eggert
  • Review Manager: TBD
  • Status: Draft

Introduction

This adds RFC 6570 URI templates to the URL type.

While there are various kinds and levels of expansion, the core concept is that you can define a template such

as

http://example.com/~{username}/
http://example.com/dictionary/{term:1}/{term}
http://example.com/search{?q,lang}

and then expand these using named values (i.e. a dictionary) into a URL.

The templating has a rich set of options for substituting various parts of URLs. RFC 6570 section 1.2 lists all 4 levels of increasing complexity.

Motivation

RFC 6570 provides a simple, yet powerful way to allow for variable expansion in URLs.

Imagine building applications in Swift that interact with web services or APIs. Often, these interactions rely on constructing URLs to access specific resources. Manually crafting these URLs, especially when they involve dynamic parts like user IDs or search terms, can become messy, error-prone, and hard to maintain.

URI Templates address this challenge by providing a clean and powerful way to define URL patterns with placeholders for variable data. Instead of concatenating strings and manually encoding parameters, you can define a template that clearly outlines the structure of your URL, highlighting the parts that will change based on your application's logic.

An example use case is the server sending a template string to a client, to inform the client how to construct URLs for accessing resources.

Proposed solution

let template = URL.Template("http://www.example.com/foo{?query,number}")
let url = template?.makeURL(variables: [
    "query": "bar baz",
    "number": "234",
])

The RFC 6570 template gets parsed as part of the URL.Template(_:) initializer. It will return nil if the passed in string is not a valid template.

The Template can then be expanded with _variables _ to create a URL:

extension URL.Template {
    public makeURL(
        variables: [URL.Template.VariableName: URL.Template.Value]
    ) -> URL?
}

Read on

You can read the full pitch including the detailed design in the pull request on the swift-foundation repository.

12 Likes
2

Direct pitch link: swift-foundation/Proposals/NNNNN-uri-templating.md at uri-templating-proposal · danieleggert/swift-foundation · GitHub

3

This looks very cool! I was just building some URLs interpolation yesterday that would’ve been easier to deal with if I had URL templates :)

The proposed solution shows try URL.Template("...") but the initializer is fallible and not throwing. Do you intend for the initializer to throw or return nil for invalid templates?

Instead of exposing String(URL.Template.VariableName), did you consider adding a rawValue property on VariableName and making it RawRepresentable?

The terms “text,” “list,” and “associative list” don’t feel very Swifty to me. I think it’s worth mentioning in the discussion for each symbol that they correspond to those concepts in the spec, but I don’t think they should be part of the API. How do you feel about exposing string/array/dictionary static members, along with initializers?

extension URL.Template.Value {
    public static func string(_ value: String) -> URL.Template.Value
    public init(_ value: String)

    public static func array(_ value: some Sequence<String>) -> URL.Template.Value
    public init(_ value: some Sequence<String>)
    
    public static func dictionary(_ value: some Sequence<(key: String, value: String)>) -> URL.Template.Value
    public init(_ value: some Sequence<(key: String, value: String)>)
}

Alternately, why not make this an enum so that people can get the values back out of the Value type? (similarly, why doesn't URL.Template expose its underlying string value?)

These two spellings of makeURL(variables:) feel more natural to me. What other spellings did you consider? Would be interested to hear your thoughts.

extension URL.Template {
    public func interpolating(
        _ variables: [URL.Template.VariableName: URL.Template.Value]
    ) -> URL?
    public func url(
        byInterpolating variables: [URL.Template.VariableName: URL.Template.Value]
    ) -> URL?
}

Under what conditions can the returned URL actually be nil? Are there particular templates or variable values that would make this happen?

It would be cool if there was an API that allowed you to specify a concrete type for the variable names up front so that you can’t accidentally typo a name, but if the point of URL templates is for them to be pulled from an external source rather than written directly in code that would make less sense.

While you did mention in Alternatives Considered that a typical use would be for a server to provide these templates to a client, another place these templates could see use would be on the server, where a set of templates could be used to route incoming requests to a handler. I see that there are some packages in other languages that expose a capability for extracting values out of a URL that matches a template. Would there be interest in exposing something like that here now or in the future?

2 Likes
4

The proposal has this under Future Directions:

Since this proposal covers all of RFC 6570, the current expectation is for it to not be extended further.

What about the so-obvious-I'm-kinda-surprised-I-can't-find-mention-of-it expansion to go from a URL back into the variables used to construct it from a template?

Or in other words, if I can do "Template + Variables → URL", what about the inverse of "URL - Template → Variables" to see if an incoming URL (deep link, route, opened URL, API endpoint, etc) matches a given template and if so, what the variable substitutions would be?

Making symmetric APIs seems like it should be the default expectation…

8 Likes
5

The URL.Template initializer is supposed to be fallible (not throwing). I’ll update the pitch. This aligns it with URL(string:) and also make it easier to chain it with URL.Template.makeURL(variables:). I’ve updated the pitch.

I have no hard feelings about RawRepresentable. My aim was to keep the API surface small. RawRepresentable feels like its mostly for legacy and ObjC compatibility, but I could be wrong. Happy to change that if there’s a general consensus to do so.

As for the terms “text,” “list,” and “associative list”: these are the term of art used in RFC 6570, and it seemed to make sense to use those. It’s true though that string and array might be more familiar. “Dictionary” is slightly problematic because it’s an ordered list of key-value pairs. Could use orderedDictionary, though. Curious to hear what other people think.

As for the naming of makeURL(variables:): This follows the API design guidelines. This is a factory method, and the guidelines suggest the makeURL name.

The makeURL function will return nil if (and only if) URL(string:) returns nil. If you had something like http://example.com:bad%port/, URL(string:) would fail.

I understand your interest in concrete type for the variable names and how that could be interesting in routing code on the server. @davedelong had some similar thoughts / questions. But RFC 6570 is really about constructing URLs. An example is RFC 9652. “The Link-Template HTTP Header Field” . Parsing a URL back into variables is a related, but quite different problem space. I’ll write more below.

1 Like
6

Going from an URL back into variables is a common problem in routing in e.g. web servers such a Hummingbird and Vapor.

I don’t think RFC 6570 lends itself to that problem even if it’s related: Going from an RFC 6570 URL template back to variables could often be ambiguous because variables can be simple strings, arrays, or (ordered) key-value pairs.

The existing routing implementations are a much better fit for this problem space. It’s an interesting thought as to whether Foundation should include such an API, or if it should be part of a more focused package for server implementations. Foundation already provides the URLComponent API for parsing URLs into their constituent parts.

Some (random?) examples for RFC 6570 “URI Templating” are:

7

Thank you for the reply! All of that makes sense.

I guess the future direction here would be for Swift HTTP Types or Vapor/Hummingbird to expose an API for converting whatever routing format they provide to a URI Template so that it can be passed to clients who will then be able to construct a URL matching that route.

Something about this method makes me feel that it isn’t actually a factory method, but I can’t put my feelings into words for some reason. Would be interested to see if others have the same feeling!

8

I appreciate that reversing the operation is outside the scope of RFC 6570, and the proposal is focused on that aspect of things.

I bring it up though because, in my experience, constructing URLs has never been the hard task. We have URLComponents and string interpolation to make that task straight-forward, if not downright easy.

However, the inverse is hard. It's tedious to match incoming URLs from requests or deep links and associate them with the proper logic in the server/app while also extracting relevant bits from the query or path. And given how frequently that seems to be needed, that's the logic that feels appropriate for a framework to take off a programmer's plate.

2 Likes
9

@deggert Your Proposed Solution here seems to be stringly typed. In your proposal there are types like URL.Template.VariableName but again there's no example in the proposal that shows how to use them to maintain type safety. I would expect that the keys in the variables array would be constants, not just raw strings like your examples (query, number). These variables are similar to the keys for NSAttributedString of which there are a finite number provided by the headers. I don't know if there should be a number of standard keys available in this proposal.

10

I would prefer that Template.init?(string:) and Template.makeURL(variables:) -> URL? throw rather than return nil; particularly if Template.init goes wrong, having to figure out what didn't parse by trial and error seems painful.

11

I'd have intuited initialising a URL from a template, like:

let template = URL.Template("http://example.com/~{username}/")
let endpoint = URL(from: template, variables: ["username": "bobjane"])

As I experience templates in other areas of life, once you've made a template, they don't do anything else because they are that by which you make something by, with, or from.

Kiel

3 Likes
12

I had the same feeling as @j-f1 and took up Jed's challenge to put these feelings in to words, but I had not yet seen the "Alternatives considered" section. It just so happens that explaining these feelings bumped into the idea that (at least for me) it was more intuitive as an initialiser on URL.

I'm not sure I buy the discoverability argument agains the URL initialiser. It seems to presume people will think of using the template type rather than the URL type to get a URL object. The argument could be turned the other way, for I could easily type "URL.init(" and get no relevant APIs from the code completion popup.

It seems like templates are a progression of complexity for initialising URLs (the Template type's definition inside the URL type's definition seems to hint at this). So in my opinion (to be honest, that's all I've got to go by) people could be reading about the URL type's documentation first to figure out the ways of instantiating them.

It also seems like the convention within Foundation favour an initialiser on URL. For example, string localisation and formatting.

13

I agree with those that find this pitch rather underwhelming. A Swift implementation of this feature should really allow encoding strongly typed parameter values into the templated URL. For example:

let template = URL.Template("http://example.com/~{username}/resource/~{subID}/~{resourceID}")

struct SomeRequestParameters {
    var username: String
    var subID: SubID
    var resourceID: ResourceID
}

let parameters = SomeRequestParameters(...)
let url = try template.url(encoding: parameters)

In fact, I think we could go even further and use a macro to parse the URL to generate an appropriate, typed accessor.

let template = #URLTemplate("http://example.com/~{username}/resource/~{subID}/~{resourceID}", 
                            parameterTypes: String.self, SubID.self, ResourceID.self)
// Generates encoding(<parameters from template>)
let url = template.encoding(username: ..., subID: ..., resourceID: ...)

I think both of these are actually a good starting point, as the original pitch gives barely anything beyond the string interpolation we already have.

Additionally, it should be possible to use the templates alongside other APIs to generically build requests, but I'll have to give more thought to what that might look like.

8 Likes
14

I hear those who want more here, but I think we should consider that this pitch provides a foundational baseline that can be built on in the future. Let’s not let the perfect be the enemy of the good. Let’s take the win while we have it, and move on from here in a stepwise fashion.

15

There's some curious API design choices here that suggests there's more context required to understand this pitch. For example, it strikes me as a fairly obvious opportunity to use the type system to avoid the fragility of strings and build correct URLs. But that's not the design. So how can we be sure it's even a step in the right direction?

Taking some inspiration from Swift's Data Formatting APIs, I'd have thought a first step could be something more like this:

// http://example.com/~{username}/
let usernameTemplate = URLTemplate(scheme: "https", host: "example.com")
    .path("~\(username)")
let usernameURL = URL(template: usernameTemplate)

// http://example.com/dictionary/{term:1}/{term}
let term = "…"
let dictionaryTemplate = URLTemplate(scheme: "https", host: "example.com")
    .path("dictionary")
    .path(term.prefix(1))
    .path(term)
let pathUrl = URL(template: dictionaryTemplate)

// http://example.com/search{?q,lang}
let lang = "…"
let searchTemplate = URLTemplate(scheme: "https", host: "example.com")
    .path("search")
    .query(item: "q", value: lang)
let searchURL = URL(template: searchTemplate)

Admittedly these are simpler examples so I do not know if it makes implementing the RFC entirely pointless. But, the point I'm trying to make is: how did we end up with the pitch we have to be sure it is a step in the right direction?

16

A few people have pointed out how the API in this pitch is stringly typed and that type-safety could be improved. In re-reading the pitch, I’ve realized that I should have made it more clear as to how this API is (generally) intended to be used.

The itended use is not to replace or improve upon existing string interpolation. The intention is not to build some generic URL templating library for Swift. But rather implement the RFC 6570 standard and how it defines URL templates.

JMAP is an example of typical use of RFC 6570 URI Templates: The server sends the client a downloadUrl, for example

https://jmap.example.com/download/{accountId}/{blobId}/{name}?accept={type}

and the client uses URI templates to expand accountId, blobId, type, and name variables to build URLs to send requests for these resources.

Similarly, The Link-Template HTTP Header Field describes how the server would send an HTTP header field such as

Link-Template: "/books/{book_id}/author"; rel="author"; anchor="#{book_id}"

to inform the client (e.g. web browser) how to build a URL for a particular resource.

In these typical use cases for URL Templates, strongly typed parameter values and/or macros are not an option: all the client gets is a string from the server, and the variable names are defined by the specific protocol.

It’s an interesting thought to be able to expand variables based on a struct, e.g.

struct Parameters {
    var accountId: String
    var blobId: String
    var type: String
    var name: String
}

but I’m not entirely convinced how this adds to type-safety: The accountId, blobId names (strings) are defined by the specific protocol. The code "accountId": myAccountID is no more or less error prone than var accountId: String.

4 Likes
17

As for throwing vs. fallible (init?): The case for fallible is that it follows precedence of URL and URLComponents parsing to return nil on error, and a fallible initializer is more convenient at the call site. There is really very limited error information that can be returned to the caller. I may be biased, but I feel that it’s relatively easy to look at a URL template to see what it wrong, potentially easier than manually parsing some error thrown from the init method.

1 Like
18

As for

let template = URL.Template("http://example.com/~{username}/")
let endpoint = URL(from: template, variables: ["username": "bobjane"])

vs.

let template = URL.Template("http://example.com/~{username}/")
let endpoint = template?.makeURL(variables: ["username": "bobjane"])

I have no strong opinions. I think either way works well. If there’s consensus for the former, I’d be happy to change it.

19

That's a great use case, thanks for calling that out! Things make much more sense.

20

I would go with URL(template:variables:), to match init(string:) and init(resource:).