Announcing WebURL - a new URL type for Swift

Hello Swift community!

For a while now, I've been working on a new URL type for Swift. It has taken a long time, but it's finally at a stage where I'm happy to make a first release. So let me introduce you to WebURL!


WebURL - A new URL type for Swift

WebURL is a brand-new URL library. It is written entirely in Swift, implements the latest URL standard, and has a great API designed to take full advantage of Swift's features.

The repository is at https://github.com/karwa/swift-url

To use WebURL in a SwiftPM project, add the following line to the dependencies in your Package.swift file:

.package(url: "https://github.com/karwa/swift-url", from: "0.1.0"),

And add a dependency on the "WebURL" product.

Additionally, I've developed a prototype port of async-http-client, which allows you to perform http(s) requests using WebURL, and shows how easy it can be to adopt in your library.

Note: This project isn't limited to "web"-related URLs, it also has full support for "file" URLs, "data" URLs, and custom schemes like you might use for app-internal links.

It is called 'WebURL' because a) 'WebURL' is short, b) it has 'URL' in the name, and c) is different enough from Foundation's 'URL' that you hopefully won't confuse them.

A Modern Standard

The history of URLs is messy. There have been several standards over the years, working groups were established and closed due to lack of engagement, people came up with names like "URIs", "IRIs" and even a thing called a "URN"s in order to describe their various ideas for, uhm... URLs. As IBM's Sam Ruby and Adobe's Larry Masinter wrote in an IETF working draft in 2015:

The main problem is conflicting specifications that overlap but don't match each other.
...
From a user or developer point of view, it makes no sense for there to be a proliferation of definitions of URL nor for there to be a proliferation of incompatible implementations. This shouldn't be a competitive feature.

and as Daniel Stenberg (lead developer of CURL) explained on his blog:

A “URL” given in one place is certainly not certain to be accepted or understood as a “URL” in another place.

Not even curl follows any published spec very closely these days, as we’re slowly digressing for the sake of “web compatibility”. There’s no unified URL standard and there’s no work in progress towards that.

Well, the WHATWG's URL Living Standard is a step towards that. The WHATWG is an association of major browser developers, and is lead by representatives from Apple, Google, Mozilla, and Microsoft, who have agreed to align their browsers with the specifications developed at the WHATWG. In fact, the WebKit team have recently been rewriting their URL parser, and the latest Safari preview is now fully aligned with the URL Living Standard.

This library's parser is derived from the reference parser described in the standard, and should be fully compatible with it. It is tested using the same, common Web-Platform-Tests repository used by the major browser vendors to ensure compliance with the standard, as well as hundreds more tests covering the additional APIs.

The WebURL library is entirely portable, having no dependencies other than the Swift standard library itself. Additionally, the parser and URL operations are free from OS-dependent behaviour; one benefit of the URL Living Standard is that all parser quirks, such as handling Windows drive letters or "/" vs "\" separators, work the same way on all platforms.

We will eventually add some OS-dependent behaviour, to convert file URLs to a local file-path, but it will be clearly apart from the rest of the API. We'll leave the actual manipulation of that path to OS libraries like swift-system (e.g. I don't think it's our place to implement something like URL.resolveSymlinksInPath()).

Speed

WebURL is fast. Parsing speed depends a lot on the particular URL, but on my Intel Mac, compared to the existing URL, I've seen improvements from 10%, all the way up to 67%. On a lower-end machine (a Raspberry Pi 4), I've seen orders of magnitude improvement - benchmarks which take 1.9 seconds with the existing URL, but require only ~62 milliseconds with WebURL. Common operations like hashing and checking for equality are consistently more than twice as fast as the existing URL type.

Measuring parsing speed against the existing URL is a bit of an apples-to-oranges comparison because the standards are so different, but it's a reference point people care about. There's still more that can be done to increase WebURL's parsing speed; I have some ideas, and I'll be trying them in the coming months.

API

But perhaps the coolest thing about WebURL is its API. Here are a couple of highlights:

  1. In-place mutation

    If you want to set a property on a WebURL, you don't need to create a URLComponents and change the value and make another URL object; with WebURL, you just set the value:

    var url = WebURL("http://github.com/karwa/swift-url/")!
    
    // Upgrade to https:
    url.scheme = "https"
    url.serialized // "https://github.com/karwa/swift-url/"
    
    // Change the path:
    url.path = "/apple/swift/"
    url.serialized // "https://github.com/apple/swift/"
    

    Because WebURL is a copy-on-write value type, this replacement can happen in-place, without needing to allocate a bunch of intermediate objects.

  2. PathComponents view

    WebURL provides an efficient Collection of the URL's path components, which shares storage with the URL object:

    var url = WebURL("http://github.com/karwa/swift-url/")!
    
    for component in url.pathComponents {
      // component = "karwa", "swift-url", ""
    }
    if url.pathComponents.dropLast().last == "swift-url" {
      // ...
    }
    

    Not only that, but you can also mutate through this view. It's amazing, you have to try it:

    var url = WebURL("file:///swift-url/Sources/WebURL/WebURL.swift")!
    
    url.pathComponents.removeLast() 
    // url = "file:///swift-url/Sources/WebURL"
    
    url.pathComponents.append("My Folder")
    // url = "file:///swift-url/Sources/WebURL/My%20Folder"
    
    url.pathComponents.removeLast(3)
    
    url.pathComponents += ["Tests", "WebURLTests", "WebURLTests.swift"]
    // url = "file:///swift-url/Tests/WebURLTests/WebURLTests.swift"
    

    It also has index-based mutations functions, including a full replaceSubrange, which you can use together with the Collection implementation to perform more complex path manipulation. Again, all of this can happen in-place, so the days when URL operations were much slower than DIY string manipulation or regexes should be a thing of the past.

  3. Query parameters

    WebURL has a similar kind of view for the key-value pairs which are often encoded in a URL's query string. Here, we use @dynamicMemberLookup to create a concise and convenient syntax:

    var url = WebURL("https://example.com/currency/convert?amount=20&from=EUR&to=USD")!
    
    url.formParams.amount // "20"
    url.formParams.to // "USD
    
    url.formParams.amount = "56"
    url.formParams.to = "Pound Sterling"
    // url = "https://example.com/currency/convert?amount=56&from=EUR&to=Pound+Sterling"
    
  4. And more!

    And there's a lot more than that! WebURL comes with a lot of utility functionality necessary for working with URLs, including:

    • pure-Swift IP address types, including parsing and serialisation compatible with inet_aton/pton and inet_ntoa/ntop.
    • Percent-encoding and decoding (both eager and lazy)
    • Host objects and Origins
    • A UTF-8 view, allowing for arbitrary slices of URLs and modifying a URL's components using generic collections of UTF-8 bytes.

Roadmap

The implementation is extensively tested, but the interfaces have not had time to stabilise.

While the package is in its pre-1.0 state, it may be necessary to make source-breaking changes. I'll do my best to keep these to a minimum, and any such changes will be accompanied by clear documentation explaining how to update your code.

Aside from stabilising the API, the other priorities for v1.0 are:

  1. file URL <-> file path conversion.
  2. Converting to/from Foundation.URL.
  3. Benchmarking and optimizing setters, including modifications via pathComponents and formParams views.

For more information, see the project's README, where I go over some of the issues I'll expect to encounter along the way. I know that for a lot of macOS/iOS code, compatibility with URLSession is critical; unfortunately I don't have an excellent answer for that right now. I go over some of the issues in the README, but the short answer is that even if we could convert reliably, round-tripping would break Hashable and Equatable because there's a good chance we'll need to add percent-encoding along the way. So the answer isn't obvious, but I'm going to be trying out a variety of approaches to see what's possible.

But yeah! That's an overview of WebURL. Again, I'd encourage you to check out the Getting Started guide, and the API Reference. If you're using async-http-client, consider experimenting with the WebURL port, or if you're developing another server library in Swift, consider trying out WebURL in a branch, and letting me know how it goes for you.

I'd love to see this library adopted by as many other libraries and applications as possible, so if there's anything I can add to make that easier, please file a GitHub issue or write a post on the Swift forums. I’ll be requesting a sub-forum in the “related projects” section to discuss specific issues and ideas.

Thanks a lot for your time! I hope you're able to make use of WebURL in your projects!

48 Likes

Looks interesting.

I don't see a way to modify the path extension. Like changing myfile.jpg to myfile.zip. I do that a lot with file URLs.

2 Likes

Not yet, but that sounds like a useful thing to add. Thanks for the suggestion.

I suppose we'd want to add something like that as an extension on String, since you can have folders with extensions anywhere in the path (e.g. macOS bundles).

Paths are just so popular. If you take a look at the Foundation.URL API, you'll see that paths are disproportionally represented - probably at least half the API is dedicated to various path operations. Finding the right set of useful functionality without adding OS functionality for file paths has been a challenge.

Why not name the type URI?

I've been looking forward to this since this thread last year. (I actually started using version 0.0.2 a few days before 0.1.0 was announced here.) My main use case is for working with partially percent-encoded URLs, and WebURL works flawlessly.

@Karl explained in the same 1-year old thread here.

2 Likes

Fair enough!

Haha yes, I've been working on it for about a year now in my spare time. I managed to port the reference parser from the standard relatively easily, but no matter how hard I tried, performance was just way off compared with Foundation.

There are a lot of reasons to consider replacing Foundation's URL. We can't really be stuck on the 1994 standard forever, even when the rest of the world has moved on (or is in the process of doing so), and we no longer parse or behave like URLs on other systems. Swift does need a modern URL library, but I didn't want it to come with the caveat that it's 10x slower.

I've had to really dig deep and study the standard and the standard library to meet those performance goals; CFURL's parser has had literally decades of attention, and is written in C (the extra safety checks added by Swift alone create a 20% overhead - it's worth it to avoid UB, but it makes getting this sort of performance exceptionally challenging compared with a language like C). Then the scope increased quite a lot once I decided that any decent URL library would also need to add a full set of APIs for things like percent-encoding and path manipulation, and all of that needed to have very, very comprehensive tests because URLs are full of obscure edge-cases.

So it all took a little longer than expected, but I'm happy with the result. I'm glad to hear that WebURL is working for you!

I'm not completely in love with the name WebURL, but the standard does make a point of standardising on 'URL'. The 'Web' part sort of has meaning, too, since the WHATWG is a web technologies working group, and their overall mission is first and foremost to unify and standardise how web browsers behave.

8 Likes

This looks great. However I am having problems in integrating the package with SPM in my app. It fails with some resolution error. Is this a known issue?

Hi @ENorris and welcome to the forums! Glad to hear you’d like to use this package in your app :slight_smile:

Can you give any more information about the error you’re seeing, and which platform and version of Xcode/Swift you’re seeing it on?

Terms of Service

Privacy Policy

Cookie Policy