[Back from revision] Foundation URL Improvements

icharleshu · January 14, 2022, 2:01am

Hi all,

Thank you all so much for the feedback on my original post on Foundation URL Improvements. I gathered so many excellent suggestions. After carefully reading through them and evaluating the pros and cons, I've revised the proposal on the URL improvements.

Here's a brief overview of the changes since the original draft:

Changed URL.currentDirectory to a closure to ensure its value does not change within the closure body.
Added @_alwaysEmitIntoClient to the StaticString initializer so the older OSs can also take advantage of it.
Introduced appending(component:) and appending(queryItem:)
Introduced URL's ExpressibleByStringLiteral conformance
Introduced DirectoryHint (read more about it in the proposal)
Added percentEncodedPath and friends as properties on URL
Changed init(fileURLWithPath: String) and func appendingPathComponent(String) -> URL and friends to soft deprecation

I also want to let everyone know that I've heard the suggestions on some "big picture" improvements loud and clear, including:

Introduce and utilize FilePath in Foundation
Decouple URLResourceValues from URL
Make URL more of a "model" type without dependencies on the file system
Update the parser for URL and URLComponents
Split URL into FileURL and WebURL

All these ideas are very intriguing. I've carefully read each one of them and I will take them into consideration for future improvements. Unfortunately, this proposal has a smaller scope so we will have to hold off on these more ambitious items for now.

Thanks, and please let me know your thoughts on the revision.

URL Enhancements

Proposal: FOU-XXXX
Author(s): Charles Hu
Status: Active review

Revision History

v1 Initial version
v2 Addressed community feedbacks
- Changed URL.currentDirectory to a closure
- Added @_alwaysEmitIntoClient when available
- Introduced appending(component:) and appending(queryItem:)
- (Re)introduced URL's ExpressibleByStringLiteral conformance
- Introduced DirectoryHint
- Added percentEncodedPath and friends
- Added clarification that we are still exploring the possibility of using FilePath
- Changed init(fileURLWithPath: String) and func appendingPathComponent(String) -> URL and friends to soft deprecation

Introduction

URL is one of the most common types used by Foundation for File IO as well as network-related tasks. The goal of this proposal is to improve the ergonomics of URL by introducing some convenience methods and renaming some verbose/confusing ones. Specifically, we proposed to:

Introduce a new StaticString initializer
Introduce a new DirectoryHint type
Refine the existing "file path" initializers
Refine appendingPathComponent and friends
Introduce common directories as static URL properties
Introduce percentEncodedPath and friends

We will discuss each improvement in the following sections.

The New `StaticString` Initializer and `ExpressibleByStringLiteral` Conformance

URL represents two concepts: 1) internet address such as https://www.apple.com initialize by init?(string: String), and 2) file path such as /System/Library/ initialized by init(fileURLWithPath: String). We propose to introduce a new StaticString initializer that could initialize URL for both concepts. This initializer will require the StaticString to contain the URL scheme and initialize Strings with the file:/// scheme as file paths and everything else as web addresses. Because this initializer takes a StaticString known at compile-time, it considers "schemeless" Strings as a programmer error and asserts the existence of a scheme. It will also precondition any invalid web addresses, which mainly includes empty strings and strings with invalid characters. This means malformed web addresses will cause a runtime error for both debug builds AND release builds. Here are some examples:

String Literal	URL Type	URL `absoluteString`
`"https://www.apple.com"`	Web Address	`https://www.apple.com`
`"file:///var/mobile"`	File Path	`file:///var/mobile`
`"www.apple.com"`	-	Assertion failure
`/System/Library/Frameworks"`	-	Assertion failure
`"file://www.ap le.com"`	File Path	`file:///current/dir/www.ap%20le.com`
`"https://www.ap le.com"`	Web Address	Assertion failure

@available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
extension URL {
    /// Initialize an URL with a String literal. This initializer **requires** the
    /// string literal contains a URL scheme such as `file://` or `https://` to correctly
    /// interpret the URL. For web addresses (URLs that doesn't start with the `file` scheme),
    /// it will `precondition` that the addresses are valid.
    @_alwaysEmitIntoClient
    init(_ string: StaticString)
}

This new "universal" StaticString initializer allows us to further add ExpressibleByStringLiteral conformance to URL:

@available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
extension URL : ExpressibleByStringLiteral {
    typealias StringLiteralType = StaticString
    /// Initialize an URL with a String literal. This initializer **requires** the
    /// string literal contains a URL scheme such as `file://` or `https://` to correctly
    /// interpret the URL. For web addresses (URLs that doesn't start with the `file` scheme),
    /// it will `precondition` that the addresses are valid.
    init(stringLiteral value: Self.StringLiteralType)
}

ExpressibleByStringLiteral conformance will further simplify various URL construction sites. For example:

// Move an item
try FileManager.default.moveItem(at: "file:///Dir/File", to: "file:///Dir2")
// URLSession tasks
let (data, _) = try await URLSession.shared.data(from: "https://example.com/file", delegate: nil)

NOTE: We are actively monitoring the pitch on Compile-Time Constant Values. When the const keyword support is ready, we will update the StaticString initializer to const String initializer:

@available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
extension URL {
    /// Initialize an URL with a String literal. This initializer **requires** the
    /// string literal contains a URL scheme such as `file://` or `https://` to correctly
    /// interpret the URL. For web addresses (URLs that doesn't start with the `file` scheme),
    /// it will `precondition` that the addresses are valid.
    init(_ string: const String)
}

The New `DirectoryHint` Type

One of the lesser-known behavior of URL is that it will try to consult the filesystem (aka lstat) to determine whether the URL refers to a directory when you call the variants of methods and file path constructors that do not have the explicit isDirectory parameter. The list of methods include:

init(fileURLWithPath: String)
init(fileURLWithPath: String, relativeTo: URL?)
func appendPathComponent(String)
func appendingPathComponent(String) -> URL

File URLs need to keep track of whether it refers to a directory for numerous reasons. For instance, if you are trying to construct a file URL relative to a base URL, the URL RFC dictates that the last component of the base URL should be stripped off if it's not a directory:

let base = URL(filePath: "/test/file", isDirectory: false)
let url = URL(filePath: "/directory/file2", relativeTo: base)
print(url.path) // prints "/test/directory/file2

However, since this behavior is not widely known and does have performance implications, we propose to change this behavior with the new methods and constructors that we are introducing in this proposal. We also propose to introduce a new type, URL.DirectoryHint, to allow the call site to decide whether a filesystem check should be performed.

extension URL {
    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public enum DirectoryHint {
        case isDirectory        // Specifies that the URL does reference a directory
        case isNotDirectory     // Specifies that the URL does NOT reference a directory
        case checkFileSystem    // Specifies that URL should check with the file system to determine whether it references a directory
        case inferFromPath      // Specifies that URL should infer whether is references a directory based on whether it has a trialing slash
    }
}

The newly introduced methods will use DirectoryHint to specify how to determine whether a URL references a directory, replacing the existing boolean parameter isDirectory. We propose to set inferFromPath as the default value for this parameter. inferFromPath leaves the developers the responsibility to insert the trailing slash in the path to specify that it is a directory. Here are some examples:

URL	`.hasDirectoryPath`
`URL(filePath: "/test/file")`	`false`
`URL(filePath: "/test/directory/")`	`true`
`URL(filePath: "/test/apple", directoryHint: .isDirectory)`	`true`
`URL(filePath: "/test/apple", directoryHint: .isNotDirectory)`	`false`
`URL(filePath: "/System/Library/Frameworks/Foundation.framework", directoryHint: .checkFileSystem)`	`true`

Revisit the `FilePath` Initializers

The new StaticString initializer conflicts with the existing FilePath based initializer because neither has an argument label and FilePath conforms to ExpressibleByStringLiteral. The FilePath initializer is problematic because it could lead to unexpected results. For example, contrary to the assumption that most developer will have, URL("https://www.apple.com") actually initializes a file path (isFileURL returns true) because "https://www.apple.com" is interpreted as a FilePath due to ExpressibleByStringLiteral conformance. We therefore propose to rename the FilePath family initializers to init(filePath: FilePath) and deprecate the original versions.

Similarly, we propose to rename the existing family of file path initializers, init(fileURLWithPath path: String) to init(filePath: String), to slightly improve the ergonomics of these methods due to the more concise naming. This will, of course, conflict with the new FilePath initializer, so we will mark them as @_disfavoredOverload because they require additional transformations to and from FilePath.

extension URL {
    @available(*, deprecated, renamed: "init?(filePath:)")
    public init?(_ path: FilePath)

    @available(*, deprecated, renamed: "init?(filePath:isDirectory)")
    public init?(_ path: FilePath, isDirectory: Bool)

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public init?(filePath: FilePath, directoryHint: DirectoryHint = .inferFromPath)
}

extension URL {
    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public init(filePath: String, directoryHint: DirectoryHint = .inferFromPath, relativeTo: URL? = nil)

    @available(macOS, introduced: 10.9, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(iOS, introduced: 7.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(tvOS, introduced: 9.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(watchOS, introduced: 2.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    public init(fileURLWithPath: String)

    @available(macOS, introduced: 10.9, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(iOS, introduced: 7.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(tvOS, introduced: 9.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(watchOS, introduced: 2.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    public init(fileURLWithPath: String, isDirectory: Bool)

    @available(macOS, introduced: 10.9, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(iOS, introduced: 7.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(tvOS, introduced: 9.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(watchOS, introduced: 2.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    public init(fileURLWithPath: String, relativeTo: URL?)

    @available(macOS, introduced: 10.9, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(iOS, introduced: 7.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(tvOS, introduced: 9.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    @available(watchOS, introduced: 2.0, deprecated: 100000.0, message: "Use init(filePath:directoryHint:relativeTo:) instead")
    public init(fileURLWithPath: String, isDirectory: Bool, relativeTo: URL?)
}

We propose to "soft deprecate" the old constructors init(fileURLWithPath) and friends because they are widely used and the new constructor init(filePath:directoryHint:relativeTo:) will have slightly different behavior due to DirectoryHint. This means we will mark the deprecation version to 100000.0 instead of a concrete version.

Revisit `appendingPathComponent` and Friends

We also propose to rename the (painfully) long, but popular appendingPathComponent() and friends to .appending(path:), as well as adding a few additions:

.appending(paths:): appends a collection of path components.
.appending(component:): appends a single component. It percent-encodes the String before appending.
.appending(queryItem:): appends a URLQueryItem to the URL.

Similarly to the old constructors, we propose to soft deprecate the old methods since they are also widely used.

extension URL {
    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public mutating func append(path: String, directoryHint: DirectoryHint = .inferFromPath)

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public func appending(path: String, directoryHint: DirectoryHint = .inferFromPath) -> URL

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public mutating func append<C: Sequence>(paths: C, directoryHint: DirectoryHint = .inferFromPath) where C.Element : StringProtocol

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public func appending<C: Sequence>(paths: C, directoryHint: DirectoryHint = .inferFromPath) where C.Element : StringProtocol -> URL

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public mutating func append(component: String, directoryHint: DirectoryHint = .inferFromPath)

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public func appending(component: String, directoryHint: DirectoryHint = .inferFromPath)

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public mutating func append(queryItem: URLQueryItem)

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public func appending(queryItem: URLQueryItem)

    @available(macOS, introduced: 10.9, deprecated: 100000.0, message: "Use append(path:directoryHint:) instead")
    @available(iOS, introduced: 7.0, deprecated: 100000.0, message: "Use append(path:directoryHint:) instead")
    @available(tvOS, introduced: 9.0, deprecated: 100000.0, message: "Use append(path:directoryHint:) instead")
    @available(watchOS, introduced: 2.0, deprecated: 100000.0, message: "Use append(path:directoryHint:) instead")
    public mutating func appendPathComponent(String)

    @available(macOS, introduced: 10.9, deprecated: 100000.0, message: "Use append(path:directoryHint:) instead")
    @available(iOS, introduced: 7.0, deprecated: 100000.0, message: "Use append(path:directoryHint:) instead")
    @available(tvOS, introduced: 9.0, deprecated: 100000.0, message: "Use append(path:directoryHint:) instead")
    @available(watchOS, introduced: 2.0, deprecated: 100000.0, message: "Use append(path:directoryHint:) instead")
    public mutating func appendPathComponent(String, isDirectory: Bool)

    @available(macOS, introduced: 10.9, deprecated: 100000.0, message: "Use appending(path:directoryHint:) instead")
    @available(iOS, introduced: 7.0, deprecated: 100000.0, message: "Use appending(path:directoryHint:) instead")
    @available(tvOS, introduced: 9.0, deprecated: 100000.0, message: "Use appending(path:directoryHint:) instead")
    @available(watchOS, introduced: 2.0, deprecated: 100000.0, message: "Use appending(path:directoryHint:) instead")
    public func appendingPathComponent(String) -> URL

    @available(macOS, introduced: 10.9, deprecated: 100000.0, message: "Use appending(path:directoryHint:) instead")
    @available(iOS, introduced: 7.0, deprecated: 100000.0, message: "Use appending(path:directoryHint:) instead")
    @available(tvOS, introduced: 9.0, deprecated: 100000.0, message: "Use appending(path:directoryHint:) instead")
    @available(watchOS, introduced: 2.0, deprecated: 100000.0, message: "Use appending(path:directoryHint:) instead")
    public func appendingPathComponent(String, isDirectory: Bool) -> URL
}

The addition of .appending(paths:) will make appending multiple path components to URL simpler. For example:

let baseURL: URL = ...
let id = UUID().uuidString

// Before
let photoURL = baseURL
    .appendingPathComponent("Photos")
    .appendingPathComponent("\(id.first!)")
    .appendingPathComponent(id)

// After
let photoURL = baseURL.appending(paths: ["Photos", "\(id.first!)", id])

Common Directories as Static URL Properties

We propose to add all "get URL" style methods from FileManager to URL as static methods. This allows the call site to use Swift's powerful static member lookup to get the URLs to predefined directories instead of always needing to spell out FileManager.default. We also propose to add a few more static directory URLs that correspond to FileManager.SearchPathDirectory. Note that these are not stored properties and will not run in O(1). We will add documentations to specify that.

@available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
extension URL {
    /// The working directory of the current process.
    /// This properties has been implemented as a closure
    /// to ensure the value of `currentDirectory` does not change
    /// within the `body`
    /// Calling this property will issue a `getcwd` syscall.
    public static func withCurrentDirectory<R>(_ body: (URL) -> throws R) rethrows -> R
    /// The home directory for the current user (~/).
    /// Complexity: O(1)
    public static var homeDirectory: URL
    /// The temporary directory for the current user.
    /// Complexity: O(1)
    public static var temporaryDirectory: URL
    /// Discardable cache files directory for the
    /// current user. (~/Library/Caches).
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var cachesDirectory: URL
    /// Supported applications (/Applications).
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var applicationDirectory: URL
    /// Various user-visible documentation, support, and configuration
    /// files for the current user (~/Library).
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var libraryDirectory: URL
    /// User home directories (/Users).
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var userDirectory: URL
    /// Documents directory for the current user (~/Documents)
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var documentsDirectory: URL
    /// Desktop directory for the current user (~/Desktop)
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var desktopDirectory: URL
    /// Application support files for the current
    /// user (~/Library/Application Support)
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var applicationSupportDirectory: URL
    /// Downloads directory for the current user (~/Downloads)
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var downloadsDirectory: URL
    /// Movies directory for the current user (~/Movies)
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var moviesDirectory: URL
    /// Music directory for the current user (~/Music)
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var musicDirectory: URL
    /// Pictures directory for the current user (~/Pictures)
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var picturesDirectory: URL
    /// The user’s Public sharing directory (~/Public)
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var sharedPublicDirectory: URL
    /// Trash directory for the current user (~/.Trash)
    /// Complexity: O(n) where n is the number of significant directories
    /// specified by `FileManager.SearchPathDirectory`
    public static var trashDirectory: URL
    /// Returns the home directory for the specified user.
    public static func homeDirectory(forUser user: String) -> URL?

    /// Locates and optionally creates the specified common directory in a domain.
    ///
    /// - parameter directory: The search path directory. The supported values are
    ///     described in FileManager.SearchPathDirectory.
    /// - parameter domain: The file system domain to search. The value for this
    ///     parameter is one of the constants described in
    ///     `FileManager.SearchPathDomainMask`. You should specify only one domain for
    ///     your search and you may not specify the allDomainsMask constant for
    ///     this parameter.
    /// - parameter url: The file URL used to determine the location of the returned
    ///     URL. Only the volume of this parameter is used. This parameter is ignored
    ///     unless the directory parameter contains the value
    ///     `FileManager.SearchPathDirectory.itemReplacementDirectory` and the domain
    ///     parameter contains the value `userDomainMask`.
    /// - parameter shouldCreate: Whether to create the directory if it does not
    ///     already exist.
    public init(
        for directory: FileManager.SearchPathDirectory, 
        in domain: FileManager.SearchPathDomainMask, 
        appropriateFor url: URL?, 
        create shouldCreate: Bool) throws
}

Now code for common file tasks such as writing files to the Downloads directory will be much cleaner:

let secretData = ...

// Before
let downloadDirectoryURL = FileManager.default.urls(for: .downloadsDirectory, in: .userDomainMask)[0]
let targetFolderURL = downloadDirectoryURL.appendingPathComponent("TopSecrets")
try FileManager.default.createDirectory(at: targetFolderURL, withIntermediateDirectories: true, attributes: nil)
let secretDataURL = targetFolderURL.appendingPathComponent("Secret.file")
try secretData.write(to: secretDataURL)

// After
let targetFolderURL: URL = .downloadsDirectory.appending(path: "TopSecrets")
try FileManager.default.createDirectory(at: targetFolderURL, withIntermediateDirectories: true, attributes: nil)
try secretData.write(to: targetFolderURL.appending(path: "Secret.file"))

`percentEncodedPath` and Friends

URL currently does not offer percentEncodedXX properties such as percentEncodedPath and percentEncodedHost, forcing the developers to fallback to URLComponents. We propose to add these missing properties:

extension URL {
    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public var percentEncodedFragment: String? { get }

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public var percentEncodedHost: String? { get }

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public var percentEncodedPassword: String? { get }

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public var percentEncodedPath: String { get }

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public var percentEncodedQuery: String? { get }

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public var percentEncodedUser: String? { get }
}

Impact on Existing Code

Very minimal. Most changes introduced are additive with a few additional method renames.

Alternatives Considered

Use `FilePath` for File IO Tasks

Swift System introduced FilePath to represent a location on the file system, which functionally overlaps with URL for File IO tasks. Although FilePath is more lightweight and has better cross-platform support, replacing URL with FilePath for File IO tasks in Foundation could lead to two major issues:

Bigger impact on existing code. Callers of FileManager will need to be updated to use FilePath.
URL supports resource value caching, a feature that many systems depends on. FilePath will need to support fetching and caching URLResourceValues before it could replace URL for file tasks.

Please note that although we are not considering FilePath for this proposal, we are not "shutting down" the possibility of using FilePath in Foundation. We will explore Foundation's usage for FilePath in the future.

Uniformed `append` Method with `AppendingType`

Since we have several append methods, we could also introduce a new type, tentatively called URL.AppendingType, that allows us to have a more "uniformed" append experience with one append method and maybe even a custom operator!

extension URL {
    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public enum AppendingType {
        case path(String)               // appends the String as a path, treating `/` as directory separators
        case component(String)          // appends the **percent encoded** String as a single component
        case components([String])       // appends an array of **percent encoded** Strings
        case pathExtension(String)      // appends the String as a path extension
        case queryItem(URLQueryItem)    // appends an URLQueryItem
    }

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public func appending(_ type: AppendingType, hint: DirectoryHint = .inferFromPath)

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public static func +(lhs: URL, rhs: AppendingType) -> URL
}

Here are some examples:

let base = URL("file:///System/Library/Frameworks")
let infoURL = base + .component("Foundation") + .pathExtension(".framework") + .path("Resources/Info.plist")
// Alteratively
let infoURL = base.appending(.components(["Foundation.framework", "Resources", "Info.plist"]))

We decided to not move forward with this approach for several reasons:

The downside of having to maintain a brand new type AppendingType out weights the value it provides.
We want to nudge developers into explicitly providing DirectoryHint, but the + operator does not give developers that ability.

tera · January 14, 2022, 2:42am

Thanks Charles, I like the new version overall. Few suggestions:

DirectoryHint - is a great addition.
Name wise - "hint" suggests (to me) it can be ignored. "We listened to your suggestion that this is a file but we decided it is a directory" - which is not what it will do (i hope). Though I can't think of a better name ATM.

inferFromPath - like that.

can we rename isNotDirectory to isFile or do you have some use case in mind when it is neither a directory nor a file?

checkFileSystem is probably ok for now to match the current behaviour, we can probably deprecate it later when there's a better separation between "URL model type" and file system.

.appending(paths:)

Is .appending(components:) better name for .appending(paths:) ?

withCurrentDirectory:

URL.withCurrentDirectory { cwd in
    foo(cwd)
    bar(cwd)
}

I don't see how that's different (or much better) compared to:

let cwd = URL.currentDirectory
foo(cwd)
bar(cwd)

The issue with cwd is that it exists... Probably ok to have API in the normal form like other static vars.

OTOH, making it more cumbersome compared to others has its own merit of soft discouraging people using it...

As a minor alternative you may consider having func versions of the standard folder locations with parameters like domain, etc defaulted to some values. That's if there is a value of having access to, say, /Library/Preferences in addition to user preferences or whatever reason domain parameter existed before. This is in addition or instead of vars. note you can have both if needed:

var foo: X
func foo(param: Y = ...) -> X

Or those other domain locations could be exposed as other constants, don't know which is better. This is absolutely minor, methinks.

Thank you!

Karl · January 14, 2022, 10:02am

Looks good! Some comments:

String Literals

What exactly does an "invalid web address" mean? For example, https:///apple.com is technically a valid URL according to RFC-2396, but the HTTP spec says this is invalid and it will fail at request-time. Since this is about string literals, there's an argument that it would be good to catch obvious misspellings like that early.

Also, some URLs may be valid but ill-formatted (such as uppercase schemes, e.g. "HTTPS://apple.com"). RFC-2396 says schemes are supposed to be lowercase, and again, since this is about string literals, there may be room to be stricter:

Scheme names consist of a sequence of characters beginning with a
lower case letter and followed by any combination of lower case
letters, digits, plus ("+"), period ("."), or hyphen ("-"). For
resiliency, programs interpreting URI should treat upper case letters
as equivalent to lower case in scheme names (e.g., allow "HTTP" as
well as "http")

rfc2396

DirectoryHint

Looks like a nice improvement!

Still, I'm not sure if it's a good idea to even have blocking filesystem calls at all, even as an option - local drives are relatively fast these days, but POSIX OSes allow you to map network resources and other things in to a shared, singular filesystem. That means a blocking FS call really has completely unpredictable performance.

So what could a better solution look like? Well, with static member syntax, we can use leading-dot syntax to resolve generic types:

func someFunc<T>(_: T) {}

someFunc(.foo) // calls someFunc<Foo>(_:)
someFunc(.bar) // calls someFunc<Bar>(_:)

If we had something like rethrowing protocol conformances but for async, theoretically we could make .is(Not)Directory and .inferFromPath be synchronous, while .checkFileSystem would be async.

Obviously that relies on language features which do not currently exist. Still, it's worth thinking about - how would this API be extended to support such a feature if were added to the language? Blocking filesystem calls are so bad that it would be really worth doing that as soon as it became possible. It's the kind of thing where I think developers shouldn't even have an option; why have an option to run with scissors?

Nit: There's a typo here: url contains an absolute path (leading /), so this example actually prints "/directory/file2".

FilePath initializers

Presumably, the same is true for FilePath.init?(URL) - that FilePath("file:///usr/bin/swift") would actually resolve to FilePath.init(String) and interpret the entire thing as some kind of file path?

It may be more of an edge case, but I'd recommend adding a url: label to that initialiser as well, just to make sure that things are clear at the call site. These sorts of overlapping overloads with literal arguments are generally not a good thing IMO.

AppendingPathComponents Changes

Looks good! I like that there's the ability to add a single component!

URLs have one path (that's why the property is called .path), so pluralising it reads a bit strangely to me. I think the real win here is the ability to append multiple components in a single call, right? Being able to avoid this:

let photoURL = baseURL
    .appendingPathComponent("Photos")
    .appendingPathComponent("\(id.first!)")
    .appendingPathComponent(id)

I'm not sure dropping the word 'components' from the argument label is really necessary.

Common Directories

I think this needs more explanation. The "current working directory" is the filesystem path against which relative filesystem paths are resolved, but file URLs are absolute. My concern in the previous thread was that the working directory is (a) not predictable and (b) often unsafe to access (it's just a mutable global, not even a thread-local - so it could, for example, tear if one thread is setting it while another thread simultaneously tries to read it).

Is the idea that this should accept a URL as an argument, which then affects how relative paths are resolved by FileManager and/or URL.init(FilePath)?

Also, the closure should be async if people are expected to perform file operations within it.

Swift globals can be lazily evaluated and cached, so it seems like they could be O(1) on average. None of these strike me as things which can change during the life of a process, so caching should be fine AFAICT.

PercentEncodedPath and co.

OMG Yes!

Currently, for Foundation.URL -> WebURL conversion, my approach is to verify every component. But this has hit a snag because there is literally no way to get an accurate path from URL. So these properties would be a big help.

Just having this would be really great. If I could ask for more (and these opportunities do not come up very often, so I will), I'd really like to get component ranges for use with the absoluteString. Currently, the conversion process spends ~65% of its time just calling getters from URL (which it only calls once).

Surely that will improve by avoiding percent-decoding, but I'd really like to avoid extra string allocations and bridging overheads. For comparison, getting the equivalent components as slices of WebURL's UTF8View takes a negligible amount of time (<1%), precisely because it can avoid those things.

Summary

Overall, some really good changes!

P.S.

Please - not that name (I know it's not being proposed).

hisekaldma · January 14, 2022, 10:21am

icharleshu:

The addition of .appending(paths:) will make appending multiple path components to URL simpler. For example:

let baseURL: URL = ...
let id = UUID().uuidString

// Before
let photoURL = baseURL
    .appendingPathComponent("Photos")
    .appendingPathComponent("\(id.first!)")
    .appendingPathComponent(id)

// After
let photoURL = baseURL.appending(paths: ["Photos", "\(id.first!)", id])

IIUC, appending(path:) and appending(paths:) don’t percent-encode, but appending(component:) does? That sounds non-obvious and like a footgun to me. Aside from the fact that it reads strangely since a URL doesn’t have multiple paths (as @Karl has pointed out), it would be really easy to miss that this will not do what you expect it to:

let artistName = "AC/DC"
let albumsURL = baseURL.appending(paths: ["artists", artistName, "albums"])

It would make more sense to me to have appending(component:) and appending(components:) which percent-encode, and something like appending(path:) which doesn’t.

aerobounce · January 14, 2022, 10:37am

The changes are basically absolutely great work while retaining compat.
I like the elegant DirectoryHint and new initializers. It's also great to hear future directions.

One thing, is there any particular reason variadic parameters should be avoided? (I guess I don't see it often in Swift stdlib nor in the other areas of Foundation):

extension URL {
    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public mutating func append<S: StringProtocol>(paths: S..., directoryHint: DirectoryHint = .inferFromPath)

    @available(macOS 9999, iOS 9999, tvOS 9999, watchOS 9999, *)
    public func appending<S: StringProtocol>(paths: S..., directoryHint: DirectoryHint = .inferFromPath)
}

let photoURL = baseURL.appending(paths: "Photos", "\(id.first!)", id)

Edit: Personally improvements regarding lstat is very, very welcome and thank you.

dlbuckley · January 14, 2022, 2:20pm

Looking great!

I agree with most of what been said above but want to add something additional regarding the appends:

It's quite a common operation to add one or more query items, is there a reason a .appending(queryItems: [URLRequeryItem]) was not added?

rhx · January 14, 2022, 11:27pm

I agree with what most people said – this looks really good. One question regarding the methods that take string (particularly the appending() family). Would it make sense to make them templated, i.e. instead of hard coding String to use something like <S: StringProtocol>, avoiding having to do explicit string conversions (as in your UUID path example)?

I understand that this isn't necessarily something we want for every API that uses strings, but given the strong relationship between URL components and strings (i.e. the expectation is that a lot of URLs will be computed from components represented as, e.g., Substrings), this might be a useful addition.

benrimmington · January 15, 2022, 12:12am

According to SE-0213:

URL("...") will use init(stringLiteral:)
URL.init("...") will use init(_:)

So maybe the unlabelled initializer isn't needed?
Or could it be repurposed for non-literal strings and substrings?

When given a "file:///" scheme, do these new initializers use directory hinting?

According to the documentation and open-source implementation, FilePath normalizes itself by removing any trailing separators, so the DirectoryHint.inferFromPath option may not be valid.

xwu · January 17, 2022, 6:10am

Note that once URL itself conforms to ExpressibleByStringLiteral, this will no longer be true: writing URL("https://www.apple.com") will be exactly the same as writing "https://www.apple.com" as URL.

Maybe I'm missing something, but FilePath("file:///usr/bin/swift") is equivalent to "file://usr/bin/swift" as FilePath, and this seems actively desirable since the two forms are often confused by users (the reason why this was made a special-cased rule in the language to begin with).

Karl · January 17, 2022, 10:43am

"file:///usr/bin/swift" is not a file path, so this would conflict with FilePath's own string literal support.

And having FilePath's string literal initialisers support both URLs and file paths would be ambiguous on Windows: in "x:/foo" as FilePath , is x: a Windows drive letter or a URL scheme? If the former, this should succeed; if the latter, this should fail (because you clearly cannot create a file path from a non-file URL).

IMO, StaticString initialisers aren't really worth the fuss. I know I suggested ExpressibleByStringLiteral in the previous pitch, but that was more of a "if you've decided to do A, you might as well do B" comment; not an endorsement that I think A is actually a good idea in the first place.

I did, in fact, consider it a long time ago for WebURL, but I don't really think it's what people really want -- it looks like a feature that people have occasionally asked for (compile-time checking), but it really isn't. It looks so close that IMO, it's misleading to the point where it could actually be harmful.

What's the use-case?

Firstly, it is very rare that you make a request directly from a pure string literal. I can't remember the last time I saw code which actually looked like this:

URLSession.shared.data(from: "https://example.com/file", delegate: nil)

Most of the time, you will be writing something like this instead. I'm not saying there is no benefit in any of these cases, but let's be honest... it's marginal at best.

var endpoint = URL("https://api.example.com/...")
var endpoint = "https://api.example.com/..." as URL
var endpoint: URL = "https://api.example.com/..."

The most compelling use-case I can think of, where people might actually be making requests from string literals is in scripts. Building a request API optimised for scripting is interesting, but I think it probably wouldn't look much like URLSession, and would probably include an overload for specifying URLs as non-static Strings anyway (script-centric APIs are all about getting the type system out of the way for the sake of convenience).

But doesn't it help you write correct code?

So what do we actually gain? Basically, the ability to omit the ! force-unwrap. And to be clear - as mentioned in the pitch, it does not remove any runtime errors or crashes, or even catch a single bad URL earlier than you could today with the force-unwrap. It just hides them. That's literally all it does.

There are Swift developers who make it their mission to avoid all force-unwraps, even though the community has repeatedly discussed it and found that whole philosophy to be a fundamental misunderstanding of the language. It tends to come from the idea that they want their apps to never crash - which is understandable, of course nobody wants their apps to crash. But lots of functionality in Swift can fail at runtime (e.g. accessing an invalid Array index), and the only way to fully guard against it is to test your code (as it always was). Stuff like this will still fail at runtime if you don't test all code paths:

let endpoint: URL
if userLocaleIsFrench {
  endpoint = "https://api.example.com/..."
} else {
  endpoint = "https://api.fr .example.com/..." // whoops! typo!
}

Again - this feature would just hide the potential crash, not remove it. Is that really a better situation? Some might say yes, others may disagree; what I think we should be able to accept is that the benefits (if there are any) are not obvious. It is not a clear improvement. Is it obvious by looking at that code that it will fail at runtime?

It is true - things that are ExpressibleByStringLiteral, or which include non-failable StaticString initialisers, do not need to accept every string value, and it is within the protocol's semantics for them to fail at runtime. However - the reason people ask for this feature is because they want a particular failure behaviour, which this initialiser can't and won't provide. They do sometimes find typos in URLs, and they want those to be flagged even in branches which are not actually tested (which... , really - there's no substitute for tests). That is something that only build-time checking can provide. This thing gives you a false sense of security; it's just as misguided as those who proclaim that all force-unwraps are evil or "code smells".

FWIW, the approach I'm planning to take for WebURL is to create a linting plugin. It hasn't been a priority so far, but my idea is to either create one of the new-fangled command plugins, or add some kind of integration with an existing linting tool. It's not compile-time checking, so it still requires an extra command/tool, but it is build-time checking, so IMO it's closer to what people actually want.

xwu · January 17, 2022, 1:33pm

Sorry, I'm now very confused, or we are talking past each other. I thought we were talking about FilePath's own string literal support. What is the conflict of which you refer?

It is desirable when a string literal is passed to FilePath(...) that it would be parsed according to FilePath's rules, and when a string literal is passed to URL(...) that it would be parsed according to URL's rules. If "file:///usr/bin/swift" is not a file path, then an error should arise when someone writes FilePath("file:///usr/bin/swift"). This, I believe, is what already happens with FilePath at this present moment in time and if this proposal is adopted as-is—am I incorrect in that?

I am not suggesting that we should make FilePath support anything it does not currently. Since you do bring it up, by the by, I'm not sure I would oppose making it understand "file:///...", as it's unlikely there would be any broken behaviors with an added code path in the corresponding FilePath initializer that's basically if string.hasPrefix("file:///") { self.init(URL(string)); return }. But to be clear, this is not what I've been talking about.

No, not only that. If it were only that, then I too would find the idea of conforming to ExpressibleBy... more take-it-or-leave-it.

What we gain is more crucially, in my opinion, the advantage I discuss above—i.e., that passing a string literal to URL in any of the syntaxes you list (be it ... as URL or URL(...) or let x: URL = ...) causes the string literal to be parsed as per URL's rules, regardless of what other overloads exist or may be added in the future. With ExpressibleByStringLiteral conformance, this becomes an ironclad guarantee that users can lean on without shenanigans with @_disfavoredOverload or other trickery.

In essence, I’m arguing that even though ExpressibleBy... was originally conceived as a “bag of syntax” protocol to support a language feature, which we now try to avoid, with subsequent evolution we’ve endowed it with the narrow but crucial semantic significance that I describe above, which URL could benefit from both because of the many converting initializers that clash and because it actually meets this semantic requirement; namely—

When a type Foo is ExpressibleByBarLiteral, then a user is correct to expect a Foo to be directly initialized from a bar literal without (notionally) conversion via an intermediate type and thus lossiness due to what such an intermediate type can represent.

[See pitch for StaticBigInt for how we will make this come to fruition for third-party types expressible by integer literals; there is also a longstanding extant bug with respect to ExpressibleByFloatLiteral since it falls short in this respect.]

I don't disagree entirely with your point in that it correctly identifies a common misunderstanding, but nor can I agree with it.

Avoiding all force unwraps is misguided in no small part because it does not avoid all crashes, as Swift's standard library APIs enforce preconditions at runtime pervasively and these are nowhere marked with !—a point which you also allude to. Indeed, even APIs that return nil can also have preconditions!

We're not serving the misguided user by being reticent to redesign APIs in order to cater to that mistaken belief; if they go on not checking their own preconditions before calling APIs, they're going to have many problems pervasively.

The criteria here are, in my view, about the same as with any other API under design where we have to determine if it should be failable: Is it reasonable to expect the user to check the input beforehand as a precondition, or is the cost of checking it sufficiently complex that it'd require a nontrivial chunk of what the API itself is going to have to do anyway (indeed, at the limit, it could be impossible to determine success or failure without actually performing the operation)? If the latter, we should return nil; if the former, we can make it a precondition.

Literal values are generally at the polar opposite from something like, say, opening a file for which success or failure cannot be determined until the actual operation is performed. It is certainly nice, from a diagnostic standpoint, when the compiler can help the user troubleshoot their own input—it would be equally nice if the compiler could tell the user that they’re guaranteed to be asking for an out-of-bounds array index. I would enthusiastically welcome such an advance.

But in the absence of such diagnostic niceties, it is a precondition failure par excellence to pass an invalid value that’s literally literal which can be diagnosed at compile time. I’d go so far as to say that even if determining whether a string value obtained at runtime is a valid value of type T isn’t easy to do and could arguably call for a failable API, that doesn’t mean that the same applies for literals known at compile time.

Besides, the user should be verifying that any literal URLs aren’t just syntactically valid but point to the resource they want at least at the time of writing, rather than some typosquatter’s domain, say.

The statement is somewhat stronger: For any non-builtin type that conforms to an ExpressibleBy... protocol, there's (currently) no option other than failing at runtime for invalid literal inputs. Indeed, since few are designing replacements for the builtin types, any other nontrivial type that conforms will not accept every possible literal value [ExpressibleByNilLiteral excepted, of course].

If either of these is not acceptable, then you're arguing that more or less no non-builtin type should conform to ExpressibleBy... protocols at all—which, then, why vend this protocol at all, since we have the _ExpressibleByBuiltin... protocols for internal use? [As a predominantly “bag of syntax” protocol (with the narrow semantic point I raise above), there are hardly any useful algorithms that could be written generic over ExpressibleByStringLiteral that couldn’t be generic over StringProtocol if we stipulate that no non-builtin conformances should exist.]

Reasoning syllogistically:

Premise: If URL shouldn’t conform to ExpressibleBy... (due to the reasons above), then no non-builtin type should conform to ExpressibleBy...
- This implies the contrapositive: if any non-builtin type should conform to the protocol, then URL should conform
Premise: The public ExpressibleBy protocols were designed to allow conformance by at least some non-builtin type.
Conclusion: URL should conform to ExpressibleBy...

Karl · January 17, 2022, 5:17pm

Right, that is what I started with:

The point I was making is that URLs can be expressed as string literals in some contexts, but not others, and that it might be worth adding a label to the FilePath.init(URL) initializer so that URLs expressed as string literals could also be used there. So you would be able to write:

FilePath(url: "file:///usr/bin/swift")

As I said, it's more of a niche case, really more about symmetry than utility - if URL.init(FilePath) is gaining a label, I think it would be nice to add a label to the reverse.

Perhaps it is a case of talking past each other - I'm not sure if you were disagreeing with me or rephrasing what I already said about how FilePath("file:///usr/bin/swift") (no label) should be interpreted.

Oh, if only converting URLs to file paths were really that simple. There's a lot more to it, I'm afraid, and FilePath would do well to stay clear of it.

So you're saying that the benefit is disambiguation in case a bunch more URL-from-String style initialisers are added. That motivation is not mentioned in the pitch, and honestly, if that is really the motivation, I'd say it's worth re-evaluating whether those future additions are a good idea.

URLs really are, fundamentally, strings. If the API is so complex that we need literals to give tighter guarantees about which rules the parser will be using, we may have designed the wrong API.

As for @ _disfavoredOverload, I don't see it as such a big problem as to warrant this. It's unpleasant, but it exists for a reason and hopefully will be de-underscored soon . And deprecated APIs are automatically disfavoured IIRC.

This isn't really what I'm talking about: my concern is that developers often ask for compile-time checking, and this would certainly look like it, but it wouldn't really be compile-time checking.

We certainly can make this a precondition, but the "should we?" part needs to take other factors in to consideration - such as what a novice developer would make of it. Developers (especially from dynamic languages) often hear that Swift is really great for compile-time checking of code, but when they first start using the language, they don't really understand what that means or what the limitations are.

The value is known at compile-time, but the diagnosis is at run-time. That's why I think it's possibly misleading.

Sure, and there are other syntactically-valid URLs which are ill-formatted or forbidden by the URL's scheme (e.g. https:///apple.com, which is the example I gave previously. In case it isn't clear, the hostname is empty, and the HTTP spec says that is invalid).

I've suggested that perhaps some of those should also be grounds for a runtime error. It would possibly be easier to debug why requests to that URL fail, so I guess I'd weakly be in favour of adding some scheme-specific checks, but you could reasonably argue that it would be too strict.

Again, that isn't really what I'm arguing. My argument is not specific to the ExpressibleByStringLiteral conformance and not generalisable to all types. My argument is that, in the context of what developers have asked for on these forums for this particular type, I don't think it is appropriate to add an initialiser which accepts only a compile-time string, is not failable, and validates its contents at run-time. To understand this argument, it is important to point out that it will live next to a failable initialiser which accepts a string whose value is only known at run-time and does allow for recoverable failure. It's all about the context, and considering the API holistically.

But since you brought it up - your logic seems to suggest that all types which possibly can be expressed as strings should conform to the ExpressibleByStringLiteral protocol. You've excluded built-in types, but that seems to be an artificial limitation - in this respect, the only difference between a built-in type and a library type is that the former is subject to the authorial judgement of the standard library developers and core team, whilst library types are subject to the authorial judgement of their respective creators. So since you're making such a broad argument, let's remove that limitation, which immediately suggests that types such as Int and Bool also should conform to ExpressibleByStringLiteral.

API design does not simply fall out of abstract logic. Not every type which can conform should conform to any particular protocol - including the ExpressibleBy... family of protocols. Despite what the US Supreme Court thinks, API design is a creative process, and like all creative processes, requires an understanding of the target audience (which may range from expert developers, all the way to users who have literally never written any sort of software before in their lives).

My personal understanding of the target audience is formed by observing the comments on these forums and elsewhere, mentoring junior colleagues, etc, and seeing the kinds of things they ask for. Based on that, I think this feature is likely to lead to confusion with actual compile-time/build-time checking, and does not counter that with significant-enough benefits even when used correctly. In other words, it's not worth it IMO.

hisekaldma · January 17, 2022, 5:31pm

Karl:

Most of the time, you will be writing something like this instead. I'm not saying there is no benefit in any of these cases, but let's be honest... it's marginal at best.
var endpoint = URL("https://api.example.com/...")
var endpoint = "https://api.example.com/..." as URL
var endpoint: URL = "https://api.example.com/..."
The most compelling use-case I can think of, where people might actually be making requests from string literals is in scripts.

I agree that expressing URLs with string literals in production code is of limited utility, but I’ve found it quite helpful in tests. There the type can often be inferred, and being able to elide the URL(string:) and the force-unwrap cuts down on visual noise, making it easier to see that the test itself is correct.

Compare:

XCTAssertEqual(someURL, URL(string: "https://www.example.com")!)

to:

XCTAssertEqual(someURL, "https://www.example.com")

Karl · January 17, 2022, 5:43pm

That's reasonable, although there are a couple of points worth mentioning:

You don't need the force-unwrap
URL(string: "...") is too verbose, even for the failable initialiser. I think everybody would benefit if it was just URL("...")
You could create a custom XCTAssertEqualURLs method to remove even the initialiser (although the method name becomes longer, so it might not be worth it)

Foundation's URL model is overly complex, so the whole idea of "equal URLs" needs some qualification and possibly warrants the XCTest framework bundling a XCTAssertEqualURLs function anyway:

let urlA = URL(string: "http://example.com/a/b/c")!
let urlB = URL(string: "/a/b/c", relativeTo: URL(string: "http://example.com")!)!
let urlC = URL(string: "b/c", relativeTo: URL(string: "http://example.com/a/")!)!

// All of these URLs have the same .absoluteString.

urlA.absoluteString == urlB.absoluteString // true
urlB.absoluteString == urlC.absoluteString // true

// But they are not interchangeable.

urlA == urlB // false (!)
urlB == urlC // false (!)
URL(string: urlB.absoluteString) == urlB // false (!)

hisekaldma · January 17, 2022, 6:25pm

Karl:

Foundation's URL model is overly complex, so the whole idea of "equal URLs" needs some qualification and possibly warrants the XCTest framework bundling a XCTAssertEqualURLs function anyway:

let urlA = URL(string: "http://example.com/a/b/c")!
let urlB = URL(string: "/a/b/c", relativeTo: URL(string: "http://example.com")!)!
let urlC = URL(string: "b/c", relativeTo: URL(string: "http://example.com/a/")!)!

// All of these URLs have the same .absoluteString.

urlA.absoluteString == urlB.absoluteString // true
urlB.absoluteString == urlC.absoluteString // true

// But they are not interchangeable.

urlA == urlB // false (!)
urlB == urlC // false (!)
URL(string: urlB.absoluteString) == urlB // false (!)

That’s a good point. But couldn’t that potentially be fixed? I get why the parsing behavior of URL can’t change, but this has always felt like a bug to me. I can’t think of any other common Swift type where values that represent the same domain value are non-equal because they were constructed differently. (Aside from NaN.)

xwu · January 17, 2022, 6:45pm

I’m making a bit of a broader claim, which is that extension URL: ExpressibleByStringLiteral isn’t some mere workaround desirable for the side effect of “tighter guarantees” for overloading but is the semantic expression in code form and offers the appropriate syntactic niceties of your statement: “URLs really are, fundamentally, strings.”

But since you brought it up - your logic seems to suggest that all types which possibly can be expressed as strings should conform to the ExpressibleByStringLiteral protocol.

Not quite; my logic is, as above, that types which “really are, fundamentally, strings” should conform to that protocol—or else nothing could really conform to that protocol.

One potential hint—though not necessarily pathognomonic—is that a type both can be expressed as a string and can be converted (via an unlabeled converting initializer, typically) from another type that does conform to ExpressibleBy but represents different data as strings. Serious consideration should be given in that case as to whether such a type should conform.

It is not exactly parallel, but let’s keep in mind that removeFirst, popFirst and dropFirst live side-by-side and we trust users to use the one they want. It is not fatal to have related APIs that differ only in error handling.

I would call your attention back to our earlier discussion.

As you say, even if syntactically valid, there are scheme-specific requirements too—and it is unlikely that a beginner (or, for that matter, perhaps anyone but the most expert in URL-related matters) is asking specifically for compile-time checking of valid URL syntax as per RFC-this-and-that but not of scheme-specific requirements. Meanwhile, on the other hand, it’d likely be too strict to enforce requirements on URL initializers other than that it’s initialized from a syntactically valid URL string.

Yet even if all users could handle that distinction, a URL with a typo in it can just as well point to a typosquatter’s domain as it can be syntactically invalid or fail to meet a scheme-specific requirement. And if a user actually tries to use that URL, it could be far more devastating if it resolves to an unintended and possibly malicious endpoint than if it fails to be a valid URL at all.

My conclusion from this is that we cannot adequately protect the user from the pitfalls of not proofreading what they’ve typed, whether with compile-time or runtime checks, and regardless of syntax. A user who today dutifully writes guard let url = URL(string: foo) else ... is also misled into an illusion of having thereby averted tragedy like the one who assumes a literal is checked at compile time.

Further, the lack of compile-time evaluation features today doesn’t automatically make things worse—indeed, if we did have compile-time evaluation of URL syntax, those users could be more misled into assuming, when they’ve addressed whatever the compiler has told them to fix in their URL string, it is also correct in other respects which the compiler cannot ever check (the most critical one being “does this URL point to the resource I have in mind instead of a different resource I may have actually typed?”).

I suppose you could conclude that therefore URL should never conform to ExpressibleBy.... However, while good API designs promote correct use and steer away from incorrect use, the bar cannot be that we make even unreasonable misuses impossible.

Strings have long been associated with lack of checking in terms of their contents—take the punny term “stringly typed,” for instance. Not only are users accustomed to typing literal strings with contents that need to adhere to some other syntax that isn’t checked by the compiler, in fact there is actually no situation at all today where Swift looks inside string contents for adherence to non-Swift rules. One might say that these contents are “stringly valued” (sorry). When regexes gain compile-time parsing, they will also gain a new literal type with distinct delimiters. As you say above so pithily, URLs are “fundamentally, strings,” and I do not think it is reasonable to design on the assumption that ordinary users will be naturally misled to assume text in quotation marks without any internal syntax highlighting is going to be adequately checked (for whatever level of adequate) at compile time for some non-Swift syntax simply because the literal is not surrounded by parens or followed by !. That would be counter to everything they’ve done with strings to date.

icharleshu · January 20, 2022, 7:54pm

I chose isNotDirectory to signify that's all this API cares about (whether a URL points to a directory or not). IMO isFile might cause more ambiguity because, as you mentioned, an item can be neither a directory nor a regular file. There are other types of "disk items" such as symbolicLink, socket, namedPipe, etc. (see URLFileResourceType). It'd be confusing if a developer has to choose between isDirectory and isFile when the URL is pointing to a symbolic link.

I agree with you that the closure approach isn't that much better, and I'm in the process of rethinking this. On the other hand, having access to cwd is very important because Swift is a general-purpose language. For example, one might argue having access to the current working directory is crucial to scripting (at least as of now), and the concerns over thread-safety don't apply to simple scripts. I don't think we should exclude this API just because it might lead to unexpected results in some environments.

icharleshu · January 20, 2022, 8:36pm

I love these ideas. Thanks for the suggestions!

Philippe_Hausler · January 20, 2022, 8:55pm

One consideration that should be highlighted: pthread_chdir_np and pthread_fchdir_np to my knowledge do not exist on non-Darwin platforms which a sensible implementation of the withCurrentDirectory would eventually result in those calls.

jrose · January 30, 2022, 8:12pm

I'm concerned about the conflict with init(_: FilePath). As you note, today this causes undesired behavior where URL("https://www.apple.com") creates a relative file path URL with two components, "https:" and "www.apple.com". However, your proposed change would break anybody currently relying on the interpretation of a string literal as a file path, for example URL("/usr/bin"). (As a FilePath that's file:///usr/bin.) Similarly, URL("My Documents") currently succeeds, translating to My%20Documents.