Foundation URL Improvements

[icharleshu (Charles Hu)]

@gwendal.roue

I very warmly welcome the first public Foundation pitch

Thanks!!!!

@gwendal.roue

I wonder, @icharleshu, if URL will become Sendable eventually.

We are looking into it and believe me, we really really would want to make it Sendable

[icharleshu (Charles Hu)]

@Max_Desiatov

Do I understand correctly that these are runtime assertions? Thus URL("invalid URL") would still compile and type-check as non-optional URL without any compile-time errors?

Correct. One might imagine Swift might have a URL literal type in the future where the URL string is checked by the compiler at compile time. Unfortunately such change is way beyond this proposal though.

[tera]

@icharleshu

Unfortunately, I think this is the case of "the ship has already sailed." URL already "tangles" with the file system in many ways. For example:

• .resourceValues(forKeys:) and friends need to talk to the file system to retrieve metadata.
• As others have pointed out, init(fileURLWithPath:) talks to the file system to determine whether the path is a directory.

Using File System within resourceValues is totally understandable (the caching issues mentioned above aside). However I do not understand the logic behind the FS check during URL construction: there could be no file/directory at the time of the check and there could be a file/directory in there soon thereafter (doing URL actual usage). Or vice versa, at the time of the check there could be a file/directory but soon thereafter (at the time of URL actual usage) the situation can change and there can be a directory/file/absence where file/directory previously was. The FS check during URL initialisation looks misplaced and fragile.

[icharleshu (Charles Hu)]

@Karl

It sounds weird - like, surely everybody would have noticed already if URLs didn't conform to Comparable - but it's true! It doesn't conform!

I'm genuinely curious: what are some use cases for this? Do people sort URLs very often?

If we were to implement Comparable, what would be the best way to compare two URLs? Lexicographically comparing the underlying URL strings would be the most obvious approach, but I'm worried about some edge cases that might not make sense. For example:

• Should scheme be part of the comparison? If so, file:///www.b.com < https:///www.a.com might be unexpected.
• What about the order of query parameters? Does https://example.com/item?color=blue&name=spaceship < https://example.com/item?name=spaceship&color=blue make sense?

[Karl]

@icharleshu

what are some use cases for this? Do people sort URL s very often?

Apparently not

But one use case is to store URLs in a sorted data structure. For those situations, it doesn't really matter what you sort by - as long as two URLs consistently compare as < or not, so you can insert them in a consistent place and look them up using binary search.

In some sense, you can think of it as something like a hash function, except that it requires 2 values as input rather than a single value, but returns a consistent result. You can use that result to apply useful algorithms (such as binary search), but the actual terms "greater than" or "less than" are not particularly meaningful.

As for whether it makes sense? You could make the same argument for String - can you even compare strings of characters as being "greater than" or "less than" other strings of characters? Does it make sense to say that "dog" > "deer" > "city" > "andromeda galaxy"?

In Swift, we have a rule (not technically enforced, but very much encouraged) that you shouldn't conform types you don't own to protocols you don't own. Since Foundation owns URL and the standard library owns Comparable, only Foundation can safely provide that conformance.

[stevapple (YR Chen)]

[Bug] Foundation.URL should support semicolon(; ) appearing in URL path

http://host/book;id=1/page;2/ this URL is a valid URL, but URLComponents will ruin it with %3B http://host/book%3Bid=1/page%3B2/ Since Swift Server is coming out, this bug should be solved.

And yes, I can prove that it had already been fixed some time.

[James_Dempsey]

One comment about the StaticString initializer, which I think is a great addition.

It seems the most ergonomic way to specify a file URL would be use the absolute file path, e.g. "/Library/Fonts" as opposed to "file:///Library/Fonts".

This is what developers are familiar with from the current fileURL methods, and the recent FilePath type is initialized in this way.

I realize that requiring the URL scheme makes the caller's intent explicit and gives a simple rule for use and assertion ("URL scheme required") with no exceptions.

But I suspect in use, folks would prefer that an absolute path string be treated as initializing a file URL, without the additional ceremony of the file:// scheme. (Personally, I find the triple '/' a bit error-prone to type.)

I also realize there is a complex web of initializers that might make that approach ambiguous or otherwise not possible.

But if it is possible, I think allowing this would make the API more ergonomic and not introduce a big issue in terms of programmer error / inability to assert correctness.

[icharleshu (Charles Hu)]

Great point! Thanks!

[icharleshu (Charles Hu)]

@zaneenders

we used an optional enum that defaults to . web and you would add something like .file for file path's?

Thanks for the suggestion. Personally I'm not 100% sure whether we should require a scheme for each string either. Initially, I was envisioning the opposite: we treat strings with the file:/// scheme or no scheme (such as "/System/Library") as file paths and all other strings as web addresses. We eventually decided to opt-in for a simpler rule: every URL must have a scheme.

However, adding another parameter is essentially splitting this requirement into two places. This, IMO, might create some confusion. For example:

• What do we do if someone passes in conflicting parameters such as init("https://www.example.com", .file)?
• If we default the second parameter to .web, we might create unexpected results such as URL("/System/Frameworks") being a web address.

[icharleshu (Charles Hu)]

@Saklad5

Why do these require Collection instead of Sequence ? Also, have you considered requiring C.Element to conform to StringProtocol rather than specifically being a String ?

I must admit there isn't a logical answer to why I chose Collection over Sequence , but I agree with you Sequence would've been a better choice here. I'll update the proposal to reflect this.

As far as String vs StringProtocol goes, I chose String because the existing API explicitly asks for String and I didn't think we need to change it.

[icharleshu (Charles Hu)]

@Saklad5

Would it be possible to apply these changes while simultaneously encouraging that users transition to Swift System

Yes absolutely.

[icharleshu (Charles Hu)]

@anon9791410

URL would match up with Date.ParseStrategy in this code, and the format argument would be whatever is the new equivalent of URLComponents .

That's an interesting idea! Thanks for the suggestion.

[Saklad5 (Jeremy Saklad)]

@James_Dempsey

But I suspect in use, folks would prefer that an absolute path string be treated as initializing a file URL, without the additional ceremony of the file:// scheme. (Personally, I find the triple '/' a bit error-prone to type.)

As mentioned earlier, Foundation.URL isn’t actually very good for file paths. You should really be moving to Swift System’s FilePaths wherever practical. In the meantime, URLs do need a scheme, so if you want to use it you’re going to need to provide one.

You’re still going to want to move to FilePath someday, though. It may as well be now.

[icharleshu (Charles Hu)]

@tera

Is international domain support on the table? RFC 5890 / 5891, etc
as of today this code doesn't work:

@tera

you have to first manually parse it into components, then feed components individually to URLComponent machinery to finally grab the resulting url:

Thanks for bringing this up. We are aware that the NSURL parser is quite old and doesn't support many of the modern URLs (for example, you can't construct fine URLs like https://i❤.ws), and there's a difference between URL and URLComponents parsing. Updating the parser is something we are investing in and evaluating for the future. Unfortunately, it won't make it in this proposal.

@tera

// After after
let photoURL = baseURL + "Photos" + id.first! + id

Haha, this was in my original proposal. We decide to not go down this route because it's potentially ambiguous. Consider this scenario:

let url = URL("file:///System/Frameworks/Foundation")
let foundation = url + ".framework"

What should foundation be? file:///System/Frameworks/Foundation.framework or file:///System/Frameworks/Foundation/.framework?

@tera

There are APIs that only accept file url and if you feed them http url you'll get a runtime error. Obviously it would be nicer if that was a compile time error.

I agree. It would be nice if we can have some compile time type safety for different URL types. This would go into the same "big picture" bucket alongside the parser update.

@tera

Does that mean the cache is very very temporary and the cached entry will not be there on the next callout of event loop, say, some 0.01 seconds after resource was placed in cache?!?!

Yes. This system was designed primarily for the use case where a URL is passed between several different layers of code to minimize the possible repeated disk access. We decide to use the RunLoop as the "trigger" for cache cleaning to minimize the chance of possible stale data.

[icharleshu (Charles Hu)]

@benrimmington

LosslessStringConvertible

Would it be possible to add this conformance?
Or might it conflict with the proposed non-failable initializer?

I don't think URL is LosslessStringConvertible because there are many Strings that can't be represented by an URL. For example, spaces in a web address URL will always be invalid: URL(https://www.e ample.com) will fail.

[icharleshu (Charles Hu)]

@tera

this is really bad. What would happen for an URL on a network drive with a slow network? If this falls into the category of "straightforward changes" I'd suggest to fix this.

I agree. I'm working on updating the proposal for the next revision.

[xwu (Xiaodi Wu)]

@icharleshu

I don't think URL is LosslessStringConvertible because there are many Strings that can't be represented by an URL . For example, spaces in a web address URL will always be invalid: URL(https://www.e ample.com) will fail.

This is a misunderstanding of the semantic requirements of LosslessStringConvertible: Bool conforms to that protocol and there are exactly two strings that convert to Bool ("true" and "false").

Conforming to this protocol requires lossless roundtripping from URL to String back to URL. There may well be certain "value-preserving" details of URL that cannot be encoded in its String representation which preclude conformance, but what you describe is not at all a requirement.

[benrimmington]

@xwu

Conforming to this protocol requires lossless roundtripping from URL to String back to URL. There may well be certain "value-preserving" details of URL that cannot be encoded in its String representation which preclude conformance, but what you describe is not at all a requirement.

If the description returns absoluteString, then it wouldn't preserve the base/relative parts, but this is also how JSONEncoder and JSONDecoder override the Codable representation of URLs.

LosslessStringConvertible conformance would mean that you can use String.init(_:) — rather than String.init(describing:) or interpolation. This is convenient because String(myURL) and URL(myString) with unlabelled arguments are the natural way to spell those inverse operations, and they could also be used by generic code (for any conforming type).

[tera]

@icharleshu

@tera

> // After after
let photoURL = baseURL + "Photos" + id.first! + id
> ```

Haha, this was in my original proposal. We decide to not go down this route because it's potentially ambiguous. Consider this scenario:

let url = URL("file:///System/Frameworks/Foundation")
let foundation = url + ".framework"

What should foundation be? file:///System/Frameworks/Foundation.framework or file:///System/Frameworks/Foundation/.framework?

I don't think this is a showstopper. For example:

let newUrl = url + "System" + "Frameworks" + "Foundation" + ".framework"
let newUrl = url + "System/Frameworks/Foundation/.framework" // also ok
let newUrl = url + "picture" + FileExtension("png")
let newUrl = url + "picture.png" // also ok

where URL has both static func + (URL, String) and func + (URL, FileExtension) and FileExtension is a simple struct holding a string.

Or alternatively:

let newUrl = url / "System" / "Frameworks" / "Foundation" / ".framework"
let newUrl = url / "System/Frameworks/Foundation/.framework" // also ok
let newUrl = url / "picture" | "png"
let newUrl = url / "picture.png" // also ok

[mickeyl (Dr. Mickey Lauer)]

Pretty cool. I like "abusing" the / operator for concatenating paths.