@compnerd I’m hoping to get some time to work on this this weekend, so is there anything in particular I should focus on so as not to collide with your work? I’m asking since I noticed you submitted a couple of PRs on path handling this week to SwiftPM and TSC.
To provide some numbers, I did a few performance tests to compare
URL and it seems that
URL is generally worse:
- In appending a relative path to an absolute path, URL seems about 8x slower:
- In splitting a path into its components, URL is about 0.3x faster:
- In initializing a path from a string, URL is about 10x slower:
AbsolutePath(pathString) URL(fileURLWithPath: pathString)
Since we care very much about path performance in the Package Manager, I'd like to take a stab at implementing full Windows path support into SwiftPM's
It seems that URL really does wrap a NSURL, which is an open class (even though subclassing NSURL is a pretty terrible idea). I wonder if there aren’t significant performance improvements to be had by either using CFURL directly, making a new type, or closing NSURL.
The kind of poor performance you mention should be very concerning, not just for SwiftPM.
That'd be awesome, thanks!
I'm experimenting with an approach at the moment that seems promising: just converting Windows paths to be Unix-like in
AbsolutePath's storage and converting back on calls to
pathString. For simple drive paths, this means that e.g.
C:\Folder\File.swift is internally represented as
/C:/Folder/File.swift. For e.g. UNC path schemes, we could prefix the internal path as e.g.
So far, I've been able to use this approach to get
swift package init --type executable. Still working on
swift build due to other issues, but it should be a lightweight solution that will work for most (and possibly all) use cases. The PR is up at https://github.com/apple/swift-tools-support-core/pull/54 (cc @hartbit).
The next issue I've run into is a crash when calling
FoundationNetworking, which seems to be from calling
Bundle.main (https://bugs.swift.org/browse/SR-12301). @compnerd, would you expect that to be covered by existing test coverage? I don't currently have a setup for building/testing Foundation on Windows.
This is a creative approach. If it's working well, it might be a viable short-term workaround until we get proper support for Windows paths.
Can you trying making a custom configuration? SwiftPM currently only needs the downloader object for macOS (for binary dependencies feature) so maybe we should avoid initializing it unless we need it.
- Deprecate the current
Pathwould serve as a view, dispatching all operations to
FileSystemwould use system APIs to implement the operations
This should at least abstract the path operations from the representation, and
FileSystem should be able to do all the necessary operations (e.g. UTF8 -> UTF16 -> Operations -> UTF8 -> Copy -> Free UTF16) as appropriate.
This would make
Path simply a lightweight wrapper for the file system operations.
Perhaps I misunderstood some aspects of our discussion. I was thinking we could take this approach:
- Define a protocol called
Pathto represent all the common platform-specific logic for
RelativePathand move all the current logic into a type called
UNIXPaththat conforms to
Path. Already done.
- Implement windows path logic in a
WindowsPathtype using the Windows API.
I wasn't planning on going through the
FileSystem type and I have a hard time understanding why it's necessary or desirable.
Why do you think the
WindowsPath shouldn't do the UTF8 -> UTF16 -> Operations -> UTF8 -> Copy -> Free UTF16 dance itself? What do we gain by going through the
This approach makes a lot of sense to me. And it's still the case that none of the operations on
RelativePath will actually access the file system, right?
Correct. No operation will actually access the file system.
They may - things like resolving symbolic links definitely need to access the file system. The Windows paths should go through the file system APIs (which can cause access ... because they may trigger things like filter drivers to get triggered). This is part of the problem of handling Windows properly - the filter drivers can actually inject themselves into the paths and do weird things.
SwiftPM's path abstractions are specifically designed to not perform any file system access themselves. Any operation that requires file system access already goes through the file system layer. These include queries like if a path is a symlink, a directory, it's attributes or to determine if the path even exists. The path abstraction is for answering basic queries that can be performed without the file system access. Some examples: getting a path's basename, dirname, appending or dropping a path fragment. Also note that SwiftPM almost always operates on paths that it got from some file system operation (like current working directory). One concern is that you might end up initializing a path that you didn't get from the file system, which is an issue that comes up from time to time. We can consider renaming the initializer to
AbsolutePath(unsafePathString: String) to signify the danger of directly initializing the path object.