Portability of Swift Package Manager

Development Package Manager
21

@compnerd I’m hoping to get some time to work on this this weekend, so is there anything in particular I should focus on so as not to collide with your work? I’m asking since I noticed you submitted a couple of PRs on path handling this week to SwiftPM and TSC.

1 Like
22

To provide some numbers, I did a few performance tests to compare AbsolutePath/RelativePath versus URL and it seems that URL is generally worse:

  • In appending a relative path to an absolute path, URL seems about 8x slower:
absolutePath.appending(relativePath)
url.appendingPathComponent(relativePathString)
  • In splitting a path into its components, URL is about 0.3x faster:
absolutePath.components
url.pathComponents
  • In initializing a path from a string, URL is about 10x slower:
AbsolutePath(pathString)
URL(fileURLWithPath: pathString)
2 Likes
23

Since we care very much about path performance in the Package Manager, I'd like to take a stab at implementing full Windows path support into SwiftPM's AbsolutePath and RelativePath.

5 Likes
24

It seems that URL really does wrap a NSURL, which is an open class (even though subclassing NSURL is a pretty terrible idea). I wonder if there aren’t significant performance improvements to be had by either using CFURL directly, making a new type, or closing NSURL.

The kind of poor performance you mention should be very concerning, not just for SwiftPM.

3 Likes
25

That'd be awesome, thanks!

1 Like
26

I'm experimenting with an approach at the moment that seems promising: just converting Windows paths to be Unix-like in AbsolutePath's storage and converting back on calls to pathString. For simple drive paths, this means that e.g. C:\Folder\File.swift is internally represented as /C:/Folder/File.swift. For e.g. UNC path schemes, we could prefix the internal path as e.g. /UNC/rest-of-path.

So far, I've been able to use this approach to get swift package init --type executable. Still working on swift build due to other issues, but it should be a lightweight solution that will work for most (and possibly all) use cases. The PR is up at Path: Handle Windows paths by converting to an internal UNIX-style representation by troughton · Pull Request #54 · apple/swift-tools-support-core · GitHub (cc @hartbit).

The next issue I've run into is a crash when calling URLSessionConfiguration.default within FoundationNetworking, which seems to be from calling Bundle.main ([SR-12301] Bundle.main crashes on Windows in Foundation.Bundle._supportsFreestandingBundles.getter : Swift.Bool · Issue #3266 · apple/swift-corelibs-foundation · GitHub). @compnerd, would you expect that to be covered by existing test coverage? I don't currently have a setup for building/testing Foundation on Windows.

27

This is a creative approach. If it's working well, it might be a viable short-term workaround until we get proper support for Windows paths.

Can you trying making a custom configuration? SwiftPM currently only needs the downloader object for macOS (for binary dependencies feature) so maybe we should avoid initializing it unless we need it.

28

@Aciid, @hartbit, and I spoke offline some more about this. I think that we came to agreement on another approach which we think could work:

  • Deprecate the current Path constructors
  • Path would serve as a view, dispatching all operations to FileSystem
  • FileSystem would use system APIs to implement the operations

This should at least abstract the path operations from the representation, and FileSystem should be able to do all the necessary operations (e.g. UTF8 -> UTF16 -> Operations -> UTF8 -> Copy -> Free UTF16) as appropriate.

This would make Path simply a lightweight wrapper for the file system operations.

Hopefully I accurately reflected the conclusions of our discussions - feel free to correct me @Aciid and @hartbit.

3 Likes
29

Perhaps I misunderstood some aspects of our discussion. I was thinking we could take this approach:

  • Define a protocol called Path to represent all the common platform-specific logic for AbsolutePath and RelativePath and move all the current logic into a type called UNIXPath that conforms to Path. Already done.
  • Implement windows path logic in a WindowsPath type using the Windows API.

I wasn't planning on going through the FileSystem type and I have a hard time understanding why it's necessary or desirable.

Why do you think the WindowsPath shouldn't do the UTF8 -> UTF16 -> Operations -> UTF8 -> Copy -> Free UTF16 dance itself? What do we gain by going through the LocalFileSystem type?

1 Like
30

This approach makes a lot of sense to me. And it's still the case that none of the operations on AbsolutePath and RelativePath will actually access the file system, right?

31

Correct. No operation will actually access the file system.

32

They may - things like resolving symbolic links definitely need to access the file system. The Windows paths should go through the file system APIs (which can cause access ... because they may trigger things like filter drivers to get triggered). This is part of the problem of handling Windows properly - the filter drivers can actually inject themselves into the paths and do weird things.

33

SwiftPM's path abstractions are specifically designed to not perform any file system access themselves. Any operation that requires file system access already goes through the file system layer. These include queries like if a path is a symlink, a directory, it's attributes or to determine if the path even exists. The path abstraction is for answering basic queries that can be performed without the file system access. Some examples: getting a path's basename, dirname, appending or dropping a path fragment. Also note that SwiftPM almost always operates on paths that it got from some file system operation (like current working directory). One concern is that you might end up initializing a path that you didn't get from the file system, which is an issue that comes up from time to time. We can consider renaming the initializer to AbsolutePath(unsafePathString: String) to signify the danger of directly initializing the path object.