Swift on Windows: Long Paths

Eugene_Gubin · March 28, 2021, 11:13am

Historically paths of Windows has length limited to 260 characters (also known as MAX_PATH).

To bypass such limitation long paths was introduced. Long path is a path with \\?\ prefix.

Also starting from Windows 10 version 1607 (released on August 2016) the limitiation could be lifted for all paths. To opt-in

this feature must be enabled by editing registry or by using gpedit.msc tool
application's manifest must include longPathAware flag

Is it worth to add support for the longPathAware in Swift toolchain? (Details: Maximum Path Length Limitation - Win32 apps | Microsoft Learn)

Also, not directly related to the Swift' toolchain, but does it make sense for open-source implementation of Foundation to check length of a path, then prepend it with \\?\ if its length over limit, before calling functions like a CreateDirectoryW?

compnerd · March 29, 2021, 4:07am

I went through some extraordinary lengths to avoid using the LongPathsEnabled in Swift. Editing the registry may not be permitted and it may get virtualized which is difficult to identify. Additionally, it makes it difficult to back port code in the future if it becomes interesting. Additionally, there are many tools, the manifests either need to be embedded (which requires additional tooling to be reverse engineered and re-implemented to be truly portable), or you need a freestanding manifest, which then has additional problems, e.g. symbols links.

Is there a particular reason that you would want to add it as a requirement to the toolchain?

Not sure what that accomplishes, it seems a bit more complicated than necessary. I tried to ensure that every single path passed to the Windows APIs always used the NT path representation to avoid the path limit. Did you find an instance that I missed?

Eugene_Gubin · March 29, 2021, 7:18am

As far as I know, test targets don't produce their own executables. They are executed by Swift toolchain.

Recently, I ported XcodeProj to Windows. Some tests were failing because paths were too long. Do I miss something?

github.com

HeMet/XcodeProj/blob/win-support/Tests/XcodeProjTests/Utils/PBXBatchUpdaterTests.swift#L131


      
          }
          
          func test_addFile_alreadyExistedWithSubgroups() {
              let sourceRoot = Path.temporary
              let mainGroupPath = UUID().uuidString
              let proj = fillAndCreateProj(
                  sourceRoot: sourceRoot,
                  mainGroupPath: mainGroupPath
              )
              let project = proj.projects.first!
              #if os(Windows)
                  // use shorter names to avoid too long paths on Windows
                  let subgroupNames = (0 ... 5).map { _ in String(UUID().uuidString.prefix(8)) }
              #else
                  let subgroupNames = (0 ... 5).map { _ in UUID().uuidString }
              #endif
              let expectedGroupCount = 1 + subgroupNames.count
              try! proj.batchUpdate(sourceRoot: sourceRoot) { updater in
                  let fileName = "file.swift"
                  let subgroupPath = subgroupNames.joined(separator: "/")
                  let filePath = sourceRoot + "\(mainGroupPath)/\(subgroupPath)/\(fileName)"

compnerd · March 29, 2021, 3:13pm

No, they are regular executables, just not named as such. The execution style you are thinking of is limited to the ObjC XCTest only, which limits it to the Darwin platforms.

I'm not familiar with this particular project, so it is difficult to even guess what happened. Do you have any additional information on what was going on? A call stack leading to the kernel may be helpful in diagnosing what is going on.

Eugene_Gubin · March 29, 2021, 5:46pm

The test calls FileManager.createDirectory(at:withIntermediateDirectories:attributes):

github.com

HeMet/XcodeProj/blob/win-support/Tests/XcodeProjTests/Utils/PBXBatchUpdaterTests.swift#L186


      
                                       name: "group",
                                       path: mainGroupPath)
                  proj.add(object: group)
                  let project = PBXProject.fixture(mainGroup: group)
                  proj.add(object: project)
                  return proj
              }
          
              private func createFile(at filePath: Path) throws {
                  let directoryURL = filePath.url.deletingLastPathComponent()
                  try FileManager.default.createDirectory(
                      at: directoryURL,
                      withIntermediateDirectories: true
                  )
                  try Data().write(to: filePath.url)
              }
          }

which calls CreateDirectoryW I think:

github.com

apple/swift-corelibs-foundation/blob/9ebb4442a45223b252ab714a2c101748a969e2da/Sources/Foundation/FileManager+Win32.swift#L186


      
                      try createDirectory(atPath: parent, withIntermediateDirectories: true, attributes: attributes)
                  }
              }
          
              try FileManager.default._fileSystemRepresentation(withPath: path) { fsr in
                var saAttributes: SECURITY_ATTRIBUTES =
                  SECURITY_ATTRIBUTES(nLength: DWORD(MemoryLayout<SECURITY_ATTRIBUTES>.size),
                                      lpSecurityDescriptor: nil,
                                      bInheritHandle: false)
                try withUnsafeMutablePointer(to: &saAttributes) {
                  if !CreateDirectoryW(fsr, $0) {
                    throw _NSErrorWithWindowsError(GetLastError(), reading: false, paths: [path])
                  }
                }
          
                if let attr = attributes {
                  try self.setAttributes(attr, ofItemAtPath: path)
                }
              }
          }

Sorry, I can't provide stack trace since error occurs inside Foundation and I have no experience with LLDB outside of Xcode.

compnerd · March 29, 2021, 5:52pm

Ah, I think that because _fileSystemRepresentation is used to back withUnsafeFileSystemRepresentation, which should return a Win32 style path, it is likely not using the NT style path as it should. This definitely is something that should get fixed.