Stochastic Errors Resolving Dependencies and/or Building with Feb-18-2021 Snapshot

mstokercricut · February 20, 2021, 4:12am

I'm building an executable that depends on a library in another repo, 3rd party libraries (ArgumentParser, SwiftProtobuf, SwiftNumerics) and FoundationXML.

When I build, it is reported that the default cache locations cannot be created. The dependencies will then resolve and start fetching. Sometimes the fetches succeed, sometimes an error reporting that the requested version does not exist, even though it does (e.g. SwiftProtobuf 1.15.0).

On subsequent runs, the fetch process will succeed, and then the build will commence, but only complete part of the build. It will report "Build Completed!", but only a single dependency or two will have been built, and none of the main project.

A search of the swift-package-manager repo gives the warning text coming from the SwiftTool.getCachePath(filesystem:) call:

github.com

apple/swift-package-manager/blob/a7815f557d966c30cf5813d8395a12364346e042/Sources/Commands/SwiftTool.swift#L479


      
                      diagnostics.emit(error: "Cannot find mandatory .netrc file at \(resolvedPath.pathString).  To make .netrc file optional, use --netrc-optional flag.")
                      throw ExitCode.failure
                  } else {
                      diagnostics.emit(warning: "Did not find optional .netrc file at \(resolvedPath.pathString).")
                      return nil
                  }
              }
              return resolvedPath
          }
          
          
private func getCachePath(fileSystem: FileSystem = localFileSystem) throws -> AbsolutePath? {
              if let explicitCachePath = options.cachePath {
                  // Create the explicit cache path if necessary
                  if !fileSystem.exists(explicitCachePath) {
                      try fileSystem.createDirectory(explicitCachePath, recursive: true)
                  }
                  return explicitCachePath
              }
          
          
    do {
                  // Create the default cache directory.

I tried deleting those directories and checking permissions, but could not seem to avoid the warning. Is anyone else having this issue?

Full command output (with executable names changed to MyExecutable, dependent library changed to MyLibrary, and company changed to mycompany):

C:\Users\Matt\Desktop\MyExecutable>swift build -c release
warning: Failed creating default cache locations, Error Domain=NSCocoaErrorDomain Code=256 "(null)"
warning: Failed creating default cache locations, Error Domain=NSCocoaErrorDomain Code=256 "(null)"
Fetching https://github.com/mycompany/MyLibrary.git
Fetching https://github.com/apple/swift-protobuf.git
Fetching https://github.com/apple/swift-argument-parser.git
Fetching https://github.com/apple/swift-numerics.git
error: Dependencies could not be resolved because 'MyLibrary' 0.9.2 contains incompatible tools version (5.3.0) and root depends on 'MyLibrary' 0.9.2.

C:\Users\Matt\Desktop\MyExecutable>swift build -c release
warning: Failed creating default cache locations, Error Domain=NSCocoaErrorDomain Code=256 "(null)"
warning: Failed creating default cache locations, Error Domain=NSCocoaErrorDomain Code=256 "(null)"
Updating https://github.com/mycompany/MyLibrary.git
Updating https://github.com/apple/swift-argument-parser.git
Updating https://github.com/apple/swift-numerics.git
Updating https://github.com/apple/swift-protobuf.git
error: Dependencies could not be resolved because no versions of 'swift-protobuf' match the requirement 1.6.0..<2.0.0 and root depends on 'swift-protobuf' 1.6.0..<2.0.0.

C:\Users\Matt\Desktop\MyExecutable>swift build -c release
warning: Failed creating default cache locations, Error Domain=NSCocoaErrorDomain Code=256 "(null)"
warning: Failed creating default cache locations, Error Domain=NSCocoaErrorDomain Code=256 "(null)"
Updating https://github.com/apple/swift-protobuf.git
Updating https://github.com/apple/swift-argument-parser.git
Updating https://github.com/apple/swift-numerics.git
Updating https://github.com/mycompany/MyLibrary.git
Cloning https://github.com/apple/swift-numerics.git
Resolving https://github.com/apple/swift-numerics.git at 0.0.8
Cloning https://github.com/apple/swift-protobuf.git
Resolving https://github.com/apple/swift-protobuf.git at 1.15.0
Cloning https://github.com/mycompany/MyLibrary.git
Resolving https://github.com/mycompany/MyLibrary.git at 0.9.2
Cloning https://github.com/apple/swift-argument-parser.git
Resolving https://github.com/apple/swift-argument-parser.git at 0.3.2
[1/6] Compiling _NumericsShims _NumericsShims.c

C:\Users\Matt\Desktop\MyExecutable>swift build -c release
warning: Failed creating default cache locations, Error Domain=NSCocoaErrorDomain Code=256 "(null)"
warning: Failed creating default cache locations, Error Domain=NSCocoaErrorDomain Code=256 "(null)"
[1/1] Planning build
[1/1] Planning build

* Build Completed!

C:\Users\Matt\Desktop\MyExecutable>ls .build/x86_64-unknown-windows-msvc/release
MyLibraryC.product                SwiftProtobuf.build      MyLibrary-client.product                description.json
MyExecutablePackageTests.product  _NumericsShims.build     MyLibrary-protobuf-test-client.product  protoc-gen-swift.product
ModuleCache                       MyLibrary-cli.product    MyLibrary-server.product

C:\Users\Matt\Desktop\MyExecutable>

yeswolf · February 20, 2021, 6:01pm

By the way, I have the same issue and currently it prevents Swift on Windows from running an executable in the CLion. We don't launch the application if we have a build error.

Workaround: invoke under the Administator account, in this case no such errors appear.

compnerd · February 20, 2021, 7:25pm

I wonder, do you have SE_CREATE_SYMBOLIC_LINK_PRIVILEGE for your account or do you have developer mode enabled?

Note that if you are using the machine for general usage in addition to development, I would recommend that you assign the SE_CREATE_SYMBOLIC_LINK_PRIVILEGE through a GPO rather than enable developer mode since that reduces restrictions for running untrusted code.

mstokercricut · February 23, 2021, 1:00am

The user on this Windows VM is an administrator, so it should. If I use the developer tools command prompt and run ln -s some/other/file it does create a functional link.

However, as @yeswolf mentioned, if I invoke the build with elevated privileges the build proceeds normally.

mstokercricut · February 23, 2021, 1:21am

I wonder, though, if this situation was created by accidentally running a build with elevated privileges after copying the module maps into the Visual Studio image and using the same terminal...

Would that create simlinks in the locations the build caching system was hoping to use, then prevent an unprivileged user from having a cache? I tried deleting ~/.swiftpm between attempts and that did not seem to help...

compnerd · February 23, 2021, 4:41am

The problem is caused by the fact that SPM assumes that it can create symbolic links. Windows does not support symbolic links in the same sense. There is a specific privilege that is required for creating symbolic links, which is not granted to users by default. Amongst the things that developer mode disables is the check, which would allow the symbolic link to be created.

yeswolf · February 24, 2021, 3:45pm

Yep. The user I was using belongs to "Administrators" group. Also, I've even manually added him to those who can create a symbolic link using this guide. Nothing changed.

It helps, by the way. I'd add it to related documentation as a required step.

compnerd · February 24, 2021, 4:34pm

I believe that the Administrators group is not sufficient (nor necessary) for this. The additional permission I think it should be, but may require a reboot afterwards? I know that I do not have developer mode enabled locally, and I am able to use Swift PM. If there is something subtle missing, it would certainly be good to figure out what is going on.

If you are able to reproduce the issue (by disabling the Developer Mode), it may be interesting to use ProcMon to see what file system operation is failing. That may help identify what is going on.

yeswolf · February 24, 2021, 5:34pm

Well, I don't mean that it's required. It just strange that all Administators have this permission by default, this particular user belongs to Administrators and it looks like it still does not have it. Probably, here it means that he has it only in actual administrator mode.

So I a) disabled Developer Mode b) added the user manually to those who have permission to create symbolic links c) rebooted. After that cleaned the project, deleted the .swiftpm folder and did not have any errors or warnings. Next, removed the permission and after reboot such errors appear again. So, it's definitely related to this permission and Developer Mode is indeed not required.

ProcMon shows "PRIVILEGE_NOT_HELD" results related to SPM cache folder. I think it also points us to the fact that this permissions must be set for the user (not sure, I'm not used to this tool).

compnerd · February 24, 2021, 6:25pm

Thank you for following up on this!

You are correctly reading the results, it is that the user must have the privilege.

Great, so it sounds like it is just the privilege is all that is needed. This is definitely good to have verified, and matches my expectation.

I wonder if there is a way to provide a GPO to grant the privilege for the user in an automated way.

sspringer · July 25, 2021, 9:36am

I had the same problem on Windows, that's what I did, I do not get the original error any more but it still does not work (see at the end):

Via gpedit.msc(*), set SE_CREATE_SYMBOLIC_LINK privilege for the user, restarted the computer, turned off Devloper Mode, the "mklink" command is working.
Removed C:\Users\<User>\.swiftpm , %TEMP%\org.swift.swiftpm, and the project's .build directory.
Even reinstalled the Swift toolchain.
Tested that git can clone all dependant packages without any account information.
Tested that swift build -c release on a standalone package does work.
Issued swift build -c release for the package containing the dependencies.

Problem:

After displaying Fetching https://github.com/..., nothing happens.

What could be the problem? Thanks.

I am using swift-5.4.2-RELEASE and the Swift package is https://github.com/stefanspringer1/SwiftXMLParserDemo.

(*) Note that on Windows Home, you first need to install gpedit.msc e.g. by running the following script as Administrator:

pushd "%~dp0" 
dir /b %SystemRoot%\servicing\Packages\Microsoft-Windows-GroupPolicy-ClientExtensions-Package~3*.mum >List.txt 
dir /b %SystemRoot%\servicing\Packages\Microsoft-Windows-GroupPolicy-ClientTools-Package~3*.mum >>List.txt 
for /f %%i in ('findstr /i . List.txt 2^>nul') do dism /online /norestart /add-package:"%SystemRoot%\servicing\Packages\%%i"

sspringer · July 26, 2021, 4:57am

It noticed that no Package.resolved file was written (else, I would have deleted it too besides the three folders mentioned; I now also deleted %TEMP%\org.swift.foundation.URLCache). I also noted that the package folder was write protected, but this is this strange Windows behaviour when every folder on your computer is "write protected" but you can still write to those folders, and in this case SPM could actually create the .build folder in the package and write to it, so probably "write protected" means nothing here... So SPM is still stuck displaying "Fetching..." and no Package.resolved file is written. Even tried to execute it as Administrator, failed too, and deactivated the Windows firewall.

sspringer · July 26, 2021, 7:46am

Via Process Monitor, I found

swift.exe: RegQueryValue: BUFFER OVERFLOW: HKCU\Control Panel\Desktop\PreferredUILanguages
swift.exe: RegQueryValue: BUFFER OVERFLOW: HKCU\Control Panel\Desktop\MuiCached\MachinePreferredUILanguages

and created ticket SR-14976.