Swift Package Manager (SPM) still not working on Windows

Yes, that works, thanks.

But building fails:

    MT: command "C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\llvm-mt.exe /nologo /manifest CMakeFiles\cmTC_39c6d.dir/intermediate.manifest /out:CMakeFiles\cmTC_39c6d.dir/embed.manifest /notify_update" failed (exit code 0x1) with the following output:
    llvm-mt: error: no libxml2
    llvm-mt: ignoring unsupported 'notify_update' option
    ninja: build stopped: subcommand failed.

But libxml2-development is available:

 Directory of S:\Library\libxml2-development
...
09/13/2021  07:18 PM    <DIR>          usr
...

What could be the problem?

UPDATE: The command for the complete build as documented on https://github.com/apple/swift/blob/main/docs/WindowsBuild.md seems to be incomplete, under "Build swift-corelibs-foundation" the following arguments are listed and should probably be added for the complete build: [UPDATE: yes, does work with my own libxml2 compilation added see below, paths adjusted]

  -D LIBXML2_DEFINITIONS="/DLIBXML_STATIC" ^
  -D LIBXML2_LIBRARY=S:\Library\libxml2-development\usr\lib\libxml2s.lib ^
  -D LIBXML2_INCLUDE_DIR=S:\Library\libxml2-development\usr\include\libxml2 ^

Question: Should more arguments be added for the complete build?

My current command for configuration for building the whole toolchain:

cmake -B "S:\build\5.4.3-RELEASE" ^
  -C S:\swift\cmake\caches\Windows-x86_64.cmake ^
  -D CMAKE_BUILD_TYPE=Debug ^
  -D CMAKE_INSTALL_PREFIX=C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr ^
  -D LLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-windows-msvc ^
  -D LLVM_ENABLE_PDB=YES ^
  -D LLVM_EXTERNAL_CMARK_SOURCE_DIR=S:\cmark ^
  -D LLVM_EXTERNAL_SWIFT_SOURCE_DIR=S:\swift ^
  -D CMAKE_C_FLAGS="/bigobj" ^
  -D CMAKE_CXX_FLAGS="/bigobj" ^
  -D LLVM_PARALLEL_LINK_JOBS=1 ^
  -D DLLVM_PARALLEL_LINK_JOBS=1 ^
  -D SWIFT_PATH_TO_LIBDISPATCH_SOURCE=S:\swift-corelibs-libdispatch ^
  -D SWIFT_WINDOWS_x86_64_ICU_I18N_INCLUDE=S:\Library\icu-67\usr\include ^
  -D SWIFT_WINDOWS_x86_64_ICU_I18N=S:\Library\icu-67\usr\lib\icuin67.lib ^
  -D SWIFT_WINDOWS_x86_64_ICU_UC_INCLUDE=S:\Library\icu-67\usr\include ^
  -D SWIFT_WINDOWS_x86_64_ICU_UC=S:\Library\icu-67\usr\lib\icuuc67.lib ^
  -D LIBXML2_DEFINITIONS="/DLIBXML_STATIC" ^
  -D LIBXML2_LIBRARY=S:\Library\libxml2-development\usr\lib\libxml2s.lib ^
  -D LIBXML2_INCLUDE_DIR=S:\Library\libxml2-development\usr\include\libxml2 ^
  -G Ninja ^
  -S S:\llvm-project\llvm

Question: I do not understand why -S S:\llvm-project\llvm when the whole toolchain should be built?

UPDATE: : It turned out that the path to Python installed by Visual Studio 2019 (C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64) has to be added to PATH, else Python is not available in the x64 Native Tools Command Prompt for VS2019.

UPDATE: still "no libxml2" (same message as above) solved: I now use my own compilation from https://github.com/stefanspringer1/Libxml2Validation

UPDATE: New (remaining) problem: [UPDATE: resolved, see below]

[3690/6194] Building LLDB Python wrapper
FAILED: tools/lldb/bindings/python/LLDBWrapPython.cpp tools/lldb/bindings/python/lldb.py
cmd.exe /C "cd /D S:\build\5.4.3-RELEASE\tools\lldb\bindings\python && S:\llvm-project\lldb\bindings\prepare_bindings.py --srcRoot=S:/llvm-project/lldb --targetDir=S:/build/5.4.3-RELEASE/tools/lldb/bindings/python --cfgBldDir=S:/build/5.4.3-RELEASE/tools/lldb/bindings/python --prefix=S:/build/5.4.3-RELEASE --use-static-binding"
usage: prepare_bindings.py [-h] [--debug] [--verbose]
                           [--config-build-dir CONFIG_BUILD_DIR] [--find-swig]
                           [--framework] [--generate-dependency-file]
                           [--prefix PREFIX] [--src-root SRC_ROOT]
                           [--swig-executable SWIG_EXECUTABLE] --target-dir
                           TARGET_DIR [--target-platform TARGET_PLATFORM]
                           [--use-static-binding]
                           [--static-binding-dir STATIC_BINDING_DIR]
prepare_bindings.py: error: the following arguments are required: --target-dir/--targetDir
[3692/6194] Building CXX object tools\lld\wasm\CMakeFiles\lldWasm.dir\Symbols.cpp.obj
ninja: build stopped: subcommand failed.

Test with variants:

S:\llvm-project\lldb\bindings\prepare_bindings.py --targetDir dummy
S:\llvm-project\lldb\bindings\prepare_bindings.py --targetDir=dummy
S:\llvm-project\lldb\bindings\prepare_bindings.py --target-dir=dummy
S:\llvm-project\lldb\bindings\prepare_bindings.py --target-dir dummy

Result each time:

prepare_bindings.py: error: the following arguments are required: --target-dir/--targetDir

I need to call python S:\llvm-project\lldb\bindings\prepare_bindings.py --target-dir dummy. Question: What can I do so a call of a python script automatically calls python including the arguments? Solution: associate Python with py.exe installed by pylauncher (launcher.amd64.msi: 64-bit launcher, installs to \Program Files\Python Launcher), also see in the Python blog, description see there.

Building the Swift toolchain "almost" done ([4149/6194]) :slight_smile: but still need some help see SR-15203. Many thanks.

UPDATE: See my log from the last configuration attached at SR-15203, maybe someone could have a look at it, thanks.

Last problem resolved: The commands listed under "One-time Setup (re-run on Visual Studio upgrades)" on https://github.com/apple/swift/blob/main/docs/WindowsBuild.md did not work, because – of course – making a symbolic link does not work if at the path of the link file a file already exists, you need to delete it first. Unfortunately, it did not see the error message. I am now using the comamnds listed listed on SR-15203 instead.

But: I am now getting a new error:

#error STL1000: Unexpected compiler version, expected Clang 12.0.0 or newer.

...Switching now back from VS 2022 to VS 2019 and try again.

UPDATE : No, after I changed from VS 2022 to VS 2019 because of the "expected Clang 12.0.0 or newer" error, and there it is again, the old "no libxml2" error despite of correct copying of the files. Note: I had to use MSVC\14.29.30133, because using Windows10SDK.17763 what only work without "Universal Windows App Development", as on https://github.com/apple/swift/blob/main/docs/WindowsBuild.md is listed as necessary. But maybe I just remove "Universal Windows App Development" and try again with Windows10SDK.17763.

I suspect that the problem is due to the CMake version that you are using. You should do a clean build adding -D CMAKE_MT=mt to the cmake invocation.

I suspect that the problem is due to the CMake version that you are using. You should do a clean build adding -D CMAKE_MT=mt to the cmake invocation.

Note that the only remaining problem is the recurring one "no libxml2". On https://github.com/stefanspringer1/BuildingSwiftOnWindows.git, I documented the exact steps I have done on a fresh (!) Windows installation, it is basically what https://github.com/apple/swift/blob/main/docs/WindowsBuild.md says but with some checks and corrections where the tests fail, incorporating the problems I had and described here.

I am willing to help to make the instructions better or at least to formulate a version that also works for the less experts, so more people could theoretically work on Swift for Windows. But this does not work without help.

(My original goal was to solve SR-15126: Swift Package Manager (SPM) on Windows gets stuck after displaying "...\swiftxml-manifest.exe -handle ...", because that means that Swift Package Manager is quite unusable for me on Windows. Your advice was that I should build a debug version of Swift to examine that problem, something that I then tried.)

See the previous message - you are using a different version of CMake, and that has an issue that can be worked around by adding -D CMAKE_MT=mt when configuring.

I also indicated that you could skip the toolchain part and only build SPM (and its dependencies) for a quicker turn around. That said, if your intention is to help improve the documentation for others getting started, that is definitely something which is welcome. I think that is valuable on its own, though I do understand that it doesn't directly help in debugging the issue that you are seeing.

1 Like

I already startet a new build process with that added to the configuration call, thanks.

Let's say I expanded my goals. But I will try to be more "economical" with my questions.

Note that the cmake is the one installed by the VS installer for VS 2019.

Even with -D CMAKE_MT=mt (and with or without LIBXML2_DEFINITIONS/LIBXML2_LIBRARY/LIBXML2_INCLUDE_DIR set) I still get:

[4197/6194] Performing configure step for 'libdispatch'
...
-- The C compiler identification is Clang 11.1.0 with MSVC-like command-line
-- The CXX compiler identification is Clang 11.1.0 with MSVC-like command-line
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: S:/b/1/bin/clang-cl.exe
-- Check for working C compiler: S:/b/1/bin/clang-cl.exe - broken
CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.20/Modules/CMakeTestCCompiler.cmake:66 (message):
  The C compiler

    "S:/b/1/bin/clang-cl.exe"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: S:/b/1/tools/swift/libdispatch-prefix/src/libdispatch-build/CMakeFiles/CMakeTmp

...
    llvm-mt: error: no libxml2
    llvm-mt: ignoring unsupported 'notify_update' option
    ninja: build stopped: subcommand failed.

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:12 (project)

Please also note that the configuration does not find:

-- Could NOT find Backtrace (missing: Backtrace_LIBRARY Backtrace_INCLUDE_DIR) 
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH) 
-- Could NOT find LibEdit (missing: LibEdit_INCLUDE_DIRS LibEdit_LIBRARIES) 
-- Could NOT find CursesAndPanel (missing: CURSES_INCLUDE_DIRS CURSES_LIBRARIES PANEL_LIBRARIES) 
-- Could NOT find LibLZMA (missing: LIBLZMA_LIBRARY LIBLZMA_INCLUDE_DIR LIBLZMA_HAS_AUTO_DECODER LIBLZMA_HAS_EASY_ENCODER LIBLZMA_HAS_LZMA_PRESET) 
-- Could NOT find Lua (missing: LUA_LIBRARIES LUA_INCLUDE_DIR) (Required is at least version "5.3")
-- Could NOT find LuaAndSwig (missing: LUA_LIBRARIES LUA_INCLUDE_DIR) 
-- Could NOT find SWIG (missing: SWIG_DIR) (Required is at least version "2.0")
-- Could NOT find LibXml2 (missing: LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR) (Required is at least version "2.8")
-- Could NOT find Backtrace (missing: Backtrace_LIBRARY Backtrace_INCLUDE_DIR) 
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Could NOT find LibXml2 (missing: LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR) 
-- Could NOT find LibEdit (missing: LibEdit_INCLUDE_DIRS LibEdit_LIBRARIES) 

(When setting LIBXML2_DEFINITIONS/LIBXML2_LIBRARY/LIBXML2_INCLUDE_DIR then LibXml2 is found during configuration).

Have you tried just building SwiftPM?

On my Mac, I can literally just clone the swift-package-manager repo, type “swift build”, and it builds.

I tried the same in my Windows VM, and there were some issues with llbuild (apparently something in LLVMSupport was building without the _WIN32 define being set for some reason, so it used a bunch of Posix types not available on Windows such as uid_t), but it downloaded all the dependencies correctly and started doing its thing.

I think the CMake build is probably in better shape, so maybe try that? Or see if you have better luck building llbuild using the pre-built package manager? In any case, that seems to me to be a simpler task and you wouldn’t have to wait hours to build a full toolchain.

Not yet, will try soon, thank you for your hints. That would certainly be a sensible thing to do. But I think being able to build the whole Swift toolchain on a fresh Windows OS installation with a complete description of what to do is a tremendously important thing, so once I began to try building anyway, I jumped on that train. (And I know something like this is never "describe once, always works"...)

That is because you are building from an older release. You are going to have to apply 2990e8bc070aee69b3a1cdecfd526ebf0f7ec5fa to the swift repository on the branch to get it to build.

Aha, I would have thought that swift-5.4.3-RELEASE works. OK, the announcements for 5.4.3 on Linux and Windows came later, so it could have been clear that one tag is not for all...

Q: Where can I get the information about which commits to use?

Q: Could tags like swift-5.4.3-RELEASE-Windows be helpful?

Q: My checkouts are now as follows, should this be OK?

S:\cmark: HEAD detached at swift-5.4.3-RELEASE
S:\indexstore-db: HEAD detached at swift-5.4.3-RELEASE
S:\llvm-project: HEAD detached at swift-5.4.3-RELEASE
S:\swift: HEAD detached at 2990e8bc070
S:\swift-argument-parser: HEAD detached at 1.0.1
S:\swift-corelibs-foundation: HEAD detached at swift-5.4.3-RELEASE
S:\swift-corelibs-libdispatch: HEAD detached at swift-5.4.3-RELEASE
S:\swift-corelibs-xctest: HEAD detached at swift-5.4.3-RELEASE
S:\swift-driver: HEAD detached at swift-5.4.3-RELEASE
S:\swift-llbuild: HEAD detached at swift-5.4.3-RELEASE
S:\swift-package-manager: HEAD detached at swift-5.4.3-RELEASE
S:\swift-tools-support-core: HEAD detached at swift-5.4.3-RELEASE
S:\Yams: HEAD detached at 4.0.6

It does if you use an older version of CMake :slight_smile:

You build and fix any issues that you run into. Once you have a build, you know what commits to use.

No, because you will find that things change too rapidly.

Don't know; build and find out? Most of my time is spent on keeping main building, so I'm afraid that for anything else, you are in uncharted waters.

OK, right now I am at least getting some "real" compilation errors like [4482/6213] ... LLVMARCOpts.cpp.obj ... ARCEntryPointBuilder.h(380): error C2039: "getTypeByName" ist kein Member von "llvm::StructType (sorry messages in German).

I understand, but maybe the following would do it:

  • You have successful builds (with tests?) in your azure builders. Would it be possible to somehow (automatically) publish the according commits of the source repositories (git -C llvm-project rev-parse HEAD, git -C swift rev-parse HEAD, etc.), or at least for some successful builds? This way someone could start with sources that should build.
  • There is a Windows toolchain published on the Download Swift page. The commits used for that build would be particularly interesting because (as in my current case) people might have problems with that published version and might want to build a debug version of it (as in my current case) or even examine further (in the source code) what the issue could be (and of course test if some newer builds already solve the problem, see first point).

Would be nice if this could be possible, thank you in advance, and, generally, for your great work.

You could always look at the build itself, the information for the revisions are recorded on the build run.

Those are already published: see the nightly tags/release tags. However, Visual Studio versions and SDKs are not part of the Swift repository, and that does impact the builds. I don't have complete control over the builders - Microsoft updates the VS/SDK at whatever cadence they deem fit. Most of that information should be in the logs, but is not fully possible to replicate - how do you get MSVC toolset version 14.12.34567? You just need to be willing to patch everything and resolve issues along the way.

Ah OK, I see. No successful builds in that list right now, so I would need to look at that regularly. The link to your pipelines is public and I may use it in a public description?

Sorry, I do not find those.

I understand that. There is a documentation by Microsoft on how to "export the workload and component information to a .vsconfig file" but of course, if this cannot be done by a command line, it is not of much use for us. I suppose the Azure builders are at least somehow "up to date" meaning "newer" Windows SDKs, I suppose.

Yes, the pipelines are public, you may use that.

git://github.com/apple/swift should have those tags

swift-DEVELOPMENT-SNAPSHOT-..

Correct

Update: See stackoverflow, but I tried devenv /ResetSettings S:\my.vssettings and that did not work.

But those are the tags I tried to use (swift-5.4.3-RELEASE), and from my current understanding those tags to not refer to the commits you were using to build the Windows release?

The builds posted on swift.org are supposed to be from the same tags (swift-5.5-DEVELOPMENT-SNAPSHOT... are the 5.5 snapshots, swift-DEVELOPMENT-SNAPSHOT... are the main snapshots). However, I do often have other builds on there which are from different points. It really is just a way of having public builds which others can audit to see that nothing is being injected. There is nothing particularly special to them. Your tools are at different versions, which means you will have differences.

Terms of Service

Privacy Policy

Cookie Policy