Can’t build libswiftDispatch.so

art-divin · 2024-01-04T15:45:30.950Z

thank you for this thread and thanks @taylorswift for raising the issue in the first place!

For the last couple of days I have been trying to reproduce this crash whilst having a debugger attached.

I have compiled the compiler using utils/build-script --release-debuginfo --skip-test-early-swift-driver --skip-test-swift -T - at least this was my last attempt after cleaning the build folder. Before that I have tried all the other configurations for utils/build-script.
Next, I have tried to use VSCode to run swift test where the subject is located (working directory), but SPM would stuck and I would not be able to continue to compilation process of the Sourcery, for whatever reason.

Good thing is that I was able to reproduce the crash in CLion instead! I need to update HowToGuides/GettingStarted.md for CLion configuration, the one written there is insufficient (i.e. does not mention that clang and clang++ must be set as compilers in CLion environment settings etc.).

Having crash reproduced, the next step is to attach to a debuggable libSwiftCore.so, but this is really unclear how to do for a beginner like me. Maybe if I have a source code and a debuggable compiled library, I would magically be able to step over inside of the IDE if all paths are set correctly, this is something for me to learn still.

BUT running the utils/build-toolchain according to the DevelopmentTips.md to build a debuggable libSwiftCore brings the error in the OP mentioned by @taylorswift -

[81/81][100%][2.511s] Linking Swift shared library libswiftDispatch.so
FAILED: libswiftDispatch.so

The crash report that I have got from reproducing locally contains libSwiftCore path pointing to my local installation of Swift toolchain rather than the one compiled in /build/Ninja-RelWithDebInfoAssert subfolder. And I am unsure how to make IDE link against libSwiftCore built along with the compiler, rather than the one installed in the system.

I hope you could guide me how to debug this crash so I can, eventually, resolve it and enable 100% Linux support in Sourcery

taylorswift · 2024-01-04T21:38:04.117Z

Finagolfin:

Does that libdispatch file and directory exist after that error? If not, maybe you forgot to "remove that libdispatch patch for utils/build-script that I suggested earlier, as you want to build libdispatch now," ie enable this override I mentioned to you earlier in the week again.

building libdispatch has never worked (for me), so if i reenable libdispatch, the build once again fails with the same error: fatalError as in this attempt.

the exact output, if it helps

/swift/swift-project/swift$ /swift/swift-project/build/cmake-linux-x86_64/bin/cmake --build /swift/swift-project/build/Ninja-RelWithDebInfoAssert/libdispatch-linux-x86_64 -- -j12 all
[1/1] Linking Swift shared library libswiftDispatch.so
FAILED: libswiftDispatch.so src/swift/CMakeFiles/swiftDispatch.dir/Block.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Data.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Dispatch.swift.o src/swift/CMakeFiles/swiftDispatch.dir/IO.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Private.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Queue.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Source.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Time.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Wrapper.swift.o src/swift/swift/Dispatch.swiftmodule 
: && /swift/swift-project/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swiftc -output-file-map src/swift/CMakeFiles/swiftDispatch.dir/RelWithDebInfo/output-file-map.json -incremental -j 12 -emit-library -o libswiftDispatch.so -module-name Dispatch -module-link-name swiftDispatch -emit-module -emit-module-path src/swift/swift/Dispatch.swiftmodule -emit-dependencies -DswiftDispatch_EXPORTS -module-cache-path "/swift/swift-project/build/Ninja-RelWithDebInfoAssert/libdispatch-linux-x86_64/module-cache" -O -g -Xcc -fblocks -Xcc -fmodule-map-file=/swift/swift-project/swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I/swift/swift-project/swift-corelibs-libdispatch -Xcc -I/swift/swift-project/swift-corelibs-libdispatch/src/swift/shims -vfsoverlay /swift/swift-project/build/Ninja-RelWithDebInfoAssert/libdispatch-linux-x86_64/dispatch-vfs-overlay.yaml -I /swift/swift-project/swift-corelibs-libdispatch/src/BlocksRuntime -I /swift/swift-project/build/Ninja-RelWithDebInfoAssert/libdispatch-linux-x86_64 -I /swift/swift-project/swift-corelibs-libdispatch -I /swift/swift-project/swift-corelibs-libdispatch/src -I /swift/swift-project/build/Ninja-RelWithDebInfoAssert/libdispatch-linux-x86_64/src /swift/swift-project/swift-corelibs-libdispatch/src/swift/Block.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Data.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Dispatch.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/IO.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Private.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Queue.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Source.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Time.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Wrapper.swift -no-toolchain-stdlib-rpath -Xlinker -soname -Xlinker libswiftDispatch.so  -L /swift/swift-project/build/Ninja-RelWithDebInfoAssert/libdispatch-linux-x86_64/src/swift  -L /swift/swift-project/build/Ninja-RelWithDebInfoAssert/libdispatch-linux-x86_64  -L /swift/swift-project/build/Ninja-RelWithDebInfoAssert/libdispatch-linux-x86_64   -L /usr/lib/gcc/x86_64-amazon-linux/11 -Xlinker -rpath -Xlinker /swift/swift-project/build/Ninja-RelWithDebInfoAssert/libdispatch-linux-x86_64:  src/swift/libDispatchStubs.a  libdispatch.so  libBlocksRuntime.so  -lstdc++  -lm  -lgcc_s  -lgcc  -lc  -lgcc_s  -lgcc && :
/swift/swift-project/swift-corelibs-libdispatch/src/swift/Dispatch.swift:99:4: warning: default will never be executed
                        default: return nil
                        ^
error: fatalError

art-divin · 2024-01-05T13:24:02.857Z

I see, thanks!

My current progress is by guessing I have built somewhat a debuggable version of libSwiftCore running utils/build-script --release-debuginfo --skip-test-early-swift-driver --skip-test-swift --debug-swift-stdlib --bootstrapping=hosttools --skip-test-swiftdocc (had to disable swiftdocc because it was failing during Generating ... phase).

Now I do have libSwiftCore that I needed in the first place to debug the crash, but the crash itself is linked against system (host) Swift toolchain. Now my task is to link against the debuggable version of libSwiftCore.so somehow tweaking the build script

Registers:
x0 0x0000ffffe9355150  02 00 00 00 00 00 00 00 e0 52 35 e9 ff ff 00 00  ········àR5éÿÿ··

x21 0x0000ffffb8cec45c  48 00 00 00 20 cd d7 ee aa aa 00 00 11 61 00 a0  H··· Í×îªª···a· 
x22 0x0000ffffb8cec458  01 00 00 00 48 00 00 00 20 cd d7 ee aa aa 00 00  ····H··· Í×îªª··
x23 0x0000ffffe9355280  00 00 00 00 00 00 00 00 8c ab aa b8 ff ff 00 00  ·········«ª¸ÿÿ··
x24 0x0000ffffb8be1b68  00 02 00 00 00 00 00 00 a8 bf c0 b8 ff ff 00 00  ········¨¿À¸ÿÿ··
x25 0x0000aaaaeed96a70  02 02 00 00 00 00 00 00 04 d4 c0 b8 ff ff 00 00  ·········ÔÀ¸ÿÿ··
x26 0x0000aaaaeed969f8  01 03 00 00 00 00 00 00 02 00 00 00 00 00 00 00  ················
x27 0x0000ffffb8be3188  00 02 00 00 00 00 00 00 68 ed c0 b8 ff ff 00 00  ········híÀ¸ÿÿ··
x28 0x0000ffffe93553a0  40 ef be b8 ff ff 00 00 02 00 00 00 00 00 00 00  @ï¾¸ÿÿ··········
fp 0x0000ffffe93551e0  a0 52 35 e9 ff ff 00 00 28 fb a9 b8 ff ff 00 00   R5éÿÿ··(û©¸ÿÿ··
lr 0x0000ffffb8a9fe58  00 01 00 b4 08 00 40 f9 e1 03 16 aa c8 2a 00 f9  ···´··@ùá··ªÈ*·ù
sp 0x0000ffffe9354fc0  281474594328512
pc 0x0000ffffb8ab10b8  fd 7b 02 a9 fc 6f 03 a9 fa 67 04 a9 f8 5f 05 a9  ý{·©üo·©úg·©ø_·©
Images (54 omitted):
0x0000aaaabb830000–0x0000aaaac0459a38 <no build ID>                            SourceryPackageTests.xctest /home/tart/Documents/projects/Sourcery-master/.build/aarch64-unknown-linux-gnu/debug/SourceryPackageTests.xctest
0x0000ffffb7320000–0x0000ffffb74a8214 b2eb52eb317bf12f124bc94b4e32643c450e6c9a libc.so.6                   /usr/lib/aarch64-linux-gnu/libc.so.6
0x0000ffffb7860000–0x0000ffffb78b9a10 <no build ID>                            libXCTest.so                /home/tart/.local/share/swiftly/toolchains/5.9.2/usr/lib/swift/linux/libXCTest.so
0x0000ffffb7ab0000–0x0000ffffb81afd90 <no build ID>                            libFoundation.so            /home/tart/.local/share/swiftly/toolchains/5.9.2/usr/lib/swift/linux/libFoundation.so
0x0000ffffb86d0000–0x0000ffffb8bbd230 <no build ID>                            libswiftCore.so             /home/tart/.local/share/swiftly/toolchains/5.9.2/usr/lib/swift/linux/libswiftCore.so

Backtrace took 8.77s
error: Exited with signal code 11

Finagolfin · 2024-01-06T03:07:25.358Z

art-divin:

I have compiled the compiler using utils/build-script --release-debuginfo --skip-test-early-swift-driver --skip-test-swift -T

A lot of redundant flags here, just utils/build-script --release-debuginfo will mostly do the same.

art-divin:

BUT running the utils/build-toolchain according to the DevelopmentTips.md to build a debuggable libSwiftCore brings the error in the OP mentioned by @taylorswift

What distro are you running on? @taylorswift says they're running on amazonlinux:2023, which is not currently officially supported.

art-divin:

The crash report that I have got from reproducing locally contains libSwiftCore path pointing to my local installation of Swift toolchain rather than the one compiled in /build/Ninja-RelWithDebInfoAssert subfolder.

This is the crash report from building libswiftDispatch.so or running the Sourcery tests?

art-divin:

And I am unsure how to make IDE link against libSwiftCore built along with the compiler, rather than the one installed in the system.

If you're unable to build libswiftDispatch.so, that means you don't have a full freshly-built toolchain, so that would mean mixing freshly-built components with a prebuilt toolchain. You have not specified how you are doing this.

Instead of building with --release-debuginfo, which builds everything including the compiler with debug info, you may want to build --release --debug-swift-stdlib instead, since your problem is in the stdlib.

taylorswift:

building libdispatch has never worked (for me), so if i reenable libdispatch, the build once again fails with the same error: fatalError as in this attempt.

You have never specified exactly what build-script command gives you that error: please detail that and list any patches you have applied.

It is possible you have found a genuine build issue, as I notice that most CI presets run with --release --assertions instead, though I do see some that set --release-debuginfo. Try swapping in the former and see if that makes a difference.

art-divin:

My current progress is by guessing I have built somewhat a debuggable version of libSwiftCore running utils/build-script --release-debuginfo --skip-test-early-swift-driver --skip-test-swift --debug-swift-stdlib --bootstrapping=hosttools --skip-test-swiftdocc (had to disable swiftdocc because it was failing during Generating ... phase).

Now I do have libSwiftCore that I needed in the first place to debug the crash, but the crash itself is linked against system (host) Swift toolchain.

This command freshly builds only the compiler and stdlib, not the corelibs like Foundation or the SwiftPM build tool, so you won't be able to build Sourcery with these components alone. You could try swapping the freshly-built debug stdlib into a prebuilt release toolchain, but you'd have to make sure the versions matched, eg you can't swap a trunk debug stdlib into a 5.9 release toolchain.

art-divin · 2024-01-06T06:15:49.178Z

Finagolfin:

What distro are you running on? @taylorswift says they're running on amazonlinux:2023, which is not currently officially supported.

I am running on Ubuntu 22.04.3 LTS codename Jammy. This is run under VM on M1, meaning architecture is aarch64. Is there a list of supported linux OSes somewhere, if you will?

This is the crash report from building libswiftDispatch.so or running the Sourcery tests?

It is a crash from Sourcery. It is connected to this topic - i.e. unable to build libswiftDispatch.so, only in terms of me not being able to build a debuggable toolchain (using utils/build-toolchain) to point to it when compiling Swift compiler, so that I'd be able to debug NSCoder-related routine, why it crashes.

Instead of building with --release-debuginfo , which builds everything including the compiler with debug info, you may want to build --release --debug-swift-stdlib instead, since your problem is in the stdlib

Did that, still the crash would point to libSwiftCore.so installed from swift.org rather the one compiled and located in Ninja-RelWithDebInfoAssert+stdlib-DebugAssert. This is why I am trying to build the toolchain.

This command freshly builds only the compiler and stdlib, not the corelibs like Foundation or the SwiftPM build tool, so you won't be able to build Sourcery with these components alone. You could try swapping the freshly-built debug stdlib into a prebuilt release toolchain, but you'd have to make sure the versions matched, eg you can't swap a trunk debug stdlib into a 5.9 release toolchain.

Oh, wow, thanks for the insight! In order to build SPM and foundation, I'd set these flags as well - --debug-foundation - in order to be able to use only what's under build/Ninja... to reproduce the crash, right?

I am not clear about why exactly when I run Build/Ninja-RelWithDebInfoAssert+stdlib-DebugAssert/swift/bin/swift test in Sourcery working directory, crashing links against libSwiftCore which is located in a system-wide-installed toolchain (as far as I tried, there's no way to build Swift compiler without having system-wide installed Swift toolchain).
In fact, I have this many libSwiftCore.so under Ninja... build folder, which is quite confusing

image2020×840 214 KB

Finagolfin · 2024-01-07T16:09:49.064Z

art-divin:

Is there a list of supported linux OSes somewhere, if you will?

The README lists most of the officially supported OS's, with only Windows missing for some reason. If you are able to reproduce @taylorswift's crash on Ubuntu, that implies this is a genuine build issue.

art-divin:

I am not clear about why exactly when I run Build/Ninja-RelWithDebInfoAssert+stdlib-DebugAssert/swift/bin/swift test in Sourcery working directory, crashing links against libSwiftCore which is located in a system-wide-installed toolchain (as far as I tried, there's no way to build Swift compiler without having system-wide installed Swift toolchain).

I am not sure how that runs for you at all, as you can't have a freshly-built swift-test without building libswiftDispatch.so first. Maybe the new swift-driver just invokes the system swift-test if it can't find one next to Build/Ninja-RelWithDebInfoAssert+stdlib-DebugAssert/swift/bin/swift, which would explain why it's still using the non-debug stdlib.

What I suggest to you is to simply build the debug stdlib for the prebuilt toolchain you're using, by first checking out that source tag then running utils/build-script --release-debuginfo --skip-build-cmark --build-llvm=0 --build-swift-tools=0 to only build the debug stdlib. Finally, copy over that debug stdlib into your prebuilt toolchain with something like cp Build/Ninja-RelWithDebInfoAssert+stdlib-DebugAssert/swift/lib/swift/linux/libswift*.so swift-5.9.2-RELEASE/usr/lib/swift/linux/, use it to build the Sourcery tests with cd Sourcery; ../swift-5.9.2-RELEASE/usr/bin/swift build --build-tests, then run the test runner in the debugger to get a proper backtrace and debug it, ../swift-5.9.2-RELEASE/usr/bin/lldb .build/x86_64-unknown-linux-gnu/debug/SourceryPackageTests.xctest.

Given the build issue you're seeing, this may be the best way for you to debug your Sourcery issue.

taylorswift · 2024-01-13T02:18:49.838Z

Finagolfin:

You have never specified exactly what build-script command gives you that error: please detail that and list any patches you have applied.

i was using the last full invocation you suggested, which was:

Finagolfin:

Now try running utils/build-script --release-debuginfo -T --xctest --skip-test-swift

i no longer have any patches applied, as i am now attempting to build libdispatch.

the build usually fails for one of two reasons:

the swift-driver segfaults, as described here, or
building libswiftDispatch.so fails with the fatalError, as described here.

the second failure of course didn’t occur when i manually disabled libdispatch as you suggested. but as i understand it, i need to build libdispatch in order to build the toolchain.

i have tried swapping out the original command with:

$ utils/build-script --release --assertions -T --xctest --skip-test-swift

but i get the exact same results.

Finagolfin · 2024-01-13T05:10:13.660Z

taylorswift:

i have tried swapping out the original command with:
$ utils/build-script --release --assertions -T --xctest --skip-test-swift
but i get the exact same results.

Huh, this suggests the problem is in your environment, as that works on the linux CI, at least when the trunk compiler is built with Swift 5.8.

Can you try running the failing release-debuginfo compilation command through the debugger to get a backtrace?

lldb /swift/swift-project/build/buildbot_linux/swift-linux-x86_64/bin/swift-frontend -- -frontend -c /swift/swift-project/..

That may just exit, so you might have to set a breakpoint on fatalError() or some other method like that.

taylorswift · 2024-01-15T23:10:49.755Z

Finagolfin:

Can you try running the failing release-debuginfo compilation command through the debugger to get a backtrace?
lldb /swift/swift-project/build/buildbot_linux/swift-linux-x86_64/bin/swift-frontend -- -frontend -c /swift/swift-project/..
That may just exit, so you might have to set a breakpoint on fatalError() or some other method like that.

how do i identify the failing compilation command? i’ve never invoked swift-frontend directly, and i’m not familiar with how that tool fits into the higher-level utils/build-script command i’ve been running.

i wondered if you were referring to one of the bash commands that the build script echos to the terminal, such as the one that begins with

: && /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swiftc

when i tried copying it and running it in isolation, it failed with:

/swift/swift-project/swift$ /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swiftc -output-file-map src/swift/CMakeFiles/swiftDispatch.dir/Release/output-file-map.json -incremental -j 12 -emit-library -o libswiftDispatch.so -module-name Dispatch -module-link-name swiftDispatch -emit-module -emit-module-path src/swift/swift/Dispatch.swiftmodule -emit-dependencies -DswiftDispatch_EXPORTS -module-cache-path "/swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/module-cache" -O -Xcc -fblocks -Xcc -fmodule-map-file=/swift/swift-project/swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I/swift/swift-project/swift-corelibs-libdispatch -Xcc -I/swift/swift-project/swift-corelibs-libdispatch/src/swift/shims -vfsoverlay /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/dispatch-vfs-overlay.yaml -I /swift/swift-project/swift-corelibs-libdispatch/src/BlocksRuntime -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64 -I /swift/swift-project/swift-corelibs-libdispatch -I /swift/swift-project/swift-corelibs-libdispatch/src -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src /swift/swift-project/swift-corelibs-libdispatch/src/swift/Block.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Data.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Dispatch.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/IO.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Private.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Queue.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Source.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Time.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Wrapper.swift -no-toolchain-stdlib-rpath -Xlinker -soname -Xlinker libswiftDispatch.so  -L /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src/swift  -L /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64  -L /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64   -L /usr/lib/gcc/x86_64-amazon-linux/11 -Xlinker -rpath -Xlinker /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64:  src/swift/libDispatchStubs.a  libdispatch.so  libBlocksRuntime.so  -lstdc++  -lm  -lgcc_s  -lgcc  -lc  -lgcc_s  -lgcc
error: unable to load output file map 'src/swift/CMakeFiles/swiftDispatch.dir/Release/output-file-map.json': FileSystemError(kind: TSCBasic.FileSystemError.Kind.noEntry, path: Optional(<AbsolutePath:"/swift/swift-project/swift/src/swift/CMakeFiles/swiftDispatch.dir/Release/output-file-map.json">))

this is of course, not the same error i get when i run the build script as a whole (that ends in error: fatalError), but i also don’t know enough about how the build script works to know if that result is even meaningful.

Finagolfin · 2024-01-16T02:53:38.293Z

taylorswift:

how do i identify the failing compilation command?

It should be the last one from the verbose output you listed in the third post in this thread. Try running that swift-frontend command alone to make sure. If it works, it may be one of the earlier commands, as it is running 12 simultaneously, so try each of the previous 11 commands too.

taylorswift:

i’ve never invoked swift-frontend directly, and i’m not familiar with how that tool fits into the higher-level utils/build-script command i’ve been running.

build-script simply invokes CMake to configure each repo, then ninja to build them. That will invoke the appropriate compiler for the source, clang for C/C++ code and swiftc for Swift code. swiftc invokes the standalone swift-driver binary to parse compilation options passed to it, then invokes swift-frontend to compile Swift source to binary objects, swift-autolink-extract to extract which libraries to link against, and finally clang to link all those objects and libraries together.

Since the last command you listed in the verbose output is swift-frontend, that is most likely the one crashing and the one you want to examine in lldb. Note the -- I added after swift-frontend above to make sure its compilation options are not passed to lldb instead.

taylorswift · 2024-01-16T22:31:04.729Z

Finagolfin:

It should be the last one from the verbose output you listed in the third post in this thread. Try running that swift-frontend command alone to make sure. If it works, it may be one of the earlier commands, as it is running 12 simultaneously, so try each of the previous 11 commands too.

okay, so that means i was running the subcommand in the wrong directory. so, retracing those steps:

$ cd ../build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/
$ /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swiftc -v -output-file-map src/swift/CMakeFiles/swiftDispatch.dir/Release/output-file-map.json -incremental -j 12 -emit-library -o libswiftDispatch.so -module-name Dispatch -module-link-name swiftDispatch -emit-module -emit-module-path src/swift/swift/Dispatch.swiftmodule -emit-dependencies -DswiftDispatch_EXPORTS -module-cache-path "/swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/module-cache" -O -Xcc -fblocks -Xcc -fmodule-map-file=/swift/swift-project/swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I/swift/swift-project/swift-corelibs-libdispatch -Xcc -I/swift/swift-project/swift-corelibs-libdispatch/src/swift/shims -vfsoverlay /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/dispatch-vfs-overlay.yaml -I /swift/swift-project/swift-corelibs-libdispatch/src/BlocksRuntime -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64 -I /swift/swift-project/swift-corelibs-libdispatch -I /swift/swift-project/swift-corelibs-libdispatch/src -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src /swift/swift-project/swift-corelibs-libdispatch/src/swift/Block.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Data.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Dispatch.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/IO.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Private.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Queue.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Source.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Time.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Wrapper.swift -no-toolchain-stdlib-rpath -Xlinker -soname -Xlinker libswiftDispatch.so  -L /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src/swift  -L /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64  -L /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64   -L /usr/lib/gcc/x86_64-amazon-linux/11 -Xlinker -rpath -Xlinker /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64:  src/swift/libDispatchStubs.a  libdispatch.so  libBlocksRuntime.so  -lstdc++  -lm  -lgcc_s  -lgcc  -lc  -lgcc_s  -lgcc

Swift version 5.11-dev (LLVM 497a64a78b2fba6, Swift 0bbcae4e9278768)
Target: x86_64-unknown-linux-gnu
/swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift-frontend -frontend -c -primary-file /swift/swift-project/swift-corelibs-libdispatch/src/swift/Block.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Data.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Dispatch.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/IO.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Private.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Queue.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Source.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Time.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Wrapper.swift -emit-module-path src/swift/CMakeFiles/swiftDispatch.dir/Block.swift.swiftmodule -emit-module-doc-path src/swift/CMakeFiles/swiftDispatch.dir/Block.swift.swiftdoc -emit-module-source-info-path src/swift/CMakeFiles/swiftDispatch.dir/Block.swift.swiftsourceinfo -emit-dependencies-path src/swift/CMakeFiles/swiftDispatch.dir/Block.swift.o.d -emit-reference-dependencies-path src/swift/CMakeFiles/swiftDispatch.dir/Block.swift.o.swiftdeps -target x86_64-unknown-linux-gnu -disable-objc-interop -I /swift/swift-project/swift-corelibs-libdispatch/src/BlocksRuntime -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64 -I /swift/swift-project/swift-corelibs-libdispatch -I /swift/swift-project/swift-corelibs-libdispatch/src -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src -vfsoverlay /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/dispatch-vfs-overlay.yaml -color-diagnostics -module-cache-path /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/module-cache -module-link-name swiftDispatch -O -D swiftDispatch_EXPORTS -empty-abi-descriptor -resource-dir /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift -Xcc -fblocks -Xcc -fmodule-map-file=/swift/swift-project/swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I/swift/swift-project/swift-corelibs-libdispatch -Xcc -I/swift/swift-project/swift-corelibs-libdispatch/src/swift/shims -module-name Dispatch -plugin-path /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/host/plugins -plugin-path /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/local/lib/swift/host/plugins -enable-default-cmo -parse-as-library -o src/swift/CMakeFiles/swiftDispatch.dir/Block.swift.o
/swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift-frontend -frontend -c /swift/swift-project/swift-corelibs-libdispatch/src/swift/Block.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Data.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Dispatch.swift -primary-file /swift/swift-project/swift-corelibs-libdispatch/src/swift/IO.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Private.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Queue.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Source.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Time.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Wrapper.swift -emit-module-path src/swift/CMakeFiles/swiftDispatch.dir/IO.swift.swiftmodule -emit-module-doc-path src/swift/CMakeFiles/swiftDispatch.dir/IO.swift.swiftdoc -emit-module-source-info-path src/swift/CMakeFiles/swiftDispatch.dir/IO.swift.swiftsourceinfo -emit-dependencies-path src/swift/CMakeFiles/swiftDispatch.dir/IO.swift.o.d -emit-reference-dependencies-path src/swift/CMakeFiles/swiftDispatch.dir/IO.swift.o.swiftdeps -target x86_64-unknown-linux-gnu -disable-objc-interop -I /swift/swift-project/swift-corelibs-libdispatch/src/BlocksRuntime -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64 -I /swift/swift-project/swift-corelibs-libdispatch -I /swift/swift-project/swift-corelibs-libdispatch/src -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src -vfsoverlay /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/dispatch-vfs-overlay.yaml -color-diagnostics -module-cache-path /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/module-cache -module-link-name swiftDispatch -O -D swiftDispatch_EXPORTS -empty-abi-descriptor -resource-dir /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift -Xcc -fblocks -Xcc -fmodule-map-file=/swift/swift-project/swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I/swift/swift-project/swift-corelibs-libdispatch -Xcc -I/swift/swift-project/swift-corelibs-libdispatch/src/swift/shims -module-name Dispatch -plugin-path /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/host/plugins -plugin-path /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/local/lib/swift/host/plugins -enable-default-cmo -parse-as-library -o src/swift/CMakeFiles/swiftDispatch.dir/IO.swift.o
/swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift-frontend -frontend -c /swift/swift-project/swift-corelibs-libdispatch/src/swift/Block.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Data.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Dispatch.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/IO.swift -primary-file /swift/swift-project/swift-corelibs-libdispatch/src/swift/Private.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Queue.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Source.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Time.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Wrapper.swift -emit-module-path src/swift/CMakeFiles/swiftDispatch.dir/Private.swift.swiftmodule -emit-module-doc-path src/swift/CMakeFiles/swiftDispatch.dir/Private.swift.swiftdoc -emit-module-source-info-path src/swift/CMakeFiles/swiftDispatch.dir/Private.swift.swiftsourceinfo -emit-dependencies-path src/swift/CMakeFiles/swiftDispatch.dir/Private.swift.o.d -emit-reference-dependencies-path src/swift/CMakeFiles/swiftDispatch.dir/Private.swift.o.swiftdeps -target x86_64-unknown-linux-gnu -disable-objc-interop -I /swift/swift-project/swift-corelibs-libdispatch/src/BlocksRuntime -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64 -I /swift/swift-project/swift-corelibs-libdispatch -I /swift/swift-project/swift-corelibs-libdispatch/src -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src -vfsoverlay /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/dispatch-vfs-overlay.yaml -color-diagnostics -module-cache-path /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/module-cache -module-link-name swiftDispatch -O -D swiftDispatch_EXPORTS -empty-abi-descriptor -resource-dir /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift -Xcc -fblocks -Xcc -fmodule-map-file=/swift/swift-project/swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I/swift/swift-project/swift-corelibs-libdispatch -Xcc -I/swift/swift-project/swift-corelibs-libdispatch/src/swift/shims -module-name Dispatch -plugin-path /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/host/plugins -plugin-path /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/local/lib/swift/host/plugins -enable-default-cmo -parse-as-library -o src/swift/CMakeFiles/swiftDispatch.dir/Private.swift.o
/swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift-frontend -frontend -c /swift/swift-project/swift-corelibs-libdispatch/src/swift/Block.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Data.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Dispatch.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/IO.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Private.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Queue.swift -primary-file /swift/swift-project/swift-corelibs-libdispatch/src/swift/Source.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Time.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Wrapper.swift -emit-module-path src/swift/CMakeFiles/swiftDispatch.dir/Source.swift.swiftmodule -emit-module-doc-path src/swift/CMakeFiles/swiftDispatch.dir/Source.swift.swiftdoc -emit-module-source-info-path src/swift/CMakeFiles/swiftDispatch.dir/Source.swift.swiftsourceinfo -emit-dependencies-path src/swift/CMakeFiles/swiftDispatch.dir/Source.swift.o.d -emit-reference-dependencies-path src/swift/CMakeFiles/swiftDispatch.dir/Source.swift.o.swiftdeps -target x86_64-unknown-linux-gnu -disable-objc-interop -I /swift/swift-project/swift-corelibs-libdispatch/src/BlocksRuntime -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64 -I /swift/swift-project/swift-corelibs-libdispatch -I /swift/swift-project/swift-corelibs-libdispatch/src -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src -vfsoverlay /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/dispatch-vfs-overlay.yaml -color-diagnostics -module-cache-path /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/module-cache -module-link-name swiftDispatch -O -D swiftDispatch_EXPORTS -empty-abi-descriptor -resource-dir /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift -Xcc -fblocks -Xcc -fmodule-map-file=/swift/swift-project/swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I/swift/swift-project/swift-corelibs-libdispatch -Xcc -I/swift/swift-project/swift-corelibs-libdispatch/src/swift/shims -module-name Dispatch -plugin-path /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/host/plugins -plugin-path /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/local/lib/swift/host/plugins -enable-default-cmo -parse-as-library -o src/swift/CMakeFiles/swiftDispatch.dir/Source.swift.o
/swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift-frontend -frontend -c /swift/swift-project/swift-corelibs-libdispatch/src/swift/Block.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Data.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Dispatch.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/IO.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Private.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Queue.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Source.swift -primary-file /swift/swift-project/swift-corelibs-libdispatch/src/swift/Time.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Wrapper.swift -emit-module-path src/swift/CMakeFiles/swiftDispatch.dir/Time.swift.swiftmodule -emit-module-doc-path src/swift/CMakeFiles/swiftDispatch.dir/Time.swift.swiftdoc -emit-module-source-info-path src/swift/CMakeFiles/swiftDispatch.dir/Time.swift.swiftsourceinfo -emit-dependencies-path src/swift/CMakeFiles/swiftDispatch.dir/Time.swift.o.d -emit-reference-dependencies-path src/swift/CMakeFiles/swiftDispatch.dir/Time.swift.o.swiftdeps -target x86_64-unknown-linux-gnu -disable-objc-interop -I /swift/swift-project/swift-corelibs-libdispatch/src/BlocksRuntime -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64 -I /swift/swift-project/swift-corelibs-libdispatch -I /swift/swift-project/swift-corelibs-libdispatch/src -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src -vfsoverlay /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/dispatch-vfs-overlay.yaml -color-diagnostics -module-cache-path /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/module-cache -module-link-name swiftDispatch -O -D swiftDispatch_EXPORTS -empty-abi-descriptor -resource-dir /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift -Xcc -fblocks -Xcc -fmodule-map-file=/swift/swift-project/swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I/swift/swift-project/swift-corelibs-libdispatch -Xcc -I/swift/swift-project/swift-corelibs-libdispatch/src/swift/shims -module-name Dispatch -plugin-path /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/host/plugins -plugin-path /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/local/lib/swift/host/plugins -enable-default-cmo -parse-as-library -o src/swift/CMakeFiles/swiftDispatch.dir/Time.swift.o
error: fatalError

(why is it using a 5.11 compiler?)

i only see 5 swift-frontend invocations here, and running them each invididually succeeds:

Block.swift.o
IO.swift.o
Private.swift.o
Source.swift.o
Time.swift.o

so i assume it must be an error in the swiftc invocation.

attaching lldb did not work initially, due to this issue.

lldb: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory

i tried to adapt your suggestion in that thread by running:

$ sudo ln -s /usr/lib64/libpython3.9.so.1.0 /usr/lib64/libpython3.7m.so.1.0

that got lldb to launch, but it just failed later with

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'lldb'

fortunately this is a well-known issue and on Amazon Linux 2023 you can just run:

$ sudo yum install python3-lldb

but then i get a different python error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib64/python3.9/site-packages/lldb/__init__.py", line 952, in <module>
    eArgTypeReproducerProvider = _lldb.eArgTypeReproducerProvider
AttributeError: module '_lldb' has no attribute 'eArgTypeReproducerProvider'

the error always occurs when running lldb, even just lldb --version

$ lldb --version
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib64/python3.9/site-packages/lldb/__init__.py", line 952, in <module>
    eArgTypeReproducerProvider = _lldb.eArgTypeReproducerProvider
AttributeError: module '_lldb' has no attribute 'eArgTypeReproducerProvider'
lldb version 13.0.0git (https://github.com/apple/llvm-project.git revision 2b42c5ce063a374fb22676e27505a22fe411ea8c)
Swift version 5.9.2 (swift-5.9.2-RELEASE)

this appears to be a new problem in swift 5.9, and i have not been able to find a solution yet.

EDIT: one hypothesis is that lldb (which comes from /opt/swift/5.9.2/usr/bin/lldb) is version 13, but the version of python-lldb installed is 15.0.7. unfortunately, i do not know of a way to install an older version of python-lldb:

$ yum --showduplicates list python3-lldb | expand
Last metadata expiration check: 0:01:41 ago on Tue Jan 16 22:40:56 2024.
Installed Packages
python3-lldb.x86_64              15.0.7-3.amzn2023.0.1              @amazonlinux
Available Packages
python3-lldb.x86_64              15.0.6-1.amzn2023.0.3              amazonlinux 
python3-lldb.x86_64              15.0.7-3.amzn2023.0.1              amazonlinux

Finagolfin · 2024-01-17T11:38:54.994Z

taylorswift:

why is it using a 5.11 compiler?

You are building the main branch from source, right? That is currently Swift 5.11, planned to branch off in a couple months and release near the end of the year. Once a fresh 5.11 compiler has been built in /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/, it is used to build the corelibs like libdispatch or foundation and SwiftPM. The prebuilt host 5.9 compiler is only used to build some components run inside of the fresh 5.11 compiler, like the swift-syntax parser and early swift-driver.

taylorswift:

i only see 5 swift-frontend invocations here, and running them each invididually succeeds:

Block.swift.o

IO.swift.o

Private.swift.o

Source.swift.o

Time.swift.o

so i assume it must be an error in the swiftc invocation.

I was worried that it would be some other command we don't see that's failing: I'm dubious that it is the swiftc invocation failing in the middle though.

Can you try the remaining four swift-frontend invocations by hand, as swiftc is run with 9 Swift files from libdispatch? It shouldn't be hard to modify one of the existing swift-frontend invocations to generate the last four invocations.

If all 9 work, then it may be swiftc that's failing.

taylorswift:

attaching lldb did not work initially

If you're having problems with the Swift-forked lldb that comes with the Swift 5.9 toolchain, just use your normal system lldb to get a backtrace. The function symbols may not be demangled but that's good enough.

taylorswift · 2024-01-17T22:32:23.131Z

Finagolfin:

I was worried that it would be some other command we don't see that's failing: I'm dubious that it is the swiftc invocation failing in the middle though.

Can you try the remaining four swift-frontend invocations by hand, as swiftc is run with 9 Swift files from libdispatch? It shouldn't be hard to modify one of the existing swift-frontend invocations to generate the last four invocations.

If all 9 work, then it may be swiftc that's failing.

so on closer inspection, i realized that each time i invoke swiftc, a different subset of 3-5 files gets compiled for some reason. so i didn’t have to reconstruct those commands, i just ran the swiftc command several times to sample the swift-frontend commands, and i ran them without the final -o to track my progress. ultimately, i was able to manually compile all nine .o files without errors.

$ ls -lt
total 2240
-rw-rw-r-- 1 ec2-user ec2-user  58816 Jan 17 22:06 Wrapper.o
-rw-rw-r-- 1 ec2-user ec2-user  61944 Jan 17 22:02 Queue.o
-rw-rw-r-- 1 ec2-user ec2-user  33208 Jan 17 22:01 Block.o
-rw-rw-r-- 1 ec2-user ec2-user  14720 Jan 17 22:01 Private.o
-rw-rw-r-- 1 ec2-user ec2-user  46112 Jan 17 22:00 IO.o
-rw-rw-r-- 1 ec2-user ec2-user  56704 Jan 17 22:00 Source.o
-rw-rw-r-- 1 ec2-user ec2-user  72008 Jan 17 21:59 Data.o
-rw-rw-r-- 1 ec2-user ec2-user  18256 Jan 17 21:59 Time.o
-rw-rw-r-- 1 ec2-user ec2-user  27200 Jan 17 21:57 Dispatch.o

Finagolfin:

If you're having problems with the Swift-forked lldb that comes with the Swift 5.9 toolchain, just use your normal system lldb to get a backtrace.

i tried doing this and setting a breakpoint on fatalError, which did not reveal anything in swiftc.

$ /usr/bin/lldb /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swiftc -- -v -output-file-map src/swift/CMakeFiles/swiftDispatch.dir/Release/output-file-map.json -incremental -j 12 -emit-library -o libswiftDispatch.so -module-name Dispatch -module-link-name swiftDispatch -emit-module -emit-module-path src/swift/swift/Dispatch.swiftmodule -emit-dependencies -DswiftDispatch_EXPORTS -module-cache-path "/swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/module-cache" -O -Xcc -fblocks -Xcc -fmodule-map-file=/swift/swift-project/swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I/swift/swift-project/swift-corelibs-libdispatch -Xcc -I/swift/swift-project/swift-corelibs-libdispatch/src/swift/shims -vfsoverlay /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/dispatch-vfs-overlay.yaml -I /swift/swift-project/swift-corelibs-libdispatch/src/BlocksRuntime -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64 -I /swift/swift-project/swift-corelibs-libdispatch -I /swift/swift-project/swift-corelibs-libdispatch/src -I /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src /swift/swift-project/swift-corelibs-libdispatch/src/swift/Block.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Data.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Dispatch.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/IO.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Private.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Queue.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Source.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Time.swift /swift/swift-project/swift-corelibs-libdispatch/src/swift/Wrapper.swift -no-toolchain-stdlib-rpath -Xlinker -soname -Xlinker libswiftDispatch.so  -L /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64/src/swift  -L /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64  -L /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64   -L /usr/lib/gcc/x86_64-amazon-linux/11 -Xlinker -rpath -Xlinker /swift/swift-project/build/Ninja-ReleaseAssert/libdispatch-linux-x86_64:  src/swift/libDispatchStubs.a  libdispatch.so  libBlocksRuntime.so  -lstdc++  -lm  -lgcc_s  -lgcc  -lc  -lgcc_s  -lgcc

(lldb) b fatalError
Breakpoint 1: no locations (pending).
WARNING:  Unable to resolve breakpoint to any actual locations.
(lldb) r
Process 96 launched: '/swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swiftc' (x86_64)
1 location added to breakpoint 1
Swift version 5.11-dev (LLVM 497a64a78b2fba6, Swift 0bbcae4e9278768)
Target: x86_64-unknown-linux-gnu
Process 96 exited with status = -1 (0xffffffff) lost connection
(lldb) breakpoint list
Current breakpoints:
1: name = 'fatalError', locations = 1, resolved = 1, hit count = 0
  1.1: where = libswiftCore.so`swift::fatalError(unsigned int, char const*, ...), address = 0x00007ffff76bb390, resolved, hit count = 0 

(lldb)

either i did not set the breakpoint correctly, or swiftc is merely parroting a fatalError printed by swift-driver.

Finagolfin · 2024-01-18T07:39:24.217Z

taylorswift:

ultimately, i was able to manually compile all nine .o files without errors.

That's good, that means swift-frontend is unlikely to be the issue as we originally suspected.

taylorswift:

either i did not set the breakpoint correctly, or swiftc is merely parroting a fatalError printed by swift-driver.

Likely the latter, I have not previously looked into exactly which process launches the swift-frontend process, swiftc or swift-driver, but it probably is the latter. Given the early swift-driver flakiness you already reported, it is probably the point of failure. Unfortunately, that is currently tough to debug, as the verbose output does not list the swift-driver invocation it is using.

Try this: run the exact swiftc command spit out by CMake but add the flags -v -disallow-use-new-driver. That invokes the legacy Driver written in C++, so if that works, you know where the problem is.

You can then avoid the early swift-driver altogether by passing --skip-early-swift-driver to build-script, after deleting the one you already built, rm /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift-driver.

taylorswift · 2024-01-18T20:57:27.161Z

Finagolfin:

Try this: run the exact swiftc command spit out by CMake but add the flags -v -disallow-use-new-driver. That invokes the legacy Driver written in C++, so if that works, you know where the problem is.

ha, i can confirm this fixes the issue; manually running the swiftc command with that option succeeds.

unfortunately, i can’t seem to disable the swift driver when building the toolchain using build-script, even when i deleted the one i compiled and adding --skip-early-swift-driver. the exact steps i did were:

$ rm /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift-driver
$ ls /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/
complete-test		 swift-ast-script		 swift-llvm-opt
lldb-moduleimport-test	 swift-autolink-extract		 swift-parse-test
lldb-with-tools		 swift-backtrace-linux-x86_64	 swift-plugin-server
sil-func-extractor	 swift-cache-tool		 swift-refactor
sil-llvm-gen		 swift-compatibility-symbols	 swift-reflection-dump
sil-nm			 swift-def-to-strings-converter  swift-reflection-test-LINUX
sil-opt			 swift-demangle			 swift-reflection-test-linux-x86_64
sil-passpipeline-dumper  swift-demangle-yamldump	 swift-remoteast-test
sourcekitd-repl		 swift-dependency-tool		 swift-scan-test
sourcekitd-test		 swift-frontend			 swift-serialize-diagnostics
swift			 swift-help			 swift-symbolgraph-extract
swift-api-digester	 swift-ide-test			 swiftc
swift-api-dump.py	 swift-indent			 swiftc-legacy-driver
swift-api-extract	 swift-legacy-driver
$ utils/build-script --release --assertions -T --xctest --skip-test-swift --skip-early-swift-driver --reconfigure

despite the --skip-early-swift-driver option, the build fails with the same fatalError as before, and moreover, the swift-driver binary reappears in swift-linux-x86_64/bin/.

Finagolfin · 2024-01-18T22:01:16.784Z

taylorswift:

ha, i can confirm this fixes the issue; manually running the swiftc command with that option succeeds.

taylorswift:

the swift-driver binary reappears in swift-linux-x86_64/bin/.

You may need a --reconfigure passed to build-script too. If that doesn't work, try deleting the compiler frontend build, rm -rf /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/.

taylorswift · 2024-01-18T23:57:09.769Z

restarting the build from scratch worked for me thanks!

i guess the next step for me is to actually get an installable toolchain packaged. i tried naïvely adding --skip-early-swift-driver to --preset buildbot_linux, but that doesn’t seem to parse. is there a way to override this setting while still getting something similar to the downloadable toolchains on swift.org?

Finagolfin · 2024-01-19T06:52:47.250Z

taylorswift:

restarting the build from scratch worked for me thanks!

Great.

taylorswift:

i tried naïvely adding --skip-early-swift-driver to --preset buildbot_linux, but that doesn’t seem to parse. is there a way to override this setting while still getting something similar to the downloadable toolchains on swift.org?

I thought that should work, if you place the additional flag after the preset flag. If it doesn't, you can always modify the preset itself.

taylorswift · 2024-01-26T23:57:57.966Z

after much trial and error, i was finally able to build a plausible toolchain using the following custom preset:

[preset: buildbot_linux_amazon_linux_2023]
mixin-preset=buildbot_linux
skip-early-swift-driver
skip-test-swift
skip-test-lldb
skip-test-swift-driver
skip-test-swiftpm

$ utils/build-script \
--preset buildbot_linux_amazon_linux_2023 \
install_destdir=/swift/ec2-user/swift-install \
installable_package=/swift/ec2-user/swift-DEVELOPMENT-SNAPSHOT-$(date +'%F')-a.tar.gz

most of the tests had to be disabled because they don’t particularly like that swift-driver is missing, so i guess only time will tell if this toolchain will actually be usable in production.

as a smoke test, i tried installing this toolchain and compiling the swift-nio project, which worked fine.

$ sudo rm -r /opt/swift/
$ sudo cp -r ../../ec2-user/swift-install/usr/bin/* /usr/bin
$ sudo cp -r ../../ec2-user/swift-install/usr/include/* /usr/include/
$ sudo cp -r ../../ec2-user/swift-install/usr/libexec/* /usr/libexec/
$ sudo cp -r ../../ec2-user/swift-install/usr/local/* /usr/local 
$ sudo cp -r ../../ec2-user/swift-install/usr/share/* /usr/share
$ source ~/.bashrc

$ swift --version
Swift version 5.11-dev (LLVM beb97e1115a1901, Swift ec7a7c6a966fbc9)
Target: x86_64-unknown-linux-gnu
$ git clone https://github.com/apple/swift-nio
$ swift build

thank you @Finagolfin for all your help!

Finagolfin · 2024-01-27T11:46:36.351Z

taylorswift:

most of the tests had to be disabled because they don’t particularly like that swift-driver is missing

I don't think that's the reason, as the compiler simply uses the legacy Driver written in C++ instead. What failed for those other three repos whose tests you disabled, other than the single Reflection test in the compiler validation suite before? And that's not "most of the tests," it looks like the corelibs and all other tests must have passed for you.

taylorswift:

i guess only time will tell if this toolchain will actually be usable in production.

There's a guy from Amazon working on an official toolchain, @sebsto, you should talk to him about getting ALI2023 in as an official platform.

taylorswift:

thank you @Finagolfin for all your help!

Glad to hear it is working. You should try building it again with this toolchain and see if the early swift-driver crashes recur.