how do i identify the failing compilation command? i’ve never invoked swift-frontend directly, and i’m not familiar with how that tool fits into the higher-level utils/build-script command i’ve been running.
i wondered if you were referring to one of the bash commands that the build script echos to the terminal, such as the one that begins with
this is of course, not the same error i get when i run the build script as a whole (that ends in error: fatalError), but i also don’t know enough about how the build script works to know if that result is even meaningful.
It should be the last one from the verbose output you listed in the third post in this thread. Try running that swift-frontend command alone to make sure. If it works, it may be one of the earlier commands, as it is running 12 simultaneously, so try each of the previous 11 commands too.
build-script simply invokes CMake to configure each repo, then ninja to build them. That will invoke the appropriate compiler for the source, clang for C/C++ code and swiftc for Swift code. swiftc invokes the standalone swift-driver binary to parse compilation options passed to it, then invokes swift-frontend to compile Swift source to binary objects, swift-autolink-extract to extract which libraries to link against, and finally clang to link all those objects and libraries together.
Since the last command you listed in the verbose output is swift-frontend, that is most likely the one crashing and the one you want to examine in lldb. Note the -- I added after swift-frontend above to make sure its compilation options are not passed to lldb instead.
that got lldb to launch, but it just failed later with
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'lldb'
fortunately this is a well-known issue and on Amazon Linux 2023 you can just run:
$ sudo yum install python3-lldb
but then i get a different python error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib64/python3.9/site-packages/lldb/__init__.py", line 952, in <module>
eArgTypeReproducerProvider = _lldb.eArgTypeReproducerProvider
AttributeError: module '_lldb' has no attribute 'eArgTypeReproducerProvider'
the error always occurs when running lldb, even just lldb --version
$ lldb --version
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib64/python3.9/site-packages/lldb/__init__.py", line 952, in <module>
eArgTypeReproducerProvider = _lldb.eArgTypeReproducerProvider
AttributeError: module '_lldb' has no attribute 'eArgTypeReproducerProvider'
lldb version 13.0.0git (https://github.com/apple/llvm-project.git revision 2b42c5ce063a374fb22676e27505a22fe411ea8c)
Swift version 5.9.2 (swift-5.9.2-RELEASE)
EDIT: one hypothesis is that lldb (which comes from /opt/swift/5.9.2/usr/bin/lldb) is version 13, but the version of python-lldb installed is 15.0.7. unfortunately, i do not know of a way to install an older version of python-lldb:
$ yum --showduplicates list python3-lldb | expand
Last metadata expiration check: 0:01:41 ago on Tue Jan 16 22:40:56 2024.
Installed Packages
python3-lldb.x86_64 15.0.7-3.amzn2023.0.1 @amazonlinux
Available Packages
python3-lldb.x86_64 15.0.6-1.amzn2023.0.3 amazonlinux
python3-lldb.x86_64 15.0.7-3.amzn2023.0.1 amazonlinux
You are building the main branch from source, right? That is currently Swift 5.11, planned to branch off in a couple months and release near the end of the year. Once a fresh 5.11 compiler has been built in /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/, it is used to build the corelibs like libdispatch or foundation and SwiftPM. The prebuilt host 5.9 compiler is only used to build some components run inside of the fresh 5.11 compiler, like the swift-syntax parser and early swift-driver.
I was worried that it would be some other command we don't see that's failing: I'm dubious that it is the swiftc invocation failing in the middle though.
Can you try the remaining four swift-frontend invocations by hand, as swiftc is run with 9 Swift files from libdispatch? It shouldn't be hard to modify one of the existing swift-frontend invocations to generate the last four invocations.
If all 9 work, then it may be swiftc that's failing.
If you're having problems with the Swift-forked lldb that comes with the Swift 5.9 toolchain, just use your normal system lldb to get a backtrace. The function symbols may not be demangled but that's good enough.
so on closer inspection, i realized that each time i invoke swiftc, a different subset of 3-5 files gets compiled for some reason. so i didn’t have to reconstruct those commands, i just ran the swiftc command several times to sample the swift-frontend commands, and i ran them without the final -o to track my progress. ultimately, i was able to manually compile all nine .o files without errors.
$ ls -lt
total 2240
-rw-rw-r-- 1 ec2-user ec2-user 58816 Jan 17 22:06 Wrapper.o
-rw-rw-r-- 1 ec2-user ec2-user 61944 Jan 17 22:02 Queue.o
-rw-rw-r-- 1 ec2-user ec2-user 33208 Jan 17 22:01 Block.o
-rw-rw-r-- 1 ec2-user ec2-user 14720 Jan 17 22:01 Private.o
-rw-rw-r-- 1 ec2-user ec2-user 46112 Jan 17 22:00 IO.o
-rw-rw-r-- 1 ec2-user ec2-user 56704 Jan 17 22:00 Source.o
-rw-rw-r-- 1 ec2-user ec2-user 72008 Jan 17 21:59 Data.o
-rw-rw-r-- 1 ec2-user ec2-user 18256 Jan 17 21:59 Time.o
-rw-rw-r-- 1 ec2-user ec2-user 27200 Jan 17 21:57 Dispatch.o
i tried doing this and setting a breakpoint on fatalError, which did not reveal anything in swiftc.
(lldb) b fatalError
Breakpoint 1: no locations (pending).
WARNING: Unable to resolve breakpoint to any actual locations.
(lldb) r
Process 96 launched: '/swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swiftc' (x86_64)
1 location added to breakpoint 1
Swift version 5.11-dev (LLVM 497a64a78b2fba6, Swift 0bbcae4e9278768)
Target: x86_64-unknown-linux-gnu
Process 96 exited with status = -1 (0xffffffff) lost connection
(lldb) breakpoint list
Current breakpoints:
1: name = 'fatalError', locations = 1, resolved = 1, hit count = 0
1.1: where = libswiftCore.so`swift::fatalError(unsigned int, char const*, ...), address = 0x00007ffff76bb390, resolved, hit count = 0
(lldb)
either i did not set the breakpoint correctly, or swiftc is merely parroting a fatalError printed by swift-driver.
That's good, that means swift-frontend is unlikely to be the issue as we originally suspected.
Likely the latter, I have not previously looked into exactly which process launches the swift-frontend process, swiftc or swift-driver, but it probably is the latter. Given the early swift-driver flakiness you already reported, it is probably the point of failure. Unfortunately, that is currently tough to debug, as the verbose output does not list the swift-driver invocation it is using.
Try this: run the exact swiftc command spit out by CMake but add the flags -v -disallow-use-new-driver. That invokes the legacy Driver written in C++, so if that works, you know where the problem is.
You can then avoid the early swift-driver altogether by passing --skip-early-swift-driver to build-script, after deleting the one you already built, rm /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift-driver.
ha, i can confirm this fixes the issue; manually running the swiftc command with that option succeeds.
unfortunately, i can’t seem to disable the swift driver when building the toolchain using build-script, even when i deleted the one i compiled and adding --skip-early-swift-driver. the exact steps i did were:
despite the --skip-early-swift-driver option, the build fails with the same fatalError as before, and moreover, the swift-driver binary reappears in swift-linux-x86_64/bin/.
You may need a --reconfigure passed to build-script too. If that doesn't work, try deleting the compiler frontend build, rm -rf /swift/swift-project/build/Ninja-ReleaseAssert/swift-linux-x86_64/.
restarting the build from scratch worked for me thanks!
i guess the next step for me is to actually get an installable toolchain packaged. i tried naïvely adding --skip-early-swift-driver to --preset buildbot_linux, but that doesn’t seem to parse. is there a way to override this setting while still getting something similar to the downloadable toolchains on swift.org?
most of the tests had to be disabled because they don’t particularly like that swift-driver is missing, so i guess only time will tell if this toolchain will actually be usable in production.
as a smoke test, i tried installing this toolchain and compiling the swift-nio project, which worked fine.
I don't think that's the reason, as the compiler simply uses the legacy Driver written in C++ instead. What failed for those other three repos whose tests you disabled, other than the single Reflection test in the compiler validation suite before? And that's not "most of the tests," it looks like the corelibs and all other tests must have passed for you.
thanks!