I'm trying to build my SPM package. It builds just fine on my Mac, but on an Ubunutu 22.04 machine, most of it builds, but this step fails with “Killed.” When run as part of swift build, it suggests adding -v to see the invocation (linked above). I wasn't able to add -v to that invocation to get more information.
Any idea how to figure out what’s failing?
I never build SPM on Linux, but I suspect a process coredumped. Can you find a core and check its stack trace? You may need to configure core dump policy (I forgot the details on Linux).
Hmm. I've done a bit of googling, and I can get "apport" crash reports when killing a test process (sleep) with -SIGSEGV, but I don't seem to get a core dump on the swift process.
I tried running this:
$ /usr/bin/time -f "%E %M" ./test.sh
Killed
Command exited with non-zero status 137
0:08.91 684572
where test.sh is just that large swift-frontend invocation. If I’m interpreting that correctly, that's 684 MB peak memory used. Is it possible something is killing the process due to using too much RAM?
On a hunch, I enabled swap (it was not enabled by default on this 1G droplet). That allowed it to compile. I wonder if it's reasonable to write a bug asking for better diagnostics in this case.
If enabling swap fixed it, then it sounds like the OOM killer was responsible for killing the compiler. I don’t think there’s much an individual process can do to improve the ability to diagnose a kernel routine that vaporizes processes when VM usage gets too high.
The related sysctl files are /proc/sys/kernel/core_pattern, etc. You can modify them directly or using sysctl. See this article for example. /proc/sys/kernel/core_pattern supports redirecting core dump to application. Ubuntu uses that feature to let apport to handle core dumps. According to apport man page, it saves core files under /var/crash. Do you see files in that directory? In your cases, the core file might be of bash process, I think.
BTW, getting core files is usually the first step in most cases. But you are lucky in this case because the fact that bash core dumped indicated something on its own.
If it's OOM issue, I think you should be able to find related messages in /var/log/kern.log (or just run dmesg).
You said you saw apport report. Does that mean you use GUI in droplet? If so, I'd suggest to install a minimal Ubuntu server installation. Another option is to install VMware Fusion on your Macbook. It works quite well (I do it all the time. I just use macOS for app development).
Maybe start a background process to collect system metrics (especially memory usage)? That may help to investigate the root cause of mysterious build failures.
I only see files in /var/crash for the sleep task I killed on purpose to test my crash reporting. I don't get it when swift is killed. I do not use a GUI droplet, only a minimal server.
It's definitely a memory exhaustion issue. I fixed it by enabling swap, but that wasn't enough to let the docker build finish. I just ended up moving the docker build to Github actions. So far, so good.
does your package have very long source files? iirc, @lorentey recently uncovered a dramatic increase in compiler memory usage as individual source files get longer. breaking up the files into smaller files resolves the issue, apparently.
My files are not especially long. I haven't checked Vapor and its dependencies. I don't think the file that was usually crashing was particularly long, but it wasn't always the same one and now I'm not sure which one it was.
i checked vapor (wc -l $(find vapor/ -name "*.swift")), but none of its sources are longer than 1K lines. perhaps it is a large file in a dependency, or an unrelated issue.
i looked through its dependencies, and i found some abnormally long files in async-http-client and swift-nio-http2:
3516 async-http-client/Tests/AsyncHTTPClientTests/HTTPClientTests.swift
3205 swift-nio-http2/Tests/NIOHTTP2Tests/ConnectionStateMachineTests.swift
they are an order of magnitude shorter than the file (40k lines) that was causing swapping when building swift-atomics on swift CI. but the swift CI also probably has much more than 1 GB memory available.
I've been frequently running into OOM issues when compiling packages on VMs or in docker containers with low RAM and without swap (and "Killed" is exactly the resulting error). I don't recall the exact numbers but 1 GB is tight for packages with a fair number of dependencies, in my experience.
I wonder what's the recommended memory size for running Swift on Linux?