I'm trying to build my SPM package. It builds just fine on my Mac, but on an Ubunutu 22.04 machine, most of it builds, but this step fails with “Killed.” When run as part of swift build, it suggests adding -v to see the invocation (linked above). I wasn't able to add -v to that invocation to get more information.
$ /usr/bin/time -f "%E %M" ./test.sh
Command exited with non-zero status 137
where test.sh is just that large swift-frontend invocation. If I’m interpreting that correctly, that's 684 MB peak memory used. Is it possible something is killing the process due to using too much RAM?
If enabling swap fixed it, then it sounds like the OOM killer was responsible for killing the compiler. I don’t think there’s much an individual process can do to improve the ability to diagnose a kernel routine that vaporizes processes when VM usage gets too high.
The related sysctl files are /proc/sys/kernel/core_pattern, etc. You can modify them directly or using sysctl. See this article for example. /proc/sys/kernel/core_pattern supports redirecting core dump to application. Ubuntu uses that feature to let apport to handle core dumps. According to apport man page, it saves core files under /var/crash. Do you see files in that directory? In your cases, the core file might be of bash process, I think.
BTW, getting core files is usually the first step in most cases. But you are lucky in this case because the fact that bash core dumped indicated something on its own.
If it's OOM issue, I think you should be able to find related messages in /var/log/kern.log (or just run dmesg).
You said you saw apport report. Does that mean you use GUI in droplet? If so, I'd suggest to install a minimal Ubuntu server installation. Another option is to install VMware Fusion on your Macbook. It works quite well (I do it all the time. I just use macOS for app development).
Maybe start a background process to collect system metrics (especially memory usage)? That may help to investigate the root cause of mysterious build failures.
I only see files in /var/crash for the sleep task I killed on purpose to test my crash reporting. I don't get it when swift is killed. I do not use a GUI droplet, only a minimal server.
It's definitely a memory exhaustion issue. I fixed it by enabling swap, but that wasn't enough to let the docker build finish. I just ended up moving the docker build to Github actions. So far, so good.
does your package have very long source files? iirc, @lorentey recently uncovered a dramatic increase in compiler memory usage as individual source files get longer. breaking up the files into smaller files resolves the issue, apparently.
My files are not especially long. I haven't checked Vapor and its dependencies. I don't think the file that was usually crashing was particularly long, but it wasn't always the same one and now I'm not sure which one it was.
they are an order of magnitude shorter than the file (40k lines) that was causing swapping when building swift-atomics on swift CI. but the swift CI also probably has much more than 1 GB memory available.
I've been frequently running into OOM issues when compiling packages on VMs or in docker containers with low RAM and without swap (and "Killed" is exactly the resulting error). I don't recall the exact numbers but 1 GB is tight for packages with a fair number of dependencies, in my experience.