i’ve noticed that swift’s memory usage will grow monotonically unless an OS-level memory limit is placed on it, such as MemoryHigh or MemoryMax.
so i added these lines to my daemon configuration:
MemoryHigh=250M
MemoryMax=249M
Restart=always
i put the MemoryMax slightly below the MemoryHigh because the latter tells the kernel to throttle the process instead of killing it, and that is functionally the same as taking the server offline without any possibility of restarting it.
this seems to work (through mechanisms i don’t really understand) to get swift to reuse memory it has already been allocated instead of continuously requesting new pages from the OS.
however, when i tried submitting a lot of sequential, memory-intensive work to the server, i found that it occasionally still stops responding once it approaches either limit. being about 1.1MB away from the limit seems to be enough to render the application completely unresponsive.
from what i have observed, this is a consequence of the daemon-level memory limit, and not system-level memory exhaustion, as no swapping was taking place and the OS was still perfectly responsive. (without the memory limit, swift for some reason will always claim all available memory on the system after enough time, which affects all other processes including SSH, making it impossible to even interrupt the process without performing a full system reboot. yikes!)
how does the swift runtime actually respond to an OS-level memory limit, and how do i get it to stop freezing when it approaches the limit?
The Swift runtime passes through to malloc and free[1], so you're really just analyzing the memory use patterns of your system allocator here. Different malloc implementations might re-use memory in ways you prefer.
It doesn't promise to, but that's what it currently actually does. ↩︎
Anecdotally I've noticed malloc libraries on Linux tend to be greedy like this, compared to Apple platforms.
If you can identify which malloc library is being used, you might be able to tune it. e.g. from memory tcmalloc is pretty eager to hang onto unused pages, by default, as it prioritises speed with an assumed surplus of RAM, but you can tune it pretty effectively to reduce its wastage.
thanks all, i discovered that issues with malloc implementations can be solved by switching to an allocator with a decay, like jemalloc. after switching to jemalloc, i am no longer seeing problems with persistent memory use!
The standard glibc allocators behavior is often not great and we have heard numerous reports in the server ecosystem where people switched to an alternative allocator and saw a way more stable memory usage. However, it is hard to universally state that either tcmalloc or jemalloc are the best allocator since it always depends on the allocation patterns. What I would like to see in Swift instead is a possibility to hook the global allocator similar to what Rust’s GlobalAllocator allows.
This would allow us to vend packages for the various allocators and people could just switch by depending on a package instead bundling and preloading the allocator
It actually calls the C++ runtime’s placement new, which leads to Fun Times when someone else in your process has replaced the global ::new operator to swizzle in their own garbage collector or diagnostics.
I would love if swift_allocObject were resilient to this, as well as the ability to use my own allocator for a specific Swift object.
EDIT: Sorry, it calls malloc first via swift_slowAlloc and then placement-allocates the Swift header into that spot using placement new. So there are actually two opportunities for conflict in the global namespace.
In C++, the standard placement operator new(void*) does not do any allocation. It is also a special case in that it cannot be legally replaced by the user, and Clang recognizes it and does not perform an actual library call, even at -O0.
Yep, I noticed that and fixed my mistake above. But I have debugged cases where third party binaries have replaced operator new in ways that are incompatible with the Swift runtime; perhaps they were not in the HeapObject allocation path. My point was really to warn “there be dragons here”.
I’ll note that test is disabled and probably has been for a long time, given this line:
// REQUIRES: rdar92102119
I don’t think stomping on dyld symbols works anymore. Of course we’re talking about Linux anyway; doing the equivalent by LD_PRELOADing your own swift_alloc might work, or it might not.