What happens to `static` variables when a shared swift library gets reloaded?

[taylorswift]

i have a swift library which is built as a shared library, and dynamically loaded and reloaded by a C API. i noticed that the library sometimes crashes when accessing non-trivial static variables. i tested this by storing a class in a static variable, and accessing it in the setup callback function which the C API invokes whenever it reloads the swift library.

the crash never happens if i configure the C API to only ever load the swift library once. so it seems like swift is losing track of the state of its static variables whenever the library gets reloaded. why is this happening?

[Joe_Groff]

The storage for static variables is mapped out of the dynamic library image when it's loaded, and if you're unloading and reloading the library multiple times, you are more than likely mapping it to a different address every time, so you're accessing dangling pointers into the unloaded copies. Beyond that, Swift libraries cannot generally be safely unloaded, because the runtime assumes that metadata addresses remain constant once instantiated.

[taylorswift]

@Joe_Groff

The storage for static variables is mapped out of the dynamic library image when it's loaded, and if you're unloading and reloading the library multiple times, you are more than likely mapping it to a different address every time, so you're accessing dangling pointers into the unloaded copies.

can you explain a bit more what this means exactly? does the library not get “refreshed” when it gets unloaded and reloaded? (if that makes any sense) or am i just completely misunderstanding how dynamic libraries work

[Varun_Gandhi]

[Not an expert but here's my understanding of what Joe is saying.]

1. Library gets loaded -> it gets assigned some pages of the virtual address space.
2. Library is used -> other pages now contain (direct) pointers into the pages for this library.
3. Library is unloaded -> you have dangling pointers.
4. Library is reloaded -> is gets assigned some pages of the virtual address space, which are likely different from the first time around.
5. You or the runtime dereferences a dangling pointer while trying to reuse the library.

Put another way, this is another flavor of iterator invalidation (or use after free) where you move the object and if you have an iterator pointing into the space where the old object was present, then dereferencing that iterator is not appropriate. There are a few possible solutions:

1. Don't move the object.
2. Don't use interior iterators, use offsets.
3. Don't maintain interior iterators across moves (or if you maintain them, don't use them which is basically the same thing).

Function calls for position-independent code use solution 2. -- which is maybe what you have in mind by the "refreshed" -- via the procedure linkage table (PLT), but that doesn't apply to arbitrary derived pointers, such as those created by the Swift runtime or your own code.

[taylorswift]

why does the runtime persist between reloads, but not the global variable storage?

[Joe_Groff]

Even if the Swift runtime were also unloaded and reloaded, that would not by itself do anything to clean up existing references to runtime state or global variables in the main executable's own data. Varun's analogy to free in C is a good one; all dlclose ultimately does is unmap the memory used by the library, and it's up to the programmer to make sure there are no dangling references into that memory afterward. Unfortunately, most nontrivial runtimes, including Swift, ObjC, and C++, are not really designed to allow for unloading code like this, and even in C it is very difficult to handle correctly.