Does Vapor close resources opened with dlopen automatically?

sbeitzel · March 15, 2022, 3:38pm

I'm working on a Vapor application which, during startup, looks for plugins and opens them:

// in code called from configure.swift
//...
if let openRes = dlopen(path.string, RTLD_NOW | RTLD_LOCAL) {
// ... see if the thing really is a plugin, and if so, cast it to something useful
// ... also, store openRes in a list for later
}

Then, I install a shutdown hook:

app.storage.set(GameControllerKey.self, to: gameController, onShutdown: { $0.shutdown(app: app) })

...and the shutdown hook stops worker threads that the controller may have spun up, and it's where I use the list of opened resources:

            for res in openResources {
                dlclose(res)
            }
            openResources.removeAll()

Now, this seems to work at the time -- log messages before and after print out, and no errors get thrown. However, later on during shutdown this message gets emitted:

[ DEBUG ] Requesting HTTP server shutdown (Vapor/HTTP/Server/HTTPServer.swift:287)
Segmentation fault

If I comment out the code that closes the resources, then there's no segmentation fault. This leads me to suspect that there's some magic happening deep inside the Vapor framework, but I just don't know, and it feels...dirty...to go around opening files and not close them.

0xTim · March 15, 2022, 5:51pm

Are you able to get any information out of the set fault? Vapor has now knowledge about plugins so you'd need to manage the lifecycle of this yourself. Incidentally Vapor has a LifecycleHandler that may be better for this than trying to make it work with the app storage

sbeitzel · March 15, 2022, 8:35pm

Unfortunately, the segfault doesn't produce a core file anywhere that I can find. (I say, "unfortunately," but it would be of limited use since I have never really figured out how to read a core dump.)

I've also not yet invested the time and effort to figure out how to attach the VSCode debugger to a process, so I'm still just launching this from a terminal window and watching log messages.

Of possible interest, I refactored the code to use LifecycleHandler. It still causes the segmentation fault on shutdown.

All I can guess is that something in the apple/swift-system package is somehow wrapping dlopen() on Linux, which is well outside the bounds of what one might expect Vapor to deal with.

lukasa · March 16, 2022, 9:57am

Getting a segmentation fault is really bad: someone somewhere has done something awful. Can you attach swift-backtrace per the instructions in the README and see if we can get a better backtrace?

sbeitzel · March 16, 2022, 4:43pm

Well, it's still a mystery.

I installed swift-backtrace and added print() statements to main.swift, so I could see if the segfault happens before or after the app.shutdown() call.

import App
import Backtrace
import Vapor

print("Installing backtrace")
Backtrace.install()
var env = try Environment.detect()
try LoggingSystem.bootstrap(from: &env)
let app = Application(env)
defer {
    app.shutdown()
    print("Vapor shutdown complete")
}
try configure(app)
try app.run()

I ran the server, with stderr redirected to stdout to make sure I capture anything backtrace might emit.

$ ./Run serve --env development --hostname 0.0.0.0 --port 8080 --log debug 2>&1
Installing backtrace
[ DEBUG ] Looking for game plugins in path: /app/Resources/Games (App/Controllers/GameController.swift:265)
[ INFO ] Found game plugin at: /app/Resources/Games/libTicTacToe.so (App/Controllers/GameController.swift:271)
[ INFO ] Loaded game: Tic-tac-toe (naughts and crosses) (App/Controllers/GameController.swift:280)
[ NOTICE ] Server starting on http://0.0.0.0:8080 (Vapor/HTTP/Server/HTTPServer.swift:270)
^C
[ DEBUG ] Application shutting down (Vapor/Application.swift:135)
[ INFO ] Waiting for GameManager to shut down (App/Controllers/GameManager.swift:19)
[ INFO ] Shutting down GameManager (App/Controllers/GameManager.swift:41)
[ INFO ] GameManager shutdown completed (App/Controllers/GameManager.swift:21)
[ INFO ] Closing game plugins... (App/Controllers/GameController.swift:308)
[ INFO ] Closed 1 plugins (App/Controllers/GameController.swift:315)
[ INFO ] GamePlugins shutdown complete (App/Controllers/GameController.swift:316)
Segmentation fault

Interestingly, if I build and run on macOS rather than in a Linux container, there's no problem:

% ./Run serve --env mac --hostname 0.0.0.0 --port 8080 --log debug 2>&1                                    ✹ ✭
Installing backtrace
[ DEBUG ] Looking for game plugins in path: /Users/sbeitzel/src/github/tbgs-swift/Resources/Games (App/Controllers/GameController.swift:265)
[ INFO ] Found game plugin at: /Users/sbeitzel/src/github/tbgs-swift/Resources/Games/libTicTacToe.dylib (App/Controllers/GameController.swift:271)
[ INFO ] Loaded game: Tic-tac-toe (naughts and crosses) (App/Controllers/GameController.swift:280)
[ NOTICE ] Server starting on http://0.0.0.0:8080 (Vapor/HTTP/Server/HTTPServer.swift:270)
^C
[ DEBUG ] Application shutting down (Vapor/Application.swift:135)
[ INFO ] Waiting for GameManager to shut down (App/Controllers/GameManager.swift:19)
[ INFO ] Shutting down GameManager (App/Controllers/GameManager.swift:41)
[ INFO ] GameManager shutdown completed (App/Controllers/GameManager.swift:21)
[ INFO ] Closing game plugins... (App/Controllers/GameController.swift:308)
[ INFO ] Closed 1 plugins (App/Controllers/GameController.swift:315)
[ INFO ] GamePlugins shutdown complete (App/Controllers/GameController.swift:316)
[ DEBUG ] Requesting HTTP server shutdown (Vapor/HTTP/Server/HTTPServer.swift:287)
[ DEBUG ] HTTP server shutting down (Vapor/HTTP/Server/HTTPServer.swift:293)
Vapor shutdown complete

This suggests to me that calling dlclose() on the opened resources is a Good Idea on macOS but not on Linux. Ugh. Now I suppose I have to go find out how these calls differ and do a conditional compilation.

hassila · March 16, 2022, 5:24pm

In swift-plugin-manager I specify RTLD_NODELETE to ensure that anyone keeping references to types dynamically loaded will be valid even if the plugin is unloaded/reloaded (as I don't have control of usage patterns).

For many use cases where a server loads plugins and really just unloads then during shutdown to let them release any allocated resources, it's perhaps pragmatic.

I guess the main question if whether you reference anything from the plugin after its unloaded - that would be my first guess.

I don't remember any real differences between the load/unload between macOS and Linux, that being said you can check also by putting in a breakpoint / sleep after the unload with (pmap / vmmap / xxx) whether the plugin is truly unmapped.

sbeitzel · March 16, 2022, 9:52pm

Brilliant! ORing in RTLD_NODELETE during dlopen does stop the crash from happening on Linux. I hope that I am not too optimistic in supposing that, if the Vapor process shuts down cleanly and exits, then any memory allocated by that process will be freed up.

lukasa · March 17, 2022, 7:45am

All processes have their memory (and indeed all other resources) automatically reclaimed by the OS when they exit, so this should be fine.