Swift shared libraries as plugins

Hello, is there an accepted way of loading one Swift module from another at runtime and using the loaded module’s symbols? This will be on Linux, if it makes a difference.

I can think of using dlopen and friends in the caller and @_silgen_name in the callee but maybe there’s a more sanctioned way about it?

Sanctioned might be overstating it but you could try something more structured like SwiftPlugin/main.swift at master · johnno1962/SwiftPlugin · GitHub

import Foundation

func LoadPlugin<T>(onto: T.Type, dylib: String) -> T.Type {
    guard let handle = dlopen(dylib, RTLD_NOW) else {
        fatalError("Could not open \(dylib) \(String(cString: dlerror()))")
    }

    var info = Dl_info()
    dladdr(unsafeBitCast(onto, to: UnsafeRawPointer?.self), &info)
    guard let replacement = dlsym(handle, info.dli_sname) else {
        fatalError("Could not locate class \(String(cString: info.dli_sname))")
    }

    return unsafeBitCast(replacement, to: T.Type.self)
}

print("Hello, World!")

// Class MyPlugin from bundle is overlayed onto this class

open class MyPlugin {

    open class func getInstance() -> MyPlugin {
        fatalError("getInstanceMain")
    }

    open func incCounter() -> Int {
        fatalError("incCounterMain")
    }
}

var info = Dl_info()
dladdr(&info, &info)

let pluginPath = URL(fileURLWithPath: String(cString: info.dli_sname)).deletingLastPathComponent()
    .appendingPathComponent("MyPlugin.bundle/Contents/MacOS/MyPlugin").path

let cl = LoadPlugin(onto: MyPlugin.self, dylib: pluginPath)

let i = cl.getInstance()
print(i.incCounter())
print(i.incCounter())
1 Like

If you don't want to involve Foundation, you could also mark your entry point in the plugin as @_cdecl to export it as a C function symbol. @_silgen_name should never be used.

1 Like

Interesting, so you’re not doing any name mangling. How? Is it because you’re using the same (identical) class definition in both? Would this work with a protocol?

Foundation would be fine! Forgive the noob question but how do I do it there? A method or technique name would suffice

If it is “by hand” I’d love to use the Swift calling convention if possible so I thought @_silgen_name would be more appropriate in that case?

Rather than tangling with mangling the code cheats slightly by having the class name and module name the same in the plugin project so the mangled names are the same. Perhaps you could unpick this assumption by doing a string replace on the module name before the class is looked up. Sorry, I don’t think protocols will work with this class overlay approach without knowing the mangled name of the target class. I may be wrong.

It isn't possible to turn the address of a Swift symbol into a Swift function value, since the convention of a Swift function type may differ from the symbol's own calling convention. On the other hand, C entry points and @convention(c) function pointers are always compatible. The best way to get a Swift-style interface from a dynamically loaded library would be by using the Swift or ObjC runtime's dynamic discovery mechanisms, which is what Foundation's NSBundle and similar APIs use, instead of using dlsym.

Ok sounds good, thanks for your response. Not sure if NSBundle is available on Linux yet but I’ll check it out when I get a chance

If you’re not keen on the "mangled class names must tie up” requirement of the above implementation I’ve updated the repo with an alternative:

func LoadPlugin<T>(onto: T.Type, dylib: String) -> T.Type {
    guard let handle = dlopen(dylib, RTLD_NOW) else {
        fatalError("Could not open \(dylib) \(String(cString: dlerror()))")
    }

    guard let replacement = dlsym(handle, "principalClass") else {
        fatalError("Could not locate principalClass function")
    }

    let principalClass = unsafeBitCast(replacement,
            to: (@convention (c) () -> UnsafeRawPointer).self)
    return unsafeBitCast(principalClass(), to: T.Type.self)
}

Usage is as before but you need something like this in the plugin:

@_cdecl("principalClass")
public func principalClass() -> UnsafeRawPointer {
    return unsafeBitCast(MyPlugin.self, to: UnsafeRawPointer.self)
}

Once you have a pointer to the class you can add functions as class methods in a type safe manner without restriction. There's not much difference between an abstract superclass and a protocol. This is not without application in the server domain where request servicing classes could be plugins, soft configured into the server. It also leaves the door open to “hot reloading” without taking the server down (provided you rename the dylib between versions)

If you're going to cast around pointers, I suggest going through Any.Type first. Not every type has a pointer-based representation.

Thanks Jordan, how do you mean? All this is premised on T being only class types. Is there a way I could express this in the generic parameter?

You can use AnyClass / AnyObject.Type in that case. (In theory a final class could also have an empty representation, though I don't know if we do that optimization right now.)

Well, if you're using @_cdecl and/or a protocol or class to manage the interface between your plugin and program, the abstraction level of type metadata values shouldn't be an issue, since the compiler will use a polymorphic calling convention for polymorphic interfaces automatically.

1 Like

Hello. I know this post is 3 years old — hopefully it's appropriate to resurrect it… with a twist.

I have a similar setup attempting to load shared libraries on Linux (.so files). I'm running swift on AWS Lambda and I would like to load and unload a different plugin (.so) on each request. I have working code:

  1. Worker (lambda runtime) GitHub - khinkson/Worker
  2. WorkerInterface (the protocol for the plugin as a .so) GitHub - khinkson/WorkerInterface
  3. WorkerPlugin (the actual plugin code as a .so) GitHub - khinkson/WorkerPlugin

I build each of these with a command similar to (no product flag for the plugin and interface):
docker run --rm -v "$PWD:/code" -w /code swift:5.6.1-amazonlinux2 swift build --product Worker -c release -Xswiftc -static-stdlib

I end up with 3 products: Worker, libWorkerInterface.so and libWorkerPlugin.so. The libWorkerInterface.so is bundled with the lambda runtime and loaded at launch. And the plugin is loaded from different folders on request.

It actually works… until I try to add any additional dependencies to the WorkerPlugin. In the code above eg: I have commented out the Crypto library. If I enable it and its code, the lambda will segfault occasionally with no output from Backtrace. This applies to any additional library I've tried to add to the plugin.

I do not know if I have a memory leak or if I need to do additional work to correctly load the plugin dependencies? If anyone has any ideas about where I'm going wrong here I would love to hear them. Thanks.

You might want to try RTLD_NODELETE And see if it makes a difference - it’s what I used in Swift dynamic loading API - #24 by Joakim_Hassila1 - let me know if it makes a difference.

Thanks. RTLD_NODELETE did not work for me. But I will take a deeper look at your dynamic loading API code and see if anything jumps out at me.

The weird thing is the segfault does not happen immediately. If I keep hitting the lambda it will run fine. If I stop for about 8 seconds before trying again, it segfaults. This even happens with the RTLD_NODELETE. That smells like a memory leak to me. And if not a leak, something is being unloaded/reset. Memory usage also jumps.

But I'll take a look at your code. Thanks again.