I've been wondering if there's any appetite for a Swift dynamic loading API. Basically a Swiftier version of dlopen and dlsym, maybe making use of reflection metadata to support more kinds of queries on loaded modules. There is precedent for it: Java has its own ClassLoader API and Go also supports plugins.
This could allow for richer interfaces than are currently practical using C interop and dlsym. Here's a rough sketch of what I think a Swift-native plugin should look like to use:
MyPluginAPI module:
/// An interface which MyApp uses to communicate with its plugins.
/// Note the use of generics and Swift stdlib types like String.
public protocol MyPluginAPI {
init()
func doYourThing<C>(_ input: String, data: C) where C: Collection
}
MyApp:
import Plugins // core library, like "Differentiation".
import MyPluginAPI
// Load the module, query exported conformers using Swift metadata.
let module = try Plugin.load("...")
let exportedPlugins: [MyPluginAPI.Type] = module.allConformers(to: MyPluginAPI.self)
// Communicate with those types via the MyPluginAPI protocol.
for pluginType in exportedPlugins {
let pluginInstance = pluginType.init()
pluginInstance.doYourThing("hello", data: 0..<10)
}
Plugin module:
import MyPluginAPI
public struct DoAThing: MyPluginAPI {
public init() {
// ...
}
public func doYourThing<C>(_ input: String, data: C) where C: Collection {
// ...
}
}
Absolutely. Dynamic loading of functionality (plugins), in conjunction with versioned (plugin) dependency management, orchestrated by an overarching service lifecycle would be the basis for lightweight plugin runtime systems in the veins of OSGi and Eclipse (both hugely successful systems).
I've wanted something like this for swift-format, which would allow users to build their own rules as plug-ins loaded at runtime. That would let folks easily create custom project- or company-specific behaviors that don't require them to be baked into the core code base or maintain a fork.
The original proposal does not imply that. @Karl proposed a hypothetical plugin interface like this:
/// An interface which MyApp uses to communicate with its plugins.
/// Note the use of generics and Swift stdlib types like String.
public protocol MyPluginAPI {
init()
func doYourThing<C>(_ input: String, data: C) where C: Collection
}
This interface exposes Swift Strings and generics. That requires an ABI that includes refcounting, ownership, and generics, none of which exist in the C ABI.
Yeah it’d be nice to move beyond using C and C-compatible types as the interface for plugins, and to expand queries beyond just “give me the address of this exact symbol”.
As for ABI: I don’t think we technically need it - after all, today you can just write a C shim which exposes the location of Swift code. See this example I found on the web; the createPlugin function returns a Swift object as a raw pointer, which is turned back in to a Swift object pointer via Unmanaged.fromOpaque. This kind of workaround, which many developers would be drawn to, also fails if the modules have incompatible ABIs. Ultimately that’s a distribution/platform consideration, I think.
IMO, the only thing we actually need to guarantee is that a successfully loaded plugin has a compatible ABI with the module that loaded it. On ABI-stable platforms, that’s a trivial “yes” (well... uhm, depending on deployment targets), and on non-ABI-stable platforms it may require some kind of version metadata in the modules.
Depending on how you deploy your plugins and which platforms you support, you may need to build them with same compiler version for compatible ABIs. I don’t know if a special build mode (e.g. Go has a “plugin” build mode) would help us loosen those restrictions, but it would be worth exploring.
I suppose? This isn’t intended to be used to reflect the current, global state of the process, but I don’t see why it shouldn’t be capable of inspecting the main binary as well as any plugin library.
I’m not hugely familiar with the runtime and Swift metadata though. I’m kind of assuming that it’s possible to iterate public protocol conformances for types which statically exist in the binary (e.g. infinitely-recursive generic types which are created at runtime wouldn’t be listed, neither would unused conditional conformances).
Will need to think through how to make this not just be a vector for malware, though that's probably at a level above in the app that's adding this plug-in capability (only load digitally signed plugins or whatever).
Just raising the flag of this seeming like a potential vector to execute arbitrary code into an app.
To expand a bit on the discussion. This is very usable for swift on the server. ABI could be an issue, but for certain kind of deployments requiring the end users (it staff) to recompile their plugins with a specific tool chain is acceptable. It could even be a hard requirement until ABI stability is in place. For such scenarios arbitrary code injection is actually the whole point We want to accomplish. Just wanted to point out there are such use cases (have deployed plug-in systems on various platforms with the above behavior - it was absolutely fine to require a specific tool chain). Of course a stable ABI and to not require specific toolchains is even nicer, but just wouldn’t want to dismiss this idea just based on that.
We usually would allow customers also to reload a plug-in (had interfaces for storing/restoring internal state) during runtime to change the behavior of the system on the fly.
I would use this to allow users to add new content to my games (modding). An example would be implementing support for an obscure game controller, or replacing the audio backend with a different backend. Or adding new levels and other in game content.
If plugins were modeled as a spm product they could potentially contain their own resources too.
A deployed executable could be expanded with functionality created by a different author. It would open Swift up to a whole new world of use cases.
If this by some miracle allowed safe unloading as well, I’d use it to reload recompiled parts of my games while they are running so I could see changes immediately. Which would be sooooo useful!
We would use this to allow customers to add their own custom code at runtime - possibly reloaded multiple times. Have been doing that for a very long time with objective-c, c and c++ very successfully. Core feature really.
