Swift dynamic loading API

Yes, please!

I wrote one of these a while ago for LLVMSwift, but it has since gone unmaintained.

Though this doesn't cover the use case of scraping metadata to find protocol conformers.

5 Likes

Let's make it happen. This is definitely going to be necessary if Swift is going to take over the world :earth_americas::sunglasses:

3 Likes

This would presumably function correctly only on platforms that have a stable ABI, correct?

2 Likes

Yeah this would be great! I'd also like to see dynamic replacement make it into the official API.

If I understand correctly it would use the C FFI so ABI stability would not be required

The original proposal does not imply that. @Karl proposed a hypothetical plugin interface like this:

/// An interface which MyApp uses to communicate with its plugins.
/// Note the use of generics and Swift stdlib types like String.
public protocol MyPluginAPI {
 init()
 func doYourThing<C>(_ input: String, data: C) where C: Collection
}

This interface exposes Swift Strings and generics. That requires an ABI that includes refcounting, ownership, and generics, none of which exist in the C ABI.

3 Likes

Would it be possible to use Plugins API to discover conforming types in the main binary?

Yeah it’d be nice to move beyond using C and C-compatible types as the interface for plugins, and to expand queries beyond just “give me the address of this exact symbol”.

As for ABI: I don’t think we technically need it - after all, today you can just write a C shim which exposes the location of Swift code. See this example I found on the web; the createPlugin function returns a Swift object as a raw pointer, which is turned back in to a Swift object pointer via Unmanaged.fromOpaque. This kind of workaround, which many developers would be drawn to, also fails if the modules have incompatible ABIs. Ultimately that’s a distribution/platform consideration, I think.

IMO, the only thing we actually need to guarantee is that a successfully loaded plugin has a compatible ABI with the module that loaded it. On ABI-stable platforms, that’s a trivial “yes” (well... uhm, depending on deployment targets), and on non-ABI-stable platforms it may require some kind of version metadata in the modules.

Depending on how you deploy your plugins and which platforms you support, you may need to build them with same compiler version for compatible ABIs. I don’t know if a special build mode (e.g. Go has a “plugin” build mode) would help us loosen those restrictions, but it would be worth exploring.

3 Likes

I suppose? This isn’t intended to be used to reflect the current, global state of the process, but I don’t see why it shouldn’t be capable of inspecting the main binary as well as any plugin library.

I’m not hugely familiar with the runtime and Swift metadata though. I’m kind of assuming that it’s possible to iterate public protocol conformances for types which statically exist in the binary (e.g. infinitely-recursive generic types which are created at runtime wouldn’t be listed, neither would unused conditional conformances).

Will need to think through how to make this not just be a vector for malware, though that's probably at a level above in the app that's adding this plug-in capability (only load digitally signed plugins or whatever).
Just raising the flag of this seeming like a potential vector to execute arbitrary code into an app.

+1 for this, many systems can greatly benefit with extensions through plugins. Would be nice with richer options than dlopen/dlsym.

To expand a bit on the discussion. This is very usable for swift on the server. ABI could be an issue, but for certain kind of deployments requiring the end users (it staff) to recompile their plugins with a specific tool chain is acceptable. It could even be a hard requirement until ABI stability is in place. For such scenarios arbitrary code injection is actually the whole point We want to accomplish. Just wanted to point out there are such use cases (have deployed plug-in systems on various platforms with the above behavior - it was absolutely fine to require a specific tool chain). Of course a stable ABI and to not require specific toolchains is even nicer, but just wouldn’t want to dismiss this idea just based on that.

We usually would allow customers also to reload a plug-in (had interfaces for storing/restoring internal state) during runtime to change the behavior of the system on the fly.

What sort of use cases would a plugin API support, that are not already supported?

I would use this to allow users to add new content to my games (modding). An example would be implementing support for an obscure game controller, or replacing the audio backend with a different backend. Or adding new levels and other in game content.

If plugins were modeled as a spm product they could potentially contain their own resources too.

A deployed executable could be expanded with functionality created by a different author. It would open Swift up to a whole new world of use cases.

If this by some miracle allowed safe unloading as well, I’d use it to reload recompiled parts of my games while they are running so I could see changes immediately. Which would be sooooo useful!

5 Likes

We would use this to allow customers to add their own custom code at runtime - possibly reloaded multiple times. Have been doing that for a very long time with objective-c, c and c++ very successfully. Core feature really.

2 Likes

I see. I certainly appreciate your elaboration. This sounds like quite the killer feature.

Some inspiration for dynamically finding plug-in types:

1 Like

For anyone interested in seeing an existing implementation of a Swift plugin system, my project Delta Client has one. Here’s the plugin api and here’s an example plugin. As you would see if you read some of that code, a nicer API for dynamic loading would definitely help! The plugin side of things isn’t too bad, plugin loading is the tricky part.

Here’s an overview of the approach I took:

  • Plugins declare a cdecl function called buildPlugin that returns a pointer to a PluginBuilder instance
  • The plugin builder holds the metatype of the plugin in a property and has a method that just calls the initialiser of the metatype to create a new instance of the plugin
  • The plugin has to be transported that way because generics don’t really work across the boundary between the plugin’s dylib and the main app. I tried a few other methods and none of them worked but I can’t remember exactly what the methods were or why they failed.
  • Plugins just conform to a basic protocol that acts as the interface between the app and the plugin

Even just getting all of the dynamic linking working correctly with SwiftPM (the entire project including the app is SwiftPM) was a pain, I ended up having to make sub packages and reexport them from the root package. Maybe making that easier could be another topic of discussion at some point.

8 Likes

For further reading I suggest this article which is about making a plugin system in Swift. It is where I got my initial implementation from. I just cleaned it up and tweaked it bit. It’s the cleanest method I’ve found.

3 Likes
Terms of Service

Privacy Policy

Cookie Policy