Swift dynamic loading API

Hi,

I've been wondering if there's any appetite for a Swift dynamic loading API. Basically a Swiftier version of dlopen and dlsym, maybe making use of reflection metadata to support more kinds of queries on loaded modules. There is precedent for it: Java has its own ClassLoader API and Go also supports plugins.

This could allow for richer interfaces than are currently practical using C interop and dlsym. Here's a rough sketch of what I think a Swift-native plugin should look like to use:

MyPluginAPI module:

/// An interface which MyApp uses to communicate with its plugins.
/// Note the use of generics and Swift stdlib types like String.
public protocol MyPluginAPI {
  init()
  func doYourThing<C>(_ input: String, data: C) where C: Collection
}

MyApp:

import Plugins // core library, like "Differentiation".
import MyPluginAPI

// Load the module, query exported conformers using Swift metadata.
let module = try Plugin.load("...")
let exportedPlugins: [MyPluginAPI.Type] = module.allConformers(to: MyPluginAPI.self) 
// Communicate with those types via the MyPluginAPI protocol.
for pluginType in exportedPlugins {
  let pluginInstance = pluginType.init()
  pluginInstance.doYourThing("hello", data: 0..<10)
}

Plugin module:

import MyPluginAPI

public struct DoAThing: MyPluginAPI {

  public init() {
    // ...
  }

  public func doYourThing<C>(_ input: String, data: C) where C: Collection {
    // ...
  }
}

Is there any interest in something like this?

23 Likes

Absolutely. Dynamic loading of functionality (plugins), in conjunction with versioned (plugin) dependency management, orchestrated by an overarching service lifecycle would be the basis for lightweight plugin runtime systems in the veins of OSGi and Eclipse (both hugely successful systems).

4 Likes

I've wanted something like this for swift-format, which would allow users to build their own rules as plug-ins loaded at runtime. That would let folks easily create custom project- or company-specific behaviors that don't require them to be baked into the core code base or maintain a fork.

9 Likes

Yes, please!

I wrote one of these a while ago for LLVMSwift, but it has since gone unmaintained.

Though this doesn't cover the use case of scraping metadata to find protocol conformers.

5 Likes

Let's make it happen. This is definitely going to be necessary if Swift is going to take over the world :earth_americas::sunglasses:

1 Like

This would presumably function correctly only on platforms that have a stable ABI, correct?

2 Likes

Yeah this would be great! I'd also like to see dynamic replacement make it into the official API.

If I understand correctly it would use the C FFI so ABI stability would not be required

The original proposal does not imply that. @Karl proposed a hypothetical plugin interface like this:

/// An interface which MyApp uses to communicate with its plugins.
/// Note the use of generics and Swift stdlib types like String.
public protocol MyPluginAPI {
 init()
 func doYourThing<C>(_ input: String, data: C) where C: Collection
}

This interface exposes Swift Strings and generics. That requires an ABI that includes refcounting, ownership, and generics, none of which exist in the C ABI.

2 Likes

Would it be possible to use Plugins API to discover conforming types in the main binary?

Yeah it’d be nice to move beyond using C and C-compatible types as the interface for plugins, and to expand queries beyond just “give me the address of this exact symbol”.

As for ABI: I don’t think we technically need it - after all, today you can just write a C shim which exposes the location of Swift code. See this example I found on the web; the createPlugin function returns a Swift object as a raw pointer, which is turned back in to a Swift object pointer via Unmanaged.fromOpaque. This kind of workaround, which many developers would be drawn to, also fails if the modules have incompatible ABIs. Ultimately that’s a distribution/platform consideration, I think.

IMO, the only thing we actually need to guarantee is that a successfully loaded plugin has a compatible ABI with the module that loaded it. On ABI-stable platforms, that’s a trivial “yes” (well... uhm, depending on deployment targets), and on non-ABI-stable platforms it may require some kind of version metadata in the modules.

Depending on how you deploy your plugins and which platforms you support, you may need to build them with same compiler version for compatible ABIs. I don’t know if a special build mode (e.g. Go has a “plugin” build mode) would help us loosen those restrictions, but it would be worth exploring.

2 Likes

I suppose? This isn’t intended to be used to reflect the current, global state of the process, but I don’t see why it shouldn’t be capable of inspecting the main binary as well as any plugin library.

I’m not hugely familiar with the runtime and Swift metadata though. I’m kind of assuming that it’s possible to iterate public protocol conformances for types which statically exist in the binary (e.g. infinitely-recursive generic types which are created at runtime wouldn’t be listed, neither would unused conditional conformances).

Will need to think through how to make this not just be a vector for malware, though that's probably at a level above in the app that's adding this plug-in capability (only load digitally signed plugins or whatever).
Just raising the flag of this seeming like a potential vector to execute arbitrary code into an app.

Terms of Service

Privacy Policy

Cookie Policy