I like the idea of using Unmanaged. It was the other API I looked at when gaming this out. But I do think I still need to use a weak reference (somewhere).
The real world API I'm writing against is AXObserver which lacks a finished event callback so you can safely do tear down and is active until the element you observe becomes invalid (which could be never).
Prior to ARC interaction with this API usually meant building your own custom reference counting and implementing weak references. Then built in self zeroing weak refs allowed us to switch to what the run time provided. It seems like I can continue using the ObjC runtime feature (via manual allocation of an UnsafePointer or via Unmanaged) but I don't think I can avoid a race in a concurrent environment between adding/removing and deinit for the whatever we passed as context.
You can see more about the specific use case and possibly point out holes in my thinking in another post where I asked an overlapping question Using SerialExecutor to make an actor that is tied to run loop