When used in _classConforming(to:), AProtocol.self is inferred by the compiler to be a Protocol (an ObjC object), but when used in generic signature classConforming<R>(to: R), R is inferred to be AProtocol.Protocol (aka (any AProtocol).Type).
The error message is correct, because there's no relation (static or dynamic) between AProtocol.Type (aka any AProtocol.Type) and AProtocol.Protocol (aka (any AProtocol).Type). You cannot cast between them.
To achieve your goal, we must pass the existential metatype directly at caller side (which I believe is the only way to express you intention in the current type system):
func macro classConforming<R>(to existentialMetaType: R.Type) -> R
let classObject = classConforming(to: (any AProtocol.Type).self)
// or the old signature
let classObject = classConforming(to: AProtocol.Type.self)
now the type of classObject is any AnyProtocol.Type, which is the existential metatype you want.