Type check protocol conformance from extension to a generic

I have a type Factory<T> which I'm extending the following way:

extension Factory {
    func callAsFunction<U>() -> U {
        guard let result = callAsFunction() as? U else {
            preconditionFailure("Found '\(T.self)', but unable to convert it to '\(U.self)'")
        return result

I wanted to restrict this so you couldn't for example

protocol P { }
struct Valid: P { }
struct Invalid { }

let result: Valid = Factory<P>.callAsFunction() // <- This is a cast from P to Valid
let invalid: Invalid = Factory<P>.callAsFunction() // <- Error: Invalid does not conform to P

But I don't want a runtime error as I have now but rather a compiler error because it is checked that Invalid does not conform to P.

Is there any way to do this?