Default Protocol Implementation Inheritance Behaviour - The current situation and what/if anything should be done about it

For a bit of background this stems from a bug I (and a number of other people) have reported on bugs.swift.org, the original being here: https://bugs.swift.org/browse/SR-103.

The Current Situation

For context let's start by looking at basic class inheritance, something we all know and can reason about:

class A {
    func whatAmI() {
        print("superclass")
    }
}

class B: A {
    override func whatAmI() {
        print("subclass")
    }
}

A().whatAmI()          // prints 'superclass'
B().whatAmI()          // prints 'subclass'
(B() as A).whatAmI()   // prints 'subclass'

If we create an instance of A then we know that the method on A will be called. If we create a subclass of A called B, override the whatAmI method and then call it, we expect the method on B to be called. This is even the case when we cast an instance of B to A because we are still operating on an instance of B.

Now let's throw a protocol into the mix. Let's call this protocol P, give it a default implementation through an extension and have A conform to it.

protocol P {
    func whatAmI()
}

extension P {
    func whatAmI() {
        print("protocol")
    }
}

class A: P {
    func whatAmI() {
        print("superclass")
    }
}

class B: A {
    override func whatAmI() {
        print("subclass")
    }
}

A().whatAmI()           // prints 'superclass'
B().whatAmI()           // prints 'subclass'
(B() as A).whatAmI()    // prints 'subclass'

As you can see the general behaviour hasn't changed. We still call through to the expected implementations just like in our first example.

But what happens if we remove the function implementation in A to rely on the default implementation in the protocol? To do that we also have to remove the override keyword for the function in B as we are no longer overriding any implementation in A. Once we do that we get the following:

protocol P {
    func whatAmI()
}

extension P {
    func whatAmI() {
        print("protocol")
    }
}

class A: P {
}

class B: A {
    func whatAmI() {
        print("subclass")
    }
}

A().whatAmI()           // prints 'protocol'
B().whatAmI()           // prints 'subclass'
(B() as A).whatAmI()    // prints 'protocol'

Calling the function on an instance of A does what we would expect, it doesn't have an implementation of the function any more so it relies on the default implementation in the protocol. With an instance of B we also get what we would expect, it just uses its implementation. The strange things start to happen when we start casting an instance of B to A, for some reason now we ignore the implementation in B and instead call through to the default protocol implementation.

This seems inconsistent to me, but it kind of makes sense if we look at it from the perspective of the compiler.

Just to note that I'm not compiler engineer and I don't pretend to know the inner workings of how the Swift compiler works, I'm simply making an assumption based on the results I'm observing.

What looks like is happening is that there are several tables at work that keep track of various method implementations in different places. Based on whichever method signatures match in these tables one implementation is being used over another. In the case of our examples above, there appears to be a table thats keeping track of methods that override methods further up the inheritance tree and the methods in this table always take president over whichever method they override in the superclass. If that assumption is correct, it makes sense that when we stop using the override keyword in B, the implementation in B never gets added to this overrides table and when we cast to A, A has nowhere to look to see if B has overridden any of its methods... which it technically hasn't as the implementation is coming from the protocol.

This is an interesting hypothesis so let's dig a bit further. Let's go back to a known state where we have a method implementation in A again, and override it in B. What happens if A calls whatAmI from within its init method?

protocol P {
    func whatAmI()
}

extension P {
    func whatAmI() {
        print("protocol")
    }
}

class A: P {
    init() {
        self.whatAmI()
    }

    func whatAmI() {
        print("superclass")
    }
}

class B: A {
    override func whatAmI() {
        print("subclass")
    }
}

let a = A()             // prints 'superclass'
a.whatAmI()             // prints 'superclass'
let b = B()             // prints 'subclass'
b.whatAmI()             // prints 'subclass'
(b as A).whatAmI()      // prints 'subclass'

So as you would expect, by using self.whatAmI() from within the init of A and creating an instance of B which inherits from A, it calls to the override of that method, not its own implementation.

So now what happens if we remove the implementation of whatAmI in A again? What method is called from the init in A? If the hypothesis is correct then A won't know where to look in any subclasses to see if the method has been overridden, so it will fall back to the implementation in the protocol extension.

protocol P {
    func whatAmI()
}

extension P {
    func whatAmI() {
        print("protocol")
    }
}

class A: P {
    init() {
        self.whatAmI()
    }
}

class B: A {
    func whatAmI() {
        print("subclass")
    }
}

let a = A()             // prints 'protocol'
a.whatAmI()             // prints 'protocol'
let b = B()             // prints 'protocol'
b.whatAmI()             // prints 'subclass'
(b as A).whatAmI()      // prints 'protocol'

Looking at the instance of A it's doing what we would expect. It has no implementation of the method whatAmI, so the call from within the init and the call from outside both go to the default protocol implementation.

Now we get to the instance of B where things appear to get very weird from a user point of view, but actually just fit nicely along with the hypothesis of whats going on in the compiler. So when we create an instance of B it goes to the init defined in A, typically the call in A would look into the overrides table to see if B had overridden any of it's methods, but since it hasn't it then reverts back to the default implementation in the protocol extension. This is in stark contrast to the behaviour you would typically experience if A did in fact have an implementation of the whatAmI method where the implementation on B would be used.

Moving on to calling the method from outside, we can see that the implementation in B is now called as you would expect, but then as soon as we cast B to A we revert back to the protocol implementation.

The Issue

We currently have an inconsistent behaviour between subtile changes in scenario. Simply removing the base implementation in a superclass that inherits from a protocol with a default implementation can change the entire behaviour of how a subclass works. This is pretty jarring to someone who doesn't know whats going on and is the source of a number of bugs (see the link at the top to see how many duplicates have been made about this).

Hopefully the examples above have highlighted what the problem is and how it can be reproduced.

The Future

The interesting part here is to discuss is what we expect to happen, what behaviour is predictable to the user and what implications any proposed changes have elsewhere.

From a totally simplistic point of view, a solution would be to have Swift understand that a superclass somewhere up the stack conforms to a protocol which has a default implementation. That default implementation needs to be overridden by any subclass no matter if the implementation is in the superclass or in a protocol extension. This would force the use of the override keyword in places that it currently isn't required, but it will make it more obvious to the user that there is some functionality that they are overriding.

The other thing to consider is how super is treated. In the above change it would mean that we could now call through to the protocol extension implementation by calling super in a subclass where the superclass didn't have an implementation. This is definitely an interesting change but I'm not sure if it's a desirable one or not. Unfortunately the implications of such a change as this are quite far reaching as it means broken source compatibility.

Another solution would be so have Swift just make the right choices without requiring adding the override keyword. As much as this is desirable from a source compatibility point of view, I would say that it's highly undesirable from a code understandability and 'sudden silent change from one version to another version' point of view.

I'm interested to know others thoughts on this and if there are other solutions that might be desirable.

9 Likes

It's good to see this being discussed!

You're correct that when the superclass relies on the default implementation of the protocol requirement, Swift doesn't add vtable entries for dynamic dispatch, and uses static dispatch instead.

One possible solution to this would be to 'opt-in' to vtable entries and dynamic dispatch by marking the method as an override(invalid code today), while deprecating the old behavior or otherwise warning if override is not present and static dispatch will be used. It's unclear if we'd want to be able to silence the warning if someone really wanted to use static dispatch though. The result would be source compatible, but potentially confusing if we aren't able to warn about the current behavior, because forgetting to include override could introduce subtle bugs.

Edit: Another case that needs to be considered is an open class which uses a default implementation being subclassed from a different module.

I think the problem here is that subclasses don't get a protocol witness table, only the class declaring the conformance does. This is why the default implementation is called in those cases.

I don't know anything about the Swift compiler and I didn't read the whole post, but your third example is weird to me too.

My mental image is that methods in Swift are dynamically dispatched. If an instance has a particular method declared, then that method should be used, no matter its compile-time type. Maybe that image is wrong, but it seems consistent with your first two examples.

I tend to agree. I think this behavior is exposing an implementation detail (PWTs) in a way that is confusing for people who are used to the traditional OOP inheritance behaviors.

That being said, I would be a little worried changing this at this point since it's a behavior some are likely to have internalized.

3 Likes

While I see the value in that I would be a bit cautious of having effectively two different behaviours separated by a warning. You are correct though, it would have the nicety of having source compatibility, so it could be an option to keep in mind.

To be perfectly honest I hadn't even thought about how this would work when using other modules, this is a very valid point :+1:t2:

That makes sense. It also makes me slightly cautious that any change proposed will effect the ABI. Does anyone know if that the case?

You are right. This is why I think we should go beyond our surprise, and look for potential use cases of this behavior. Try to internalize it and see what it can give. Without this, we won't suite be able to classify this behavior as a bug that needs fixing, or a rare trick that fills a niche and maybe needs some warnings where it looks misused.

2 Likes

I wonder if it might be helpful for a conformer to be able to explicitly call a default implementation from the protocol it is conforming to. This would open up the possibility of superclasses opting in to dynamic dispatch by having a one-liner:

func methodWithDefaultImpl(args...) { protocol.methodWithDefaultImple(args) }

This way, existing code works as-is, but new code could offer dynamic dispatch to its subclasses.

1 Like

This is an interesting approach and one that would work with source compatibility. I actually would like to explore this as a stand alone feature outside of this particular topic as it might have other interesting applications, but I need to have a think about it a bit more.

But coming back to the issue at hand I'm wondering if this particular solution for this particular issue just opens up doors for mistakes. It feels like putting the responsibility of circumventing an issue in the hands of the user rather than trying to fix the issue itself. If we really want to leave the user to fix it then as it stands we can circumvent it by always having an implementation in the superclass and then we get the behaviour that we would expect (but probably more copy and pasted code).

I think the main problem is discoverability about this behaviour and making it obvious whats going on rather than the user being surprised at an unexpected behaviour.

I wonder if it might be helpful for a conformer to be able to explicitly call a default implementation from the protocol it is conforming to

I think that would be great! At the moment, there's a workaround (although ugly):

protocol P {
  func foo()
}

extension P {
  func foo() { print("Hello, world") }
}

class A: P {
  func foo() {
    struct Dummy: P {}
    Dummy().foo()
  }
}
1 Like

That workaround only helps for such functions that are pure. Or, perversely, only modify nonlocal state.

2 Likes

Yeah

Some related discussions (there is a lot more, though):



1 Like

There is a variant of our surprise. Now with existentials (the whatIsIt function below which calls whatAmI):

protocol P {
    func whatAmI()
}

extension P {
    func whatAmI() { print("protocol") }
}

class A: P {
}

class B: A {
    func whatAmI() { print("subclass") }
}

A().whatAmI()           // prints 'protocol'
B().whatAmI()           // prints 'subclass'
(B() as A).whatAmI()    // prints 'protocol'

func whatIsIt(_ p: P) {
    p.whatAmI()
}

whatIsIt(A())       // prints 'protocol'
whatIsIt(B())       // prints 'protocol' (????)
whatIsIt(B() as A)  // prints 'protocol'

It really looks like A acts as a "mask" which can't be lifted. The method is not virtual at all, and subclasses are unable to get back the status of regular protocol adopters.

1 Like

Thanks @sjavora. Does any of those threads talks about the "blocking super-class" confusion that is discussed here? It looked like a genuine new topic to me?

The following also prints protocol.

(B() as P).whatAmI()

I don't think either example is exhibiting new behavior. Anything treated as P, whether a concrete type such as a A, or the existential for P, will use the default implementation provided by P.

Yes :-) And this is QUITE unexpected, because:

Not at all, not when the method is a protocol requirement:

protocol P {
    // requirement
    func whatAmI()
}

extension P {
    func whatAmI() { print("protocol") }
}

class C: P {
    func whatAmI() { print("C") }
}

(C() as P).whatAmI()    // prints 'C', as expected

Protocol requirements are supposed to be always dynamically dispatched. Here our super class blocks this mechanism.

5 Likes

In line with this, what do you think about the following explanation?

Invocations of a method gained via conformance to a protocol with a default implementation will always invoke the default implementation. This applies to concrete instances of the conforming type, subtypes which are coerced to the conforming (super)type, and the existential for the protocol which contains the default implementation.

OK, I actually expect that, but only because I remember that protocol requirements present customization points. If the conforming type customizes the implementation, it will be used. The sticky point is when a subclass tries to customize, and finds that it isn't really doing so.

Yes, precisely. The contract "protocol requirement implies dynamic dispatch" is broken by the super class. So either the contract was inexistent (a wrong assumption), either there is a bug in the implementation of the contract.

2 Likes
Terms of Service

Privacy Policy

Cookie Policy