Proposal: Universal dynamic dispatch for method calls

[thorsten (thorsten@portableinnovations.de)]

Makes sense. Thanks for the explanation!

-Thorsten

···

Am 08.12.2015 um 00:52 schrieb Kevin Ballard via swift-evolution <swift-evolution@swift.org>:

On Mon, Dec 7, 2015, at 02:51 PM, Paul Cantrell wrote:

The ability to use static dispatch in most cases is one of the great features of Swift, and it's something that needs to be preserved, not relegated to being an optimization that can be done occasionally.

Can you think of a situation where that’s the desired semantics, as
opposed to either a desirable optimization or a confusing workaround for
the “monkey collision” problem? (I can’t see the use case, but I’m open
to hearing it!)

Yes. Any time you're replacing a global function with a protocol
extension, to preserve the semantics the protocol extension's method
implementation must be called every time the call is resolved via the
protocol (e.g. with generics or protocol objects, as opposed to calling
it on a concrete type that just happens to implement the protocol). If
you want the method implementation to be overridden by the concrete
implementing type, then the method should be declared in the protocol
instead of just in a protocol extension. Whether or not you want these
semantics is entirely up to you and your particular use-case.

I also disagree with the claim that the current rules are confusing. I
think they're very straightforward. Whenever you call a method on a
protocol value (either a generic type that's bound by the protocol, or a
protocol object), the only question that matters for method resolution
is "was this method declared in the protocol?". If it was declared in
the protocol, then concrete implementations can override it. If it was
not declared in the protocol, then concrete implementations cannot
override it. And this is a good thing. If concrete implementations could
override methods that were only provided through protocol extensions,
then that's a great way to accidentally override methods without
realizing it. Or more generally, if I declare an extension on some
protocol with some method, subjecting that method to overriding would be
a serious hazard, because there's no real way for me to know whether any
implementing type ends up accidentally overriding the method (and if
they did, they may not match the behavior expected from my method). The
only sane behavior here is to not subject it to overriding by default.

Incidentally, I rather like ilya's suggestion of declaring another
protocol with conditional conformance as the solution for declaring an
overridable method in a protocol extension. That provides a good way of
having this behavior without changing the semantics of protocols, and
all it requires is to be able to combine `where` clauses with protocol
conformance (which is something that's already known to be desired on
types, e.g. so Optional can conform to Equatable when Wrapped:
Equatable, so the only change here is extending that to protocols).

-Kevin Ballard
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Paul_Cantrell]

There’s no semantic difference between static & dynamic dispatch here. Static dispatch is an optimization the compiler can make.

(I’m assuming that compete() is secretly module-scoped, as Joe said it would be.)

P

···

On Dec 7, 2015, at 5:04 PM, ilya <ilya.nikokoshev@gmail.com> wrote:

On Tuesday, December 8, 2015, Paul Cantrell via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
> The ability to use static dispatch in most cases is one of the great features of Swift, and it's something that needs to be preserved, not relegated to being an optimization that can be done occasionally.

Can you think of a situation where that’s the desired semantics, as opposed to either a desirable optimization or a confusing workaround for the “monkey collision” problem? (I can’t see the use case, but I’m open to hearing it!)

Sure, private protocol extension. For example,

public protocol Animal {
  func greet()
  func sit()
  func run()
}

private extension Animal {
  func compete() { greet(); sit(); run()}
}

// we are provided with animal
... animal.compete()...

> It's also worth noting that the current behavior, where methods defined in protocol extensions can't be overridden by the implementing type, may actually be the desired behavior in many cases.

If you don’t want a method to be overridable, that’s what “final” is for.

The current behavior, in which a method is silently shadowed, does not seem to be to be a desirable behavior in any situation. The compiler warnings/errors Joe Groff proposed seem like a bare minimum, but I’d prefer to eliminate the situation that needs to be warned about in the first place.

Cheers, P

> On Dec 7, 2015, at 4:11 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org <>> wrote:
>
> I'd love to see these two key things that Joe mentioned get added.
> Diagnostics around confusing situations is a good idea (although there
> needs to be a way to disable it too; what if I intentionally need to use
> the same name in a type and in an extension to one of the conformed-to
> protocols, for various reasons?).
>
> I can't support the original proposal though, of making foo.bar() use
> dynamic dispatch. The ability to use static dispatch in most cases is
> one of the great features of Swift, and it's something that needs to be
> preserved, not relegated to being an optimization that can be done
> occasionally.
>
> It's also worth noting that the current behavior, where methods defined
> in protocol extensions can't be overridden by the implementing type, may
> actually be the desired behavior in many cases. Heck, the ability to
> trigger this different behavior is one of the three reasons to even
> define methods in protocol extensions instead of in the protocol (the
> others being when you're defining it for a subset of protocol
> implementations, and when you're defining extensions on protocols
> imported from other modules). Removing this functionality (by making
> foo.bar() always dynamic) means reverting back to the Swift 1.x
> convention of having global functions to provide this behavior.
>
> I wouldn't mind seeing something like the `dynamic` keyword to allow
> concrete types to override these protocol extension methods (although
> `dynamic` in this case would not imply @objc, like it does today, which
> may be a little confusing), but that would be opt-in behavior.
>
> -Kevin Ballard
>
> On Mon, Dec 7, 2015, at 12:14 PM, Joe Groff via swift-evolution wrote:
>> The main reason to constrain dynamic dispatch is modularity. The problems
>> of interfering categories or monkey-patches in languages like ObjC and
>> Ruby with late-bound dispatch and open classes are well-known. In Swift's
>> model, it isn't possible for an extension in one module to interfere with
>> existing protocol conformances or class hierarchies at runtime (except
>> with @objc of course). Modules need to be compiled together to be aware
>> of each other to interact with each other's interfaces.
>>
>> The particular example Alexandros brings up is more an artifact of our
>> existing implementation than desirable behavior. Two key things are
>> missing:
>>
>> - the ability for extensions to protocols to add new
>> dynamically-dispatched methods, and
>> - compiler quality work to diagnose confusing cases where concrete types
>> obviously shadow non-dynamic protocol extensions.
>>
>> There is a related behavior change proposal Doug Gregor's been working on
>> for classes, to have conforming to a protocol implicitly re-declare all
>> the protocol's methods as class methods, so that they can be overridden
>> by subclasses in the expected way.
>>
>> -Joe
>>
>>> On Dec 6, 2015, at 8:17 PM, Paul Cantrell via swift-evolution <swift-evolution@swift.org <>> wrote:
>>>
>>> One of the few things in Swift 2 that feels to me like a design flaw is the way Swift mixes static and dynamic method dispatch.
>>>
>>> Alexandros Salazar gives an excellent explanation of this problem — and I agree wholeheartedly with his title for the article:
>>>
>>> The Ghost of Swift Bugs Future
>>>
>>> The upshot is that when we see this:
>>>
>>> foo.bar()
>>>
>>> …it’s very hard to know how the compiler will determine which implementation of bar() to use. It might use static dispatch; it might use dynamic dispatch.
>>>
>>> The rules that govern this are arcane, and hard to remember. They have the feeling of being a “gotcha” question for job interviews — always a red flag for language features.
>>>
>>> Even if you remember the rules, the information needed to determine whether dispatch is static or dynamic is hard to track down. It depends on whether bar()’s implementation comes from an extension, whether the extension method appeared on the extended protocol, and whether the inferred type of foo is the protocol itself or an implementing type.
>>>
>>> A crucial part of the meaning of “foo.bar()” is implicit, and hard to determine. This runs contrary to Swift’s stated goal of prioritizing clarity at the point of API use, and its general pattern of making intent explicit. And it feels dangerous — a wellspring of insidious bugs.
>>>
>>> Thus:
>>>
>>>
>>> PROPOSAL
>>>
>>> Make the syntax “foo.bar()” always use dynamic dispatch, i.e. always use _only_ the runtime type of foo to determine which implementation of bar() to use. If an extension method collision occurs when a type implements multiple protocols, require the type to explicitly specify which one to use (as Swift already requires the caller to do at the point of invocation).
>>>
>>>
>>> I mean this proposal somewhat as a strawman. It’s such an obvious choice, I’m sure there were good reasons not to do it. But I’d like to propose the obvious solution in order to understand what’s wrong with it. I realize static dispatch precludes some optimizations, but I doubt that this alone drove the design choice. I see no safety or expressiveness upside to the way it works now.
>>>
>>> Cheers,
>>>
>>> Paul
>>>
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution@swift.org <>
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution@swift.org <>
>> https://lists.swift.org/mailman/listinfo/swift-evolution
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org <>
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
<>swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[Lily_Ballard]

> Yes. Any time you're replacing a global function with a protocol
> extension, to preserve the semantics [etc].

Indeed, I (mistakenly?) took this to be the original rationale. It seems
to me that, this case would be better served by some mechanism that makes
the non-override intent explicit at the point of declaration.

There is already such a mechanism. It's the fact that the declaration
happens in an extension instead of in the original protocol.

> I also disagree with the claim that the current rules are confusing.

In college, my brilliant mathematician friend Fazil divided all problems
into two categories: “impossible” and “trivial.” The difference was
whether Fazil knew the answer. Several times, I saw him tied in knots at
the whiteboard, and then a problem’s status would suddenly flip in a
single aha moment: “This is impossible. Impossib– Wait, no this is
trivial!” [starts erasing everything]

Point is, everything seems obvious when it’s no longer confusing. I stand
firm on my assertion that these rules are not obvious to those who
haven’t already studied them carefully, and that a reshuffling that makes
intent more explicit (and possibly allows dynamic dispatch in more
circumstances) would prevent a lot of bugs out there in the wild.

I'm saying it's not confusing because there is an extremely simple rule
to determine whether method call through a protocol type allows
overriding (e.g. dynamic dispatch). And the rule is: Was the method
declared in the protocol? If it was, great! You get dynamic dispatch. If
it wasn't, there's no dynamic dispatch. The reason why this is confusing
to people is because nobody's clearly stating the rule.

I also disagree with the claim that changing this would prevent a lot of
bugs. I think it would introduce plenty of other bugs, because methods
in protocol extensions would start being unexpectedly overridden.

Dynamic dispatch in Swift is something that is opt-in, and it should
stay that way. The only implicit dynamic dispatch is methods/properties
on non-final classes can be overridden in subclasses, but that's an
explicit feature of classes (and still requires the `override` keyword
on the subclass method/property). Protocols of course have dynamic
dispatch, but only when used as protocol objects; it's still static
dispatch when used as generic type constraints (nitpick: the change
being discussed here also affects static method dispatch resolution in
specialization of generic functions, and lumping this under "dynamic
dispatch" is slightly misleading).

If we must mix static and dynamic dispatch, I’d like to make sure every
Swift programmer _chooses_ one instead of accidentally using it. I’m sure
the language could do better on that front, given Joe’s comment about
module-based scoping of names. I need to give this some more thought.

The language already makes you choose it today. If this is unclear to
people, then we need to perhaps work on how protocols and protocol
extensions are explained instead of further muddying the waters by
making things dynamically-dispatched that really don't need to be.

-Kevin Ballard

···

On Mon, Dec 7, 2015, at 07:45 PM, Paul Cantrell wrote:

[Brent_Royal-Gordon]

It seems
to me that, this case would be better served by some mechanism that makes
the non-override intent explicit at the point of declaration.

There is already such a mechanism. It's the fact that the declaration
happens in an extension instead of in the original protocol.

No, what you’re saying is that there’s a simple rule governing which declarations are dispatched statically. This does not make it *explicit*. In the Swift 1 betas, there was a simple rule governing whether `if foo` was a nil check (is `foo` of type Optional?), but this was deemed insufficiently explicit and removed. Making the protocol extension behaivor *explicit* would mean, for instance, forcing the user to mark statically-dispatched protocol extension methods `final`, much like how you have to use `override` purely to reassure the compiler that you mean to do what your code says.

Given the tenor of the rest of the language, here’s how I would design this:

1. You can mark a protocol extension member as `final`. This means that, when casted to the protocol’s type, all instances will use *this* implementation, so it can safely be statically dispatched.

  protocol Foo {
    func bar()
  }
  extension Foo {
    final func baz() { … }
  }

2. Currently, all protocol extension members which aren’t listed in the protocol itself must be final:

  protocol Foo {
    func bar()
  }
  extension Foo {
    func baz() { … } // error: declaration must be ‘final’.
  }

If a future version of the language were to allow overriding of extension methods, this rule would go away, and use of final would be purely voluntary.

3. Attempting to make a type conform to a protocol that has `final` members conflicting with its own causes an error:

  class ConcreteFoo {
    func baz() { … }
  }
  extension ConcreteFoo: Foo { // error: Foo.baz() conflicts with ConcreteFoo.baz()
    func bar() { ... }
  }

4. The preferred way to handle these conflicts is to rename one of the members so they no longer conflict. However, if that isn’t possible, you can use a keyword to acknowledge and silence the error, either at the site of the conformance (preferred):

  extension ConcreteFoo: @incoherent Foo {

Or at the site of the extension (as a fallback for when someone else wrote the conformance):

  @incoherent(ConcreteFoo) extension Foo {
    final func baz() { … }
  }

It may also be necessary to allow it at the site of an import:

  import FooKit
  import ConcreteFooKit
  @incoherent(ConcreteFoo: Foo) import FooExtensions

I like “incoherent” because, to me, it expresses the symptom you’re going to see (viewing the object as a Foo will not give you the same members as viewing it as a ConcreteFoo). But @conflicting would be more obvious, and of course we could bikeshed this endlessly, in what appears to be the manner of our people.

···

--
Brent Royal-Gordon
Architechies

[Lily_Ballard]

How does this handle the case where the extension was not done in the
same module that defined the protocol, and the extension method's name
is the same as a method on one of its implementing types (from the same
module, or another module that can see the protocol but not the
extension)? Calling the method via the protocol cannot possibly call the
implementing type's version, it must call the default implementation in
the extension, for two reasons:

1. The module that defined the type doesn't know about the extension, so
the method on the type was written without any knowledge of the
extension's method and may not match the expected behavior for the
protocol method.
2. There's no protocol witness table that can contain the extension
method for that type, so even if #1 was irrelevant, there'd be no way to
actually call it. The module that defined the type has no knowledge of
the protocol extension, so it can't construct a protocol witness table.
And the module that defined the extension doesn't know about the type
(since it's in a different module).

The only way this could work is if the extension is in the same module
as the protocol definition, or if the type that conforms to the protocol
can also see the extension.

These same problems occur with an opt-in `dynamic` keyword for protocol
extension methods, of course, but at least there the method is
explicitly called-out as being special and so it's a little more
reasonable to understand that it will only be overridable by types that
can see the extension.

-Kevin Ballard

···

On Tue, Dec 8, 2015, at 01:51 PM, Brent Royal-Gordon wrote:

>> It seems
>> to me that, this case would be better served by some mechanism that makes
>> the non-override intent explicit at the point of declaration.
>
> There is already such a mechanism. It's the fact that the declaration
> happens in an extension instead of in the original protocol.

No, what you’re saying is that there’s a simple rule governing which
declarations are dispatched statically. This does not make it *explicit*.
In the Swift 1 betas, there was a simple rule governing whether `if foo`
was a nil check (is `foo` of type Optional?), but this was deemed
insufficiently explicit and removed. Making the protocol extension
behaivor *explicit* would mean, for instance, forcing the user to mark
statically-dispatched protocol extension methods `final`, much like how
you have to use `override` purely to reassure the compiler that you mean
to do what your code says.

Given the tenor of the rest of the language, here’s how I would design
this:

1. You can mark a protocol extension member as `final`. This means that,
when casted to the protocol’s type, all instances will use *this*
implementation, so it can safely be statically dispatched.

  protocol Foo {
    func bar()
  }
  extension Foo {
    final func baz() { … }
  }

2. Currently, all protocol extension members which aren’t listed in the
protocol itself must be final:

  protocol Foo {
    func bar()
  }
  extension Foo {
    func baz() { … } // error: declaration must be ‘final’.
  }

If a future version of the language were to allow overriding of extension
methods, this rule would go away, and use of final would be purely
voluntary.

3. Attempting to make a type conform to a protocol that has `final`
members conflicting with its own causes an error:

  class ConcreteFoo {
    func baz() { … }
  }
  extension ConcreteFoo: Foo { // error: Foo.baz() conflicts with ConcreteFoo.baz()
    func bar() { ... }
  }

4. The preferred way to handle these conflicts is to rename one of the
members so they no longer conflict. However, if that isn’t possible, you
can use a keyword to acknowledge and silence the error, either at the
site of the conformance (preferred):

  extension ConcreteFoo: @incoherent Foo {

Or at the site of the extension (as a fallback for when someone else
wrote the conformance):

  @incoherent(ConcreteFoo) extension Foo {
    final func baz() { … }
  }

It may also be necessary to allow it at the site of an import:

  import FooKit
  import ConcreteFooKit
  @incoherent(ConcreteFoo: Foo) import FooExtensions

I like “incoherent” because, to me, it expresses the symptom you’re going
to see (viewing the object as a Foo will not give you the same members as
viewing it as a ConcreteFoo). But @conflicting would be more obvious, and
of course we could bikeshed this endlessly, in what appears to be the
manner of our people.

--
Brent Royal-Gordon
Architechies

[Brent_Royal-Gordon]

How does this handle the case where the extension was not done in the
same module that defined the protocol, and the extension method's name
is the same as a method on one of its implementing types (from the same
module, or another module that can see the protocol but not the
extension)?

I’m not sure I understand what you’re asking; an annotated code sample might help. But if I *do* understand what you’re saying, that’s what @incoherent on an import is meant to resolve—the case where you import a bunch of modules which don’t individually conflict, but as a group do.

Keep in mind that what I’m proposing here doesn’t actually change Swift’s runtime behavior at all. Note #2—you’re *required* to make protocol extension members `final` unless they’re listed in the protocol declaration. What I’m proposing is merely a series of compile-time statements used to tell swiftc that, yes, I know this is going to happen.

···

--
Brent Royal-Gordon
Architechies

[Paul_Cantrell]

Syntactic details aside, I strongly suppose the general spirit of Brent’s proposal.

I’m not sure about @incoherent and all the details, but the general approach seems clearly correct to me: explicitness, clarity, and a simplified mental model that do not limit existing capabilities.

How does this handle the case where the extension was not done in the
same module that defined the protocol

Did Joe Groff not already address this early on in this thread?

Joe wrote:

It's helpful to think of method names as being namespaced in Swift, by both their enclosing module and type. If two modules independently extend a protocol with a method of the same name, you still semantically have two distinct methods that dispatch independently. The extension would have to be factored into a common module both modules see for them to interact.

P

···

On Dec 8, 2015, at 4:53 PM, Kevin Ballard <kevin@sb.org> wrote:

How does this handle the case where the extension was not done in the
same module that defined the protocol, and the extension method's name
is the same as a method on one of its implementing types (from the same
module, or another module that can see the protocol but not the
extension)? Calling the method via the protocol cannot possibly call the
implementing type's version, it must call the default implementation in
the extension, for two reasons:

1. The module that defined the type doesn't know about the extension, so
the method on the type was written without any knowledge of the
extension's method and may not match the expected behavior for the
protocol method.
2. There's no protocol witness table that can contain the extension
method for that type, so even if #1 was irrelevant, there'd be no way to
actually call it. The module that defined the type has no knowledge of
the protocol extension, so it can't construct a protocol witness table.
And the module that defined the extension doesn't know about the type
(since it's in a different module).

The only way this could work is if the extension is in the same module
as the protocol definition, or if the type that conforms to the protocol
can also see the extension.

These same problems occur with an opt-in `dynamic` keyword for protocol
extension methods, of course, but at least there the method is
explicitly called-out as being special and so it's a little more
reasonable to understand that it will only be overridable by types that
can see the extension.

-Kevin Ballard

On Tue, Dec 8, 2015, at 01:51 PM, Brent Royal-Gordon wrote:

It seems
to me that, this case would be better served by some mechanism that makes
the non-override intent explicit at the point of declaration.

There is already such a mechanism. It's the fact that the declaration
happens in an extension instead of in the original protocol.

No, what you’re saying is that there’s a simple rule governing which
declarations are dispatched statically. This does not make it *explicit*.
In the Swift 1 betas, there was a simple rule governing whether `if foo`
was a nil check (is `foo` of type Optional?), but this was deemed
insufficiently explicit and removed. Making the protocol extension
behaivor *explicit* would mean, for instance, forcing the user to mark
statically-dispatched protocol extension methods `final`, much like how
you have to use `override` purely to reassure the compiler that you mean
to do what your code says.

Given the tenor of the rest of the language, here’s how I would design
this:

1. You can mark a protocol extension member as `final`. This means that,
when casted to the protocol’s type, all instances will use *this*
implementation, so it can safely be statically dispatched.

  protocol Foo {
    func bar()
  }
  extension Foo {
    final func baz() { … }
  }

2. Currently, all protocol extension members which aren’t listed in the
protocol itself must be final:

  protocol Foo {
    func bar()
  }
  extension Foo {
    func baz() { … } // error: declaration must be ‘final’.
  }

If a future version of the language were to allow overriding of extension
methods, this rule would go away, and use of final would be purely
voluntary.

3. Attempting to make a type conform to a protocol that has `final`
members conflicting with its own causes an error:

  class ConcreteFoo {
    func baz() { … }
  }
  extension ConcreteFoo: Foo { // error: Foo.baz() conflicts with ConcreteFoo.baz()
    func bar() { ... }
  }

4. The preferred way to handle these conflicts is to rename one of the
members so they no longer conflict. However, if that isn’t possible, you
can use a keyword to acknowledge and silence the error, either at the
site of the conformance (preferred):

  extension ConcreteFoo: @incoherent Foo {

Or at the site of the extension (as a fallback for when someone else
wrote the conformance):

  @incoherent(ConcreteFoo) extension Foo {
    final func baz() { … }
  }

It may also be necessary to allow it at the site of an import:

  import FooKit
  import ConcreteFooKit
  @incoherent(ConcreteFoo: Foo) import FooExtensions

I like “incoherent” because, to me, it expresses the symptom you’re going
to see (viewing the object as a Foo will not give you the same members as
viewing it as a ConcreteFoo). But @conflicting would be more obvious, and
of course we could bikeshed this endlessly, in what appears to be the
manner of our people.

--
Brent Royal-Gordon
Architechies

[Lily_Ballard]

> How does this handle the case where the extension was not done in the
> same module that defined the protocol, and the extension method's name
> is the same as a method on one of its implementing types (from the same
> module, or another module that can see the protocol but not the
> extension)?

I’m not sure I understand what you’re asking; an annotated code sample might help. But if I *do* understand what you’re saying, that’s what @incoherent on an import is meant to resolve—the case where you import a bunch of modules which don’t individually conflict, but as a group do.

Ah, yeah, looks like your @incoherent does cover this, although I'm not really sure what the benefit of putting @incoherent on an import is. People will just end up putting that there whenever the compiler tells them to, without really understanding why. And when reading the resulting source, it doesn't tell the reader what it's actually trying to resolve, beyond "something to do with type Foo".

Keep in mind that what I’m proposing here doesn’t actually change Swift’s runtime behavior at all. Note #2—you’re *required* to make protocol extension members `final` unless they’re listed in the protocol declaration. What I’m proposing is merely a series of compile-time statements used to tell swiftc that, yes, I know this is going to happen.

I missed the part about `final` being required.

It seems to me that the only real benefit here is making it so that when you look at a protocol extension you can tell immediately which methods are just default implementations and which ones are actually declaring new extension methods (that can't be overridden). And even that's just saving you from having to look at the protocol declaration to see what methods are in there.

If that visual distinction is useful, I'd rather see this done by using the already-existing `default` keyword as a modifier for methods that are just providing default implementations. And if you try and declare a method in an extension that matches the type signature of a protocol method but doesn't say `default` the compiler would yell at you. This would look something like

protocol Foo {
    func foo() -> String
}

extension Foo {
    // default implementation of Foo.foo()
    default foo() -> String {
        return "foo"
    }
    // new method on all Foos that can't be overridden
    func bar() -> String {
        return "bar"
    }
}

-Kevin Ballard

···

On Tue, Dec 8, 2015, at 03:00 PM, Brent Royal-Gordon wrote:

[gwendal.roue (Gwendal Roué)]

Whatever the direction this proposal is aiming at, please remember that it is desirable for some APIs to let adopting types "override" the default implementation provided by protocols.

The current static dispatch of protocol extensions allows adopting types to perform such an "override":

    protocol Feature { }
    extension Feature {
        func run() {
            // default implementation
        }
    }
    struct S : Feature {
        func run() {
            // "inherited" from Feature:
            (self as Feature).run()
            // extra code
            ...
        }
    }

This is important, because *without* this static dispatch, the protocol P *must* be turned into a *base class* as soon as three conditions are met:

- the implementation inside the run() function is rather complex
- it is desirable to let adopting types able to alter the default behavior.
- bloating the API and the documentation with extra global functions or extra methods that work around this limitation of the language is not desirable

    // Terse API with a Feature base class:
    class Feature {
        func run() {
            // rather complex implementation
        }
    }
    class C : Feature {
        func run() {
            // Inherited from Feature:
            super.ext()
            // extra code
            ...
        }
    }
    
    // Non-terse API with extra methods and documentation:
    protocol Feature { }
    extension Feature {
        func run() {
            self._run()
        }
        /// Don't call this method directly, but use run() instead.
        /// Adopting types may call this method in their own implementation
        /// of run().
        func _run() {
            // default implementation
        }
    }
    struct S : Feature {
        func run() {
            // "inherited" from Feature:
            self._run()
            // extra code
            ...
        }
    }

You have understood that I'm against those extra methods and documentation, and that I prefer the base class solution, despite protocols being all the rage.

Actually, the use case I'm talking here is not properly addressed by the current state of Swift because of a bug with mutating methods [SR-142] mutating function in protocol extension erroneously requires `var` declaration of class variables · Issue #42764 · apple/swift · GitHub which has the consequence of forcing APIs to keep on exposing a base class instead of a protocol (see Protocol with CRUD operations for structs · Issue #12 · groue/GRDB.swift · GitHub for a practical problem).

Anyway, please remember that it is desirable for some APIs to let adopting types "override" the default implementation provided by protocols.

Gwendal Roué

···

Le 9 déc. 2015 à 00:26, Kevin Ballard via swift-evolution <swift-evolution@swift.org> a écrit :

Keep in mind that what I’m proposing here doesn’t actually change Swift’s runtime behavior at all. Note #2—you’re *required* to make protocol extension members `final` unless they’re listed in the protocol declaration. What I’m proposing is merely a series of compile-time statements used to tell swiftc that, yes, I know this is going to happen.

[Paul_Cantrell]

Gwendal Roué wrote:

Whatever the direction this proposal is aiming at, please remember that it is desirable for some APIs to let adopting types "override" the default implementation provided by protocols.

I would certainly hope nobody is proposing changing that! Once a method is virtually dispatched, it should be virtual all the way down to the implementing type.

P

···

On Dec 8, 2015, at 10:21 PM, Gwendal Roué via swift-evolution <swift-evolution@swift.org> wrote:

Le 9 déc. 2015 à 00:26, Kevin Ballard via swift-evolution <swift-evolution@swift.org> a écrit :

Keep in mind that what I’m proposing here doesn’t actually change Swift’s runtime behavior at all. Note #2—you’re *required* to make protocol extension members `final` unless they’re listed in the protocol declaration. What I’m proposing is merely a series of compile-time statements used to tell swiftc that, yes, I know this is going to happen.

Whatever the direction this proposal is aiming at, please remember that it is desirable for some APIs to let adopting types "override" the default implementation provided by protocols.

The current static dispatch of protocol extensions allows adopting types to perform such an "override":

   protocol Feature { }
   extension Feature {
       func run() {
           // default implementation
       }
   }
   struct S : Feature {
       func run() {
           // "inherited" from Feature:
           (self as Feature).run()
           // extra code
           ...
       }
   }

This is important, because *without* this static dispatch, the protocol P *must* be turned into a *base class* as soon as three conditions are met:

- the implementation inside the run() function is rather complex
- it is desirable to let adopting types able to alter the default behavior.
- bloating the API and the documentation with extra global functions or extra methods that work around this limitation of the language is not desirable

   // Terse API with a Feature base class:
   class Feature {
       func run() {
           // rather complex implementation
       }
   }
   class C : Feature {
       func run() {
           // Inherited from Feature:
           super.ext()
           // extra code
           ...
       }
   }

   // Non-terse API with extra methods and documentation:
   protocol Feature { }
   extension Feature {
       func run() {
           self._run()
       }
       /// Don't call this method directly, but use run() instead.
       /// Adopting types may call this method in their own implementation
       /// of run().
       func _run() {
           // default implementation
       }
   }
   struct S : Feature {
       func run() {
           // "inherited" from Feature:
           self._run()
           // extra code
           ...
       }
   }

You have understood that I'm against those extra methods and documentation, and that I prefer the base class solution, despite protocols being all the rage.

Actually, the use case I'm talking here is not properly addressed by the current state of Swift because of a bug with mutating methods [SR-142] mutating function in protocol extension erroneously requires `var` declaration of class variables · Issue #42764 · apple/swift · GitHub which has the consequence of forcing APIs to keep on exposing a base class instead of a protocol (see Protocol with CRUD operations for structs · Issue #12 · groue/GRDB.swift · GitHub for a practical problem).

Anyway, please remember that it is desirable for some APIs to let adopting types "override" the default implementation provided by protocols.

Gwendal Roué

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[gwendal.roue (Gwendal Roué)]

Yes, and it’s why I’m concerned. When a protocol provides a default implementation, how will an adopting type invoke that default implementation in its own implementation?

With classes, it’s easy: you call super.

With protocols, it’s much less easy, if possible at all, as soon as all methods are dynamically dispatched.

Gwendal Roué

···

Le 9 déc. 2015 à 06:31, Paul Cantrell <cantrell@pobox.com> a écrit :

Gwendal Roué wrote:

Whatever the direction this proposal is aiming at, please remember that it is desirable for some APIs to let adopting types "override" the default implementation provided by protocols.

I would certainly hope nobody is proposing changing that! Once a method is virtually dispatched, it should be virtual all the way down to the implementing type.

[griotspeak (TJ Usiyan)]

I actually suggest something along the lines of

    protocol Foo {
        final func doIt() -> String
    }

    extension Foo {
        final func doIt() -> String {
            print("I did it.")
        }
        final func doThat() {
            print("I did that.")
        }
    }

to indicate that foo will be provided in this module and is not ever to be
dynamically dispatched. This draws attention to the fact that dispatch is
static in a clear consistent manner.

TJ

···

On Wed, Dec 9, 2015 at 1:31 PM, Gwendal Roué <swift-evolution@swift.org> wrote:

> Le 9 déc. 2015 à 06:31, Paul Cantrell <cantrell@pobox.com> a écrit :
>
> Gwendal Roué wrote:
>> Whatever the direction this proposal is aiming at, please remember that
it is desirable for some APIs to let adopting types "override" the default
implementation provided by protocols.
>
>
> I would certainly hope nobody is proposing changing that! Once a method
is virtually dispatched, it should be virtual all the way down to the
implementing type.

Yes, and it’s why I’m concerned. When a protocol provides a default
implementation, how will an adopting type invoke that default
implementation in its own implementation?

With classes, it’s easy: you call super.

With protocols, it’s much less easy, if possible at all, as soon as all
methods are dynamically dispatched.

Gwendal Roué

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[gwendal.roue (Gwendal Roué)]

In this particular example, do you mean that a class that implements Foo would have to declare its own doIt() as final, and can not implement its own doThat() ?

    class C : Foo {
        final func doIt() -> String { … } // final required
        final func doThat() { … } // illegal
    }

Gwendal

···

Le 9 déc. 2015 à 11:22, T.J. Usiyan <griotspeak@gmail.com> a écrit :

I actually suggest something along the lines of

    protocol Foo {
        final func doIt() -> String
    }

    extension Foo {
        final func doIt() -> String {
            print("I did it.")
        }
        final func doThat() {
            print("I did that.")
        }
    }

to indicate that foo will be provided in this module and is not ever to be dynamically dispatched. This draws attention to the fact that dispatch is static in a clear consistent manner.

TJ

[griotspeak (TJ Usiyan)]

I did not mean that but I can sort of see how you came to that. My intent
was that `final func doIt()` is a statement that `Foo.doIt()` is provided
in the same module but is not dynamic. It would be the same as what
currently happens when a method is not in a protocol but a default
implementation is provided.

···

On Wed, Dec 9, 2015 at 4:00 PM, Gwendal Roué <gwendal.roue@gmail.com> wrote:

> Le 9 déc. 2015 à 11:22, T.J. Usiyan <griotspeak@gmail.com> a écrit :
>
> I actually suggest something along the lines of
>
> protocol Foo {
> final func doIt() -> String
> }
>
> extension Foo {
> final func doIt() -> String {
> print("I did it.")
> }
> final func doThat() {
> print("I did that.")
> }
> }
>
> to indicate that foo will be provided in this module and is not ever to
be dynamically dispatched. This draws attention to the fact that dispatch
is static in a clear consistent manner.
>
> TJ

In this particular example, do you mean that a class that implements Foo
would have to declare its own doIt() as final, and can not implement its
own doThat() ?

    class C : Foo {
        final func doIt() -> String { … } // final required
        final func doThat() { … } // illegal
    }

Gwendal

[thorsten (thorsten@portableinnovations.de)]

This won't work if the protocol is provided by someone else (meaning I can't change its source code) and I want to provide a protocol extension in my own module.

I like the original idea of having to mark extension methods not in the protocol definition as final. This would make the semantics much clearer.

-Thorsten

···

Am 09.12.2015 um 11:22 schrieb T.J. Usiyan via swift-evolution <swift-evolution@swift.org>:

I actually suggest something along the lines of

    protocol Foo {
        final func doIt() -> String
    }

    extension Foo {
        final func doIt() -> String {
            print("I did it.")
        }
        final func doThat() {
            print("I did that.")
        }
    }

to indicate that foo will be provided in this module and is not ever to be dynamically dispatched. This draws attention to the fact that dispatch is static in a clear consistent manner.

TJ

On Wed, Dec 9, 2015 at 1:31 PM, Gwendal Roué <swift-evolution@swift.org> wrote:

> Le 9 déc. 2015 à 06:31, Paul Cantrell <cantrell@pobox.com> a écrit :
>
> Gwendal Roué wrote:
>> Whatever the direction this proposal is aiming at, please remember that it is desirable for some APIs to let adopting types "override" the default implementation provided by protocols.
>
>
> I would certainly hope nobody is proposing changing that! Once a method is virtually dispatched, it should be virtual all the way down to the implementing type.

Yes, and it’s why I’m concerned. When a protocol provides a default implementation, how will an adopting type invoke that default implementation in its own implementation?

With classes, it’s easy: you call super.

With protocols, it’s much less easy, if possible at all, as soon as all methods are dynamically dispatched.

Gwendal Roué

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Lily_Ballard]

Any thoughts on instead having to mark the default method
implementations as `default`?

It still seems strange to me to consider a protocol extension that uses
the `final` keyword, because a) protocols have nothing to do with
classes and `final` means no subclassing, and b) methods defined in
protocol extensions by definition can't be overridden already, so the
`final` keyword is just sort of like saying "no really, I mean it, you
can't do the thing you already can't do!". But since I see the value in
making it clear which methods are default implementations (and thus can
be overridden), and which are actually just new methods, I'm in favor of
using the `default` keyword on the former:

protocol Foo { func foo() -> String }

extension Foo { default func foo() -> String { return "foo!" }
func bar() -> String { return "bar" } }

-Kevin

This won't work if the protocol is provided by someone else (meaning I
can't change its source code) and I want to provide a protocol
extension in my own module.

I like the original idea of having to mark extension methods not in
the protocol definition as final. This would make the semantics much
clearer.

-Thorsten

I actually suggest something along the lines of

protocol Foo { final func doIt() -> String }

extension Foo { final func doIt() -> String {
print("I did it.") } final func doThat() {
print("I did that.") } }

to indicate that foo will be provided in this module and is not ever
to be dynamically dispatched. This draws attention to the fact that
dispatch is static in a clear consistent manner.

TJ

Gwendal Roué wrote:

Whatever the direction this proposal is aiming at, please remember
that it is desirable for some APIs to let adopting types "override"
the default implementation provided by protocols.

I would certainly hope nobody is proposing changing that! Once a
method is virtually dispatched, it should be virtual all the way down
to the implementing type.

Yes, and it’s why I’m concerned. When a protocol provides a default
implementation, how will an adopting type invoke that default
implementation in its own implementation?

With classes, it’s easy: you call super.

With protocols, it’s much less easy, if possible at all, as soon as all
methods are dynamically dispatched.

Gwendal Roué

···

On Wed, Dec 9, 2015, at 09:37 AM, thorsten--- via swift-evolution wrote:

Am 09.12.2015 um 11:22 schrieb T.J. Usiyan via swift-evolution <swift- > evolution@swift.org>:

On Wed, Dec 9, 2015 at 1:31 PM, Gwendal Roué <swift- >> evolution@swift.org> wrote:

Le 9 déc. 2015 à 06:31, Paul Cantrell <cantrell@pobox.com> a écrit :

_______________________________________________

swift-evolution mailing list

swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_________________________________________________
swift-evolution mailing list swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Brent_Royal-Gordon]

Any thoughts on instead having to mark the default method implementations as `default`?

There are three reasons I like `final`:

1) It echoes the meaning of `final` in a class—“this is the implementation and you cannot provide any other”.

2) Swift’s general slant is towards using virtual dispatch unless it’s forbidden. (Leaving aside cases where the optimizer can prove at compile time what a dynamic dispatch would do.) That’s why we have a `final` keyword instead of a `virtual` keyword. Because of this, it’s not surprising to Swift users that you *can* override some extension methods; rather, the surprise is that you *can’t* override others. (And the *real* surprise is that you can kind-of-but-not-really override them, and Swift doesn’t complain.) I want developers to mark the surprise.

3) Since, as I said, Swift’s slant is towards virtual dispatch, I think that if Swift eventually permits overriding of protocol extension methods, the overridable behavior should be the default, and the current static behavior should be something you request.

But this is not a terribly strong preference; `default` would be alright too. The more important point to me is that the user should acknowledge the weird behavior of shadowing a protocol extension method. In other words, I think the `@incoherent` annotation on the conformance is actually more important than the `final` or `default` keyword on the declaration.

It still seems strange to me to consider a protocol extension that uses the `final` keyword, because a) protocols have nothing to do with classes and `final` means no subclassing

The way I see it, `final` in a class is a way of saying “All instances of this class (including subclasses) must use this implementation and cannot substitute a different one”. Similarly, `final` in a protocol means “All instances conforming to this protocol must use this implementation and cannot substitute a different one”.

That’s why @incoherent is important. It’s basically saying “I know there are final members in this type and I can’t really override them, but I want to shadow them anyway, even though the result is going to be kind of half-assed.” You could even imagine it being applied to superclasses, allowing you to shadow their `final` members in the same way you can shadow protocol extension methods.

and b) methods defined in protocol extensions by definition can't be overridden already, so the `final` keyword is just sort of like saying "no really, I mean it, you can't do the thing you already can't do!”.

That is *exactly* the point of what I’m proposing. Just like the `override` keyword sort of says “no really, I mean it, I want to do the thing I’m doing!"

···

--
Brent Royal-Gordon
Architechies

[gwendal.roue (Gwendal Roué)]

Methods defined in protocol extension actually can, sort of, be overridden, and this is very useful:

    protocol P { }
    extension P {
        func f() { … }
    }
    struct S {
        func f() {
            ...
            (self as P).f()
            …
        }
    }

I know only one use case for this technique, in the groue/GRDB.swift SQLite wrapper:

In this library, a DatabasePersistable protocol provides basic CRUD operations, and a Record class adopts this protocol and "overrides" with the technique above the protocol methods with extra features provided by the class (especially change tracking).

The benefits of this architecture are:

- You can subclass use the full-featured Record base class, and get CRUD + change tracking for free.
- The Record subclasses can override the CRUD methods, and add custom code (validation, for example).
- You can have a custom struct adopt DatabasePersistable, and get CRUD for free.
- The custom structs that can also "override" the CRUD methods, and add custom code (validation, for example).

This is, in my opinion, a very valid use case for this "overriding".
Gwendal Roué

···

Le 9 déc. 2015 à 19:52, Kevin Ballard via swift-evolution <swift-evolution@swift.org> a écrit :

b) methods defined in protocol extensions by definition can't be overridden already,

[Lily_Ballard]

I don't agree that Swift's general slant is towards using virtual dispatch. Only classes and protocol objects use virtual dispatch[1]. And Swift even provides a mechanism (final / static) to remove dynamic dispatch from classes. It seems to me that Swift's general slant is towards encouraging static dispatch whenever possible.

[1] Generic types bounded by protocols is a borderline case. Semantically, they use static dispatch. In optimized builds, specializations are produced for the various type parameterizations that use static dispatch (although a generic virtual dispatch version is still emitted too, but I don't know when that's ever used). I assume the virtual dispatch is done just for the sake of improving compile times (by not having to generate the specializations), but to the best of my knowledge there's no behavioral difference here and it should be indistinguishable from static dispatch

-Kevin Ballard

···

On Wed, Dec 9, 2015, at 04:09 PM, Brent Royal-Gordon wrote:

2) Swift’s general slant is towards using virtual dispatch unless it’s forbidden. (Leaving aside cases where the optimizer can prove at compile time what a dynamic dispatch would do.) That’s why we have a `final` keyword instead of a `virtual` keyword. Because of this, it’s not surprising to Swift users that you *can* override some extension methods; rather, the surprise is that you *can’t* override others. (And the *real* surprise is that you can kind-of-but-not-really override them, and Swift doesn’t complain.) I want developers to mark the surprise.

[Paul_Cantrell]

There are three reasons I like `final`:

1) It echoes the meaning of `final` in a class—“this is the implementation and you cannot provide any other”.

2) Swift’s general slant is towards using virtual dispatch unless it’s forbidden. (Leaving aside cases where the optimizer can prove at compile time what a dynamic dispatch would do.) That’s why we have a `final` keyword instead of a `virtual` keyword. Because of this, it’s not surprising to Swift users that you *can* override some extension methods; rather, the surprise is that you *can’t* override others. (And the *real* surprise is that you can kind-of-but-not-really override them, and Swift doesn’t complain.) I want developers to mark the surprise.

3) Since, as I said, Swift’s slant is towards virtual dispatch, I think that if Swift eventually permits overriding of protocol extension methods, the overridable behavior should be the default, and the current static behavior should be something you request.

Well said, Brent. I agree completely with all these points.

#2 is precisely the reason I threw out that strawman proposal to open the discussion. That strawman long since torn down to satisfaction, I’ve updated the subject line.

The details of the solution are tricky, but I like this general approach with “final” (or whatever the right keyword is). It passes the smell test for me.

• • •

I’m not yet sold on @incoherent. What about preserving the current approach: whenever there are two semantically distinct methods with the same name, require the caller to disambiguate at the call site? For example:

    //––– Module Foo ———

    extension String {
        final func flip() {
            // reverse order of letters
        }
    }

    //––– Module Bar ———

    extension String {
        final func flip() {
            // turn letters upside down using funny Unicode chars
        }
    }

    //——— Third module ———

    import Foo
    import Bar

    // Please ignore the ugly syntax, and just imagine something
    // prettier with these semantics:

    "Swift".Foo::flip() // → "tfiwS"
    "Swift".Bar::flip() // → "ʇɟıʍs"
    "Swift".flip() // compiler error: Ambiguous method "flip()"

    // Another imperfect alternative syntax (same semantic intent, though):

    ("Swift" as Foo.String).flip() // → "tfiwS"
    ("Swift" as Bar.String).flip() // → "ʇɟıʍs"
    "Swift".flip() // compiler error: Ambiguous method "flip()"

Both syntaxes are ugly, but set syntax aside for a moment. I’m asking about that general approach.

This disambiguation requirement would only apply if there is a pair of methods such that:

1. both methods are visible at the call site, and
2. neither one is an ancestor of the other in the same virtually dispatched override hierarchy.

Cheers,

Paul

···

On Dec 9, 2015, at 6:09 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

Any thoughts on instead having to mark the default method implementations as `default`?

There are three reasons I like `final`:

1) It echoes the meaning of `final` in a class—“this is the implementation and you cannot provide any other”.

2) Swift’s general slant is towards using virtual dispatch unless it’s forbidden. (Leaving aside cases where the optimizer can prove at compile time what a dynamic dispatch would do.) That’s why we have a `final` keyword instead of a `virtual` keyword. Because of this, it’s not surprising to Swift users that you *can* override some extension methods; rather, the surprise is that you *can’t* override others. (And the *real* surprise is that you can kind-of-but-not-really override them, and Swift doesn’t complain.) I want developers to mark the surprise.

3) Since, as I said, Swift’s slant is towards virtual dispatch, I think that if Swift eventually permits overriding of protocol extension methods, the overridable behavior should be the default, and the current static behavior should be something you request.

But this is not a terribly strong preference; `default` would be alright too. The more important point to me is that the user should acknowledge the weird behavior of shadowing a protocol extension method. In other words, I think the `@incoherent` annotation on the conformance is actually more important than the `final` or `default` keyword on the declaration.

It still seems strange to me to consider a protocol extension that uses the `final` keyword, because a) protocols have nothing to do with classes and `final` means no subclassing

The way I see it, `final` in a class is a way of saying “All instances of this class (including subclasses) must use this implementation and cannot substitute a different one”. Similarly, `final` in a protocol means “All instances conforming to this protocol must use this implementation and cannot substitute a different one”.

That’s why @incoherent is important. It’s basically saying “I know there are final members in this type and I can’t really override them, but I want to shadow them anyway, even though the result is going to be kind of half-assed.” You could even imagine it being applied to superclasses, allowing you to shadow their `final` members in the same way you can shadow protocol extension methods.

and b) methods defined in protocol extensions by definition can't be overridden already, so the `final` keyword is just sort of like saying "no really, I mean it, you can't do the thing you already can't do!”.

That is *exactly* the point of what I’m proposing. Just like the `override` keyword sort of says “no really, I mean it, I want to do the thing I’m doing!"

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution