Bridging the gap between protocols and protocol extensions

Evolution Discussion
1

A few questions about that:

1. Do people really want this?

2. People familiar with the implementation of protocol witnesses: is this feasible?

3. Do people want the current non-overridable, statically-dispatched protocol extension members available as an option?

4. Should this be proposed alongside other changes that would reduce the difference between a protocol and a protocol extension, like allowing default implementations in the protocol declaration?

5. Especially with #3 or #4, this sounds like a very large change which could be broken up into several smaller ones. Core team members: what is the best way to propose this? One big proposal with all the details, several small proposals, or a big vague proposal for the overall direction, followed by several smaller proposals with a detailed design for each part?

···

From my previous thread, "[Draft Proposal] Require `final` on protocol extension members", I'm getting the sense that many people want straight-up dynamic dispatch from protocol extension methods. That is, protocol extension methods should be treated as if they were included as requirements of the protocol; the implementations provided should be treated as overridable defaults.

--
Brent Royal-Gordon
Architechies

2

From my previous thread, "[Draft Proposal] Require `final` on protocol extension members", I'm getting the sense that many people want straight-up dynamic dispatch from protocol extension methods. That is, protocol extension methods should be treated as if they were included as requirements of the protocol; the implementations provided should be treated as overridable defaults.

A few questions about that:

1. Do people really want this?

I can’t speak for all people, but I don’t.

Protocol extensions which are not part of the protocol are not part of any explicit contract, and are not ‘opted into’ by the implementing types. Code in protocol extensions will break if they have interdependent methods which change behavior because I implemented a method with the same name but different behavior.

2. People familiar with the implementation of protocol witnesses: is this feasible?

3. Do people want the current non-overridable, statically-dispatched protocol extension members available as an option?

Yes.

4. Should this be proposed alongside other changes that would reduce the difference between a protocol and a protocol extension, like allowing default implementations in the protocol declaration?

Having default implementations inside the protocol would help illustrate a default implementation is available. Currently, it can be confusing what you do or do not need to implement in order to conform to a protocol.

-DW

···

On Jan 9, 2016, at 3:05 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

3

I do. There are lots of useful things you could do with dynamically dispatched protocol extension methods that you cannot easily do elegantly. As an example, I could declare this extension on ErrorType:

extension ErrorType {
  var isFileNotFoundError: Bool {
    // Check for all of the possible “File not found” errors in POSIXError, NSCocoaError, NSURLError, NSOSStatusErrorDomain, etc.
    // Then check for NSUnderlyingErrorKey and if it exists, rinse and repeat with that.
  }

  func toNSError() -> NSError {
    return self as NSError
  }
}

A method like this allows one to do things like this:

do {
  try NSFileManager.defaultManager().removeItemAtURL(someURL)
} catch {
  if error.isFileNotFoundError {
    // The file we were trying to delete didn’t exist. Ignore the error
  } else {
    // Okay, something actually went wrong
    NSApp.presentError(error.toNSError())
  }
}

Then, perhaps you define your own error type that has a FileNotFound error condition in it, and would like to override these methods:

enum MyErrorType: ErrorType {
  case FileNotFound(url: NSURL)
  // more cases

  var isFileNotFoundError: Bool {
    // Unfortunately, this will never get called.
    return case .FileNotFound(_) = self
  }

  func toNSError() -> NSError {
    // This won’t get called either. The user will see a lovely “MyErrorType error (Int)” instead of our localized error message.

    switch self {
      case let .FileNotFound(url: url):
        let userInfo = [NSLocalizedFailureReasonErrorKey : String(format: NSLocalizedString(“FNF %@“, comment: “String format: File Not Found error”), url.lastPathComponent)]
        return NSError(domain: “MyErrorType”, code: 1, userInfo: userInfo)
      // other cases
    }
  }
}

For any of the above to work as is, each and every error handler has to do an as? check against every custom error type in your project.

Charles

···

On Jan 9, 2016, at 4:05 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

From my previous thread, "[Draft Proposal] Require `final` on protocol extension members", I'm getting the sense that many people want straight-up dynamic dispatch from protocol extension methods. That is, protocol extension methods should be treated as if they were included as requirements of the protocol; the implementations provided should be treated as overridable defaults.

A few questions about that:

1. Do people really want this?

4

1. Do people really want this?

Yes, if for no other reason than that the current behavior (imho) violates
the principle of least surprise - the surprising part being that the
behavior is different than other method calls. Further, there is no way to
know at the call site what the behavior is going to be, hindering
readability.

2. People familiar with the implementation of protocol witnesses: is this
feasible?

I am not familiar with this so I can't comment specifically, but I would
just like to mention that (with apologies to whoever might eventually have
to implement this) the compiler is meant to serve the programmer, not the
other way around.

3. Do people want the current non-overridable, statically-dispatched
protocol extension members available as an option?

Sure, why not? I would prefer it not be the default behavior, though. It
should be triggered by some keyword (final, default, etc) in the protocol
extension.

4. Should this be proposed alongside other changes that would reduce the
difference between a protocol and a protocol extension, like allowing
default implementations in the protocol declaration?

I don't feel strongly one way or the other about this.

Thanks!
Andy

5

Please see my response inline.

-Rod

From my previous thread, "[Draft Proposal] Require `final` on protocol extension members", I'm getting the sense that many people want straight-up dynamic dispatch from protocol extension methods. That is, protocol extension methods should be treated as if they were included as requirements of the protocol; the implementations provided should be treated as overridable defaults.

A few questions about that:

1. Do people really want this?

Yes. I think it makes the most sense. While I appreciate the optimisation, I think the idea of protocols forcing an implementation on the type is backwards, and potentially dangerous when the protocol has no understanding of the potential repercussions on how the implementing type is implemented.

2. People familiar with the implementation of protocol witnesses: is this feasible?

Unfortunately I have no experience here.

3. Do people want the current non-overridable, statically-dispatched protocol extension members available as an option?

I think there might be a use case, but I debate the safety and usability of a protocol as such. When you’re implementing the protocol, as has been discussed above, you cannot be aware of all the ramifications of your code on another potential type. Whilst a “non-overridable, final” implementation would be be better for inlining, I question whether the benefit is worth the risks.

4. Should this be proposed alongside other changes that would reduce the difference between a protocol and a protocol extension, like allowing default implementations in the protocol declaration?

It appears like it should go through at a similar time frame, if not together. They are all consistent with a streamlining of the protocol/protocol extension design. How that is implemented in proposals, would be best left to the Core Swift Team’s guidance.

5. Especially with #3 or #4, this sounds like a very large change which could be broken up into several smaller ones. Core team members: what is the best way to propose this? One big proposal with all the details, several small proposals, or a big vague proposal for the overall direction, followed by several smaller proposals with a detailed design for each part?

N/A as I am not a core team member.

···

On 10 Jan 2016, at 9:05 AM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

6

I am not familiar with this so I can't comment specifically, but I would just like to mention that (with apologies to whoever might eventually have to implement this) the compiler is meant to serve the programmer, not the other way around.

Certainly, but the programmer will be ill-served if the compiler can only implement this with a very slow dispatch, or an expensive mandatory translation of protocol instances accessed across module lines, or with gaping semantic defects, or by dropping a dozen more-desirable features to implement it. And if it just can't be done, then it doesn't matter how well the compiler would serve the programmer by offering it.

···

--
Brent Royal-Gordon
Architechies

7

Forgive my ignorance if I am wrong, but without the objective-C runtime, we’re looking at dynamism from the perspective of vtables rather than from dynamic dispatch. I would assume the vtable approach would far closer approach vtable lookup than Dynamic dispatch. This appears to have no more overhead than standard protocols, and I would be very hazardous to call that “very slow dispatch”.

Protocol extensions that are final and not “default only" create 4 issues that I can see:

1. They lock a programmer into an implementation rather than an interface. This means if the implementations don’t perfectly match how you would perform the function (due to some internal state) then you must simply suffer and deal with it.

2. They create headaches for adhering to multiple protocols with the same method. This is currently being discussed on Swift Evolution as well.

3. They start to confuse what a protocol is - is it an implementation, or is it a guaranteed interface?

4. They seem to more closely model multiple inheritance, a model that was dismissed by the Swift team since Swift 1.0

···

On 10 Jan 2016, at 12:17 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

I am not familiar with this so I can't comment specifically, but I would just like to mention that (with apologies to whoever might eventually have to implement this) the compiler is meant to serve the programmer, not the other way around.

Certainly, but the programmer will be ill-served if the compiler can only implement this with a very slow dispatch, or an expensive mandatory translation of protocol instances accessed across module lines, or with gaping semantic defects, or by dropping a dozen more-desirable features to implement it. And if it just can't be done, then it doesn't matter how well the compiler would serve the programmer by offering it.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

8

Forgive my ignorance if I am wrong, but without the objective-C runtime, we’re looking at dynamism from the perspective of vtables rather than from dynamic dispatch. I would assume the vtable approach would far closer approach vtable lookup than Dynamic dispatch. This appears to have no more overhead than standard protocols, and I would be very hazardous to call that “very slow dispatch”.

That's a large part of what I mean when I ask if it's feasible. Can the protocol witness vtables be made extensible in some way? If they can, and a dynamic protocol extension call is no slower than any other protocol member call, that's great. If they can't, and protocol extension methods need a slower form of dispatch or some kind of translation when crossing module boundaries, that's probably not acceptable.

Class vtables can be extended in this way (and early in Swift's evolution, they even had overridability problems like protocol extensions now do), so I'm guessing it can be done. But protocols are in many ways more complicated than classes—particularly in their need to support multiple conformances rather than single inheritance—so I'm not willing to assume that it can be done without soliciting expert opinions.

···

--
Brent Royal-Gordon
Architechies

9

Forgive my ignorance if I am wrong, but without the objective-C runtime, we’re looking at dynamism from the perspective of vtables rather than from dynamic dispatch. I would assume the vtable approach would far closer approach vtable lookup than Dynamic dispatch. This appears to have no more overhead than standard protocols, and I would be very hazardous to call that “very slow dispatch”.

That's a large part of what I mean when I ask if it's feasible. Can the protocol witness vtables be made extensible in some way? If they can, and a dynamic protocol extension call is no slower than any other protocol member call, that's great. If they can't, and protocol extension methods need a slower form of dispatch or some kind of translation when crossing module boundaries, that's probably not acceptable.

If we make protocol extension methods dynamically dispatched (even as just an opt-in thing), we’d probably use a caching technique akin to objc_msgSend, keyed by the address of some global structure unique to the extension method. This has the huge advantage of not requiring eager resolution of every extension method on every conformance, and it can still be made very fast — on the order of a few dozen cycles, i.e. maybe 50-75% slower than a v-table dispatch. So it will not be as fast as a core requirement, but it’s likely to be close enough that all but the most performance-conscious programmers will be unaware of the difference, in the same way that most Java programmers have no need to be aware that dispatch through an interface type is usually slower than dispatch through a class type.

We would use a very similar approach for making out-of-module class extension methods dynamically dispatched.

John.

···

On Jan 9, 2016, at 7:37 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

Class vtables can be extended in this way (and early in Swift's evolution, they even had overridability problems like protocol extensions now do), so I'm guessing it can be done. But protocols are in many ways more complicated than classes—particularly in their need to support multiple conformances rather than single inheritance—so I'm not willing to assume that it can be done without soliciting expert opinions.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

10

If we make protocol extension methods dynamically dispatched (even as just an opt-in thing), we’d probably use a caching technique akin to objc_msgSend, keyed by the address of some global structure unique to the extension method. This has the huge advantage of not requiring eager resolution of every extension method on every conformance, and it can still be made very fast — on the order of a few dozen cycles, i.e. maybe 50-75% slower than a v-table dispatch. So it will not be as fast as a core requirement, but it’s likely to be close enough that all but the most performance-conscious programmers will be unaware of the difference, in the same way that most Java programmers have no need to be aware that dispatch through an interface type is usually slower than dispatch through a class type.

Thanks, John. So if I'm reading this right:

1. It is a significant engineering effort (you can't just rearrange things in the existing vtables), but definitely doable.

2. It would have reasonably good performance, but performance-sensitive code might want a statically-dispatched alternative.

Is that an accurate reading? Is it reasonable to propose for Swift 3, or would it require too much work for that timeframe?

···

--
Brent Royal-Gordon
Architechies

11

It’s something we need to figure out before committing permanently to an ABI, and that makes it a logical part of the Swift 3 effort to at least leave room for this.

John.

···

On Jan 10, 2016, at 5:21 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

If we make protocol extension methods dynamically dispatched (even as just an opt-in thing), we’d probably use a caching technique akin to objc_msgSend, keyed by the address of some global structure unique to the extension method. This has the huge advantage of not requiring eager resolution of every extension method on every conformance, and it can still be made very fast — on the order of a few dozen cycles, i.e. maybe 50-75% slower than a v-table dispatch. So it will not be as fast as a core requirement, but it’s likely to be close enough that all but the most performance-conscious programmers will be unaware of the difference, in the same way that most Java programmers have no need to be aware that dispatch through an interface type is usually slower than dispatch through a class type.

Thanks, John. So if I'm reading this right:

1. It is a significant engineering effort (you can't just rearrange things in the existing vtables), but definitely doable.

2. It would have reasonably good performance, but performance-sensitive code might want a statically-dispatched alternative.

Is that an accurate reading? Is it reasonable to propose for Swift 3, or would it require too much work for that timeframe?

12

I have put a proposal in for generic protocols and covariant generics:

  [swift-evolution] Make generics covariant and add generics to protocols <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006367.html&gt; <>

This proposal requires that protocol functions are dynamically dispatched, i.e. it would require this change.

···

On 11 Jan 2016, at 12:34 PM, John McCall via swift-evolution <swift-evolution@swift.org> wrote:

On Jan 10, 2016, at 5:21 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

If we make protocol extension methods dynamically dispatched (even as just an opt-in thing), we’d probably use a caching technique akin to objc_msgSend, keyed by the address of some global structure unique to the extension method. This has the huge advantage of not requiring eager resolution of every extension method on every conformance, and it can still be made very fast — on the order of a few dozen cycles, i.e. maybe 50-75% slower than a v-table dispatch. So it will not be as fast as a core requirement, but it’s likely to be close enough that all but the most performance-conscious programmers will be unaware of the difference, in the same way that most Java programmers have no need to be aware that dispatch through an interface type is usually slower than dispatch through a class type.

Thanks, John. So if I'm reading this right:

1. It is a significant engineering effort (you can't just rearrange things in the existing vtables), but definitely doable.

2. It would have reasonably good performance, but performance-sensitive code might want a statically-dispatched alternative.

Is that an accurate reading? Is it reasonable to propose for Swift 3, or would it require too much work for that timeframe?

It’s something we need to figure out before committing permanently to an ABI, and that makes it a logical part of the Swift 3 effort to at least leave room for this.

John.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

13

I have put a proposal in for generic protocols and covariant generics:

  [swift-evolution] Make generics covariant and add generics to protocols

This proposal requires that protocol functions are dynamically dispatched, i.e. it would require this change.

What you propose is a fairly serious redesign of how protocols handle related types; I'm trying to focus on changes that hopefully are relatively uncontroversial and mostly straightforwardly extend what currently exists.

(Frankly, I'm also not in favor of replacing associated types, because I think they're a far better fit for the task than generics. But I don't want to derail this thread into a critique of your proposal.)

···

--
Brent Royal-Gordon
Architechies

14

@Brent,

I don't want to derail this thread either.

If you want to post on the other thread your concerns I am happy to take a
look.

Cheers,

-- Howard.

···

On 13 January 2016 at 17:42, Brent Royal-Gordon <brent@architechies.com> wrote:

> I have put a proposal in for generic protocols and covariant generics:
>
> [swift-evolution] Make generics covariant and add generics to
protocols
>
> This proposal requires that protocol functions are dynamically
dispatched, i.e. it would require this change.

What you propose is a fairly serious redesign of how protocols handle
related types; I'm trying to focus on changes that hopefully are relatively
uncontroversial and mostly straightforwardly extend what currently exists.

(Frankly, I'm also not in favor of replacing associated types, because I
think they're a far better fit for the task than generics. But I don't want
to derail this thread into a critique of your proposal.)

--
Brent Royal-Gordon
Architechies

--
  -- Howard.