[Proposal Draft] automatic protocol forwarding

Evolution Discussion
41

Sent from my iPad

-Dave

I don’t want this thread to get distracted with memberwise initialization

Makes sense in general, but Kotlin solves those problems as a whole, and the major benefit of their approach is that everything fits together really fine.
But I'll skip everything that is not related to forwarding.

One approach I considered would look like this:

class Forwarder: P {
   // I didn’t like `implements` but didn’t have a better idea before I abandoned this approach
   var forwardee: Forwardee implements P
}

Honestly, I'm can't see the value this discarded variant has for this discussion… you have to pit your proposal against Kotlin if you really want to convince anyone of its superiority.

With the memberwise initialization proposal you also have the initializer synthesized automatically. The only thing it doesn’t do that your Kotlin example does is automatically declare conformance. This was an intentional design decision because it allows for additional expressivity. This is addressed in the alternatives considered section of the proposal.

Can you be more precise? Kotlin clearly states what a class is doing in its first line, with all expressivity that is necessary by practical means.

What I mean is this. In the example you gave and the syntax Kotlin uses:

classs Forwarder(forwardee: Forwardee): P by forwardee {}

Forwarding is coupled to protocol conformance. This means I cannot use forwarding without conforming to the protocol that is forwarded.

Here is a quick example using the syntax of my proposal to demonstrate the difference:

class Forwarder {
  let forwardee: Forwardee
  forward P to forwardee
}

vs

class Forwarder: P {
  let forwardee: Forwardee
  forward P to forwardee
}

In the first example Forwarder does not conform to P. Forwarding is only used to synthesize the members of P. I am greatly expanding the motivation section of the proposal and will have examples showing where this is what you want. The lazy collections section I posted last night includes the first examples where this is the case.

In the second example here Forwarder does conform to P. The author of Forwarder has the flexibility to specify whether conformance is desired or not.

There are ways to handle that, including factoring the APIs of interest out of P and into a private protocol Q, then declaring the Forwardee’s conformance to Q. Now, there’s an expressivity problem with our current access control system that you can’t use an internal or private protocol to provide public API, but that should be fixed separately.

I'm not sure where Q comes into play in this specific example. The idea here is that forwarding implementations of all members of P are synthesized by the forward declaration. It is left up to Forwarder to decide whether or not to declare actual conformance to P. I am also confused by "then declaring the Forwardee’s conformance to Q" because we are discussing Forwarder's conformance here, not Forwardee’s.

Presumably both ends of the forwarding arrangement would have to conform to the same protocol, no?

No. This is addressed in the proposal and the lazy collections motivating example I replied with last night. I don’t think it’s a good idea to require this.

The forwardee needs to implement the members of the protocol but does not need to conform. The forwarder will receive forwarding implementations of the members, but again does not need to declare conformance. Forwarding is orthogonal to conformance, just as it is today when you manually write forwarding members today.

What do you have in mind when you mention using a private or internal protocol to provide public API? It sounds like that might be interesting but I'm having trouble imagining what the syntax would look like and exactly how it would work. Is this something that is planned?

Not planned, but desired.

What might it look like?

Details need to be worked out. One thing we were doing for a while in the stdlib, before the rules got tightened and made it impossible, was

struct X : PublicProtocol, PrivateProtocol {
   ...
}

extension PublicProtocol where Self : PrivateProtocol {
   // API that uses only PublicProtocol in its implementation here
}

Presumably you implement public API here where members of PrivateProtocol can be accessed? This would address the problem of leaking implementation details when adding default forwarding implementations so it is definitely better than current state.

In any case, I don't see why that is related to requiring a Forwarder to conform to the forwarded protocol. There doesn't appear to me to be a good reason to require that and there are reasons not to require it. Protocols enable and drive the forwarding member synthesis mechanism but that mechanism doesn't need to require or provide conformance. It is a third major way to use protocols in addition to generic constraints and existential types.

Of course I could be wrong, but my instincts tell me that is an unneeded dimension of complexity, which is why I am resisting it. The generic/existential duality is already problematic in some ways, IMO.

I don’t really understand what is complex about it. Can you elaborate on why you think it introduces complexity? The idea seems pretty simple to me - if you can write a forwarding implementation that compiles and works the compiler should be able to synthesize it for you. I don’t see any reason to disallow that.

Is your concern that we may not be able to support forwarding of every protocol someone might come up with and there would be confusion around what protocols are eligible for forwarding and which aren’t (similar to the current confusion around which protocols can be used as existentials and which can’t)?

No, it just feels like an ad-hoc use of protocols to use their APIs as constraints without creating a conformance.

I can understand this perspective, but I think there is a fundamental difference between protocols as generic constraints or existential where conformance is required by the language model and using protocols to drive forwarding. The forwarding mechanism is really just syntactic sugar for forwarding members that can be written manually without any need for conformance. If the manually written code doesn't require conformance I don't think the syntactic sugar should.

Requiring conformance would artificially restrict the power of the feature in the language as it exists today. A possible future language where that is not the case is speculative and does not help us in the meantime.

I will continue to work on motivating examples, several of which will take advantage of this relaxed requirement.

···

Sent from my iPad

On Dec 31, 2015, at 11:52 AM, Dave Abrahams <dabrahams@apple.com> wrote:

On Dec 31, 2015, at 9:47 AM, Matthew Johnson <matthew@anandabits.com> wrote:

On Dec 31, 2015, at 11:18 AM, Dave Abrahams <dabrahams@apple.com> wrote:
On Dec 31, 2015, at 9:01 AM, Matthew Johnson <matthew@anandabits.com> wrote:

On Dec 31, 2015, at 10:09 AM, Dave Abrahams <dabrahams@apple.com> wrote:

On Dec 31, 2015, at 7:33 AM, Matthew Johnson via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 31, 2015, at 5:04 AM, Tino Heth <2th@gmx.de> wrote:

Isn’t the primary problem with generic / existential the fact that existentials are pretty limited currently? This is an unfortunate limitation and my understanding is that there is a desire to lift at least some parts of this limitation eventually, whether that starts to happen in Swift 3 or is a feature that comes later. [swift-evolution] Swift 3 Generics

Yes, there are plans to address this (and I’m looking forward to that!), but I think the difference is still going to be there and when to use one or the other is still going to remain a point of confusion.

One specific problem with requiring conformance is that forwarding to existentials would not be straightforward because they do not conform to their protocol. Obviously this is a limitation that should be lifted eventually but it isn’t clear when that might happen.

Another difference that is maybe subtle but I think important is that with the approach I considered forwarding is declared in the body of a type or extension which emphasizes the fact that forwarding is an implementation detail, not something users of the type should be concerned with.

But what is the benefit of this emphasis? No solution requires to make the details visible in the public interface, and the ability to bury an important thing like protocol conformance somewhere in the class implementation is no advantage for me.

Protocol conformance is not buried in the implementation in my solution. I hope the previous example makes that clear. What is buried in the implementation is the forwarding declaration which causes the compiler to synthesize forwarding member implementations. This synthesis is an implementation detail and should not be visible outside the implementation.

This approach was abandoned as it leads to problems in expressivity and clarity. Please see alternatives considered for an elaboration of that. This is especially true with the new approach to handling Self values that Brent suggested. That approach requires additional syntax around the forwarding declaration, but adds both clarity and expressiveness.

I think there is little need to worry about expressiveness for a feature that most potential users will probably never utilize in real code — and I don't think options like not conforming to a protocol that is forwarded is a big win here. It looks to me like you are optimizing for very uncommon cases, and sacrificing ease of use in the situations that are the most common by far.

Like I stated, I am working on adding several examples of how this feature can be used in real code. Please have a look at the lazy collections example I shared last night. This example, as well as at least one other coming examples take advantage of the ability to use forwarding without requiring conformance.

As with the memberwise initialization proposal, the syntax you would like to see can easily be added as syntactic sugar on top of the current proposal. I would not support that as I do not like the syntax Kotlin uses for reasons already stated, but that shouldn’t stop you from pursuing a proposal for it. Maybe a lot of people would agree with you and it would be accepted.

Matthew

It appears to me that you value conciseness very highly. I do value conciseness but I also value safety, clarity, and expressiveness.

No, I value elegance and simplicity — they often lead to clarity and safety.

Why not simply make this feature syntactic sugar and just auto-generate the forwarding methods?

That is exactly what this proposal does. Why do you feel it is not doing that?

you're right, guess I mixed up the proposal with something else; so at least we agree on how it should work ;-)

I'm not saying Swift has to copy another language, but I doubt that anyone who knows Kotlin would actually consider to drop their solution in favor of what is currently discussed...

Tino

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

-Dave

-Dave

42

You are right, I included it in that section. It is definitely something I want to see supported eventually. I just don’t want it to hold up the initial proposal.

I also want to make sure any subtleties have been thoroughly considered and addressed before it goes forward. I understand that it is in some sense similar in some sense to protocol methods with default implementations. However I think it is actually closer to inheritance (this proposal can in some respects be considered a limited and controlled form of multiple inheritance).

The author of a protocol understands and intends the default implementations to be just that - defaults. The author of the type that is forwarded to may have expectations that it receives all calls for all members of a protocol it implements and the implementation may not work correctly or as expected if that does not happen. IMO, it is important to think about the implications of partial forwarding, limitations that may be necessary, etc.

Despite my caution around partial forwarding, there is one reason that it might make sense to support partial forwarding in the initial proposal. Specifically, it would be easy to get around the limitation by simply declaring a new protocol that contains only the members you want to forward and use that in your forwarding declaration. And of course you can also write manual forwarding code. So a limitation of the automatic forwarding mechanism wouldn’t stop you from doing partial forwarding, it would just make it more annoying. It might be better to “complete” the design from the start for this reason. I will definitely give this further thought.

Do you have specific use cases in mind for partial forwarding? If so, please share them.

Matthew

···

On Dec 29, 2015, at 5:45 PM, Charles Srstka <cocoadev@charlessoft.com> wrote:

On Dec 29, 2015, at 5:38 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Dec 29, 2015, at 5:25 PM, Charles Srstka <cocoadev@charlessoft.com <mailto:cocoadev@charlessoft.com>> wrote:

Strong +1 on this proposal. I use Objective-C’s forwarding mechanisms quite often in my custom view code, in order to separate the code managing the outer view, the layout of subviews within the view, and business logic into separate classes, all while presenting a single, monolithic interface to the user. The loss of this ability without writing tons of boilerplate is one of the things about Swift that makes me sad.

The one thing I’d change is upgrading the partial forwarding synthesis to the original proposal, as that’s a rather important feature IMO.

Thanks Charles. Do you disagree with the reasons I decided not to include partial forwarding in the initial proposal (it is discussed in alternatives considered)?

I see it in Future Enhancements, but not in Alternatives Considered (unless I’m just missing it). However, my thinking is that this isn’t really that hard to implement; just have the forwarding mechanism forward only methods that aren’t already implemented in the class. This would be similar to the way protocol extensions work, where if you implement the method you get that, but if you don’t implement it, you get the protocol extension’s implementation.

43

* Does it have to be a protocol? Why not also allow the concrete type of the property you're forwarding to? Obviously you couldn't form a subtype relationship (unless you could...), but this might be useful to reduce boilerplate when you're proxying something.

This is addressed in the alternatives considered section.

Sorry, I missed that, probably because the sample code in that section didn't show such a forwarding.

The short answer is that there the direct interface of the concrete type does not contain sufficient information about potential Self parameters to do this well. This information does exist in the protocol declarations. Allowing this information to be specified in concrete interfaces would add enough complexity to the language that I don’t think it is worthwhile.

That's a good point. You could perhaps add a way to tweak the forwarding of certain members, but that'd be a little tricky.

I gave this enough consideration to be leaning pretty strongly in the direction that protocols are the best way to do this.

Doing this properly for the concrete interface would require an `InvariantSelf` type that could be used in any method signature. It would also need to be actually used correctly in practice by types that were forwarded to. The distinction is pretty subtle when you’re dealing with a concrete interface and my instinct is that people would get it wrong a lot of the time.

Because protocols are inherently generic it is a little more straightforward to think about when you mean `Self` and when you mean a concrete type.

The good news is that you can declare a protocol containing the full interface of a concrete type if you really want or need to and use that for forwarding. The advantage of requiring a protocol here is that it requires you to consciously think about whether each parameter and return type is intended to be concrete or an abstract Self. It also allows you to properly forward an interface even if the original author did not consider these issues when implementing the type.

In the following example, should the other parameter and the return type be `Self ` or `Double`? It is not possible to know unless there is a protocol that declares foo.

extension Double {
    func foo(other: Double) -> Double {
        return self
    }
}

One of the things I'd like to see is the ability to proxy for an instance without the person writing the proxy knowing which instances it'll be used with. Think, for example, of the Cocoa animator proxy, or `NSUndoManager.prepareWithInvocationTarget(_:)`. It'd be nice if a Swift equivalent could return, say, `NSAnimatorProxy<View>` or `NSUndoManager.InvocationTarget<Target>`, which has all the methods of the generic type but records the calls for later use.

Of course, what you really want is for only a particular subset of the methods to be available on the proxy (animated methods on `NSAnimatorProxy`, Void methods on `NSUndoManager.InvocationTarget`), and of course in these cases you're not calling directly through to the underlying methods. So I might just be barking up the wrong tree here.

I can see how it might be desirable to forward to a type you receive as a generic parameter. However, you would need to constrain that type to a protocol(s) in order to actually do anything useful with it. That same protocol(s) could also be used in the forwarding declaration.

If you want is to be able to forward a set of methods that is determined by the generic parameter, that just isn’t going to be possible. At least not without significant changes to other parts of the language providing capabilities that would allow you to implement something like that manually.

* Why the method-based conversion syntax for return values, rather than something a little more like a property declaration?

  var number: Int
  forward IntegerType to number {
    static return(newValue: Int) {
      return NumberWrapper(newValue)
    }
    return(newValue: Int) {
      return NumberWrapper(newValue)
    }
  }

This is actually a really good idea to consider! I didn’t consider something like this mostly because I didn’t think of it. I’m going to seriously consider adopting an approach along these lines.

Great.

One possible advantage of the approach I used is that the initializer may already exist for other reasons and you would not need to do any extra work.

True. But it may also exist and *not* do what you want in the forwarding case. It's easier to explicitly use the right initializer than it is to work around the forwarding system implicitly using the wrong one.

Right, I am generally leaning pretty strongly towards changing the proposal to use a solution similar to what you suggest.

A big advantage of this approach is that it would work even when you are forwarding different protocols to more than one member with the same type.

But again, if that's the wrong behavior, there's no good way to fix it.

I was actually indicating an advantage of the approach you suggested because it provides a solution to that problem. :)

* If you want to keep the method-based syntax, would it make sense to instead have an initializer for instance initializers too, and just have it take a second parameter with the instance?

  init(forwardedReturnValue: Int) {...}
  init(forwardedReturnValue: Int, from: NumberWrapper) {…}

Part of the reason the instance method was used is because sometimes the right thing to do might be to mutate and then return self. Using an instance method gives you the flexibility to do that if necessary.

In practice, I'm not sure that's actually the case very often. How frequently will the internal type return a changed value, but your identically-named cover method ought to mutate the property? That completely changes the semantics of the underlying call.

I would guess you are right about this being something that is never or almost never the right thing to do. Maybe a solution that actually prevents you from doing this would be a better one for that reason.

I mean, what you're proposing would be something like this:

  class ListOfThings {
    private var actualList: [Thing]
    
    func filter(predicate: Thing -> Bool) -> ListOfThings {
      let returnValue = actualList.filter(predicate)
      actualList = returnValue
      return ListOfThings(returnValue)
    }
  }

Is that a thing you actually expect people to do?

I hope not! :) I hadn’t looked too hard for an example where it would be the right thing to do and I think you are right that such an example would be pretty hard to come by. Thanks for pushing back on this one! :)

* Does this mean that a `public forward` declaration would forward `internal` members through synthesized `public` interfaces, if the forwarder and forwardee happened to be in the same module?

All synthesized members recieve access control modifiers matching the access control modifier applied to the forward declaration.

Yes, if the forwardee had internal visibility and the forwarder was public the forwarder could publicly forward the interface. This is intentional behavior. The forwardee may well be an internal implementation detail while the methods of the protocol are part of the public interface of the forwarder. It is possible to write code that does this manually today.

I suppose that, if it's always a protocol you're forwarding to, you can assume that none of the protocol methods are internal-only implementation details. But I have to admit that I'm still concerned about this; it just seems like a recipe for accidentally exposing things you meant to keep private.

Think of a state or strategy pattern. You might have an internal protocol with several implementations representing the various states or strategies. You forward to a private member that is an existential of the protocol type. The synthesized forwarding methods represent the public interface. However, the protocol itself is not public as it is an implementation detail and the forwarder doesn’t even conform to the protocol. The protocol is strictly used as an implementation detail of the public interface of your type.

I honestly don’t understand the concern about this proposal accidentally exposing things you meant to keep private. You have full control over access control of everything, including the synthesized methods. The proposal doesn’t do anything you couldn’t already do manually and it makes it very easy to keep them private if desired. If somebody wanted to manually write a forwarder that declares forwarding methods public while forwarding to internal methods on the forwardee they can do that already today!

IMO the language needs to provide the tools to specify your intent clearly and easily and offer sensible defaults, but it can’t do more than that. I believe this proposal falls in line with that principle and also inline with how access control already works in the language today (the default for synthesized members matches the default for manual member declarations).

I am very concerned about not exposing details that shouldn’t be exposed and that is one of the big drawbacks of the approach to protocol forwarding that Kevin Ballard shared.

* You don't explicitly mention this, but I assume mutating methods work and mutate `self`?

Mutating methods are something I didn’t think about carefully yet. Thanks for pointing that out! But generally, yes a forwarding implementation of a mutating method would need to mutate the forwardee which is part of self, thus mutating self.

Well, as long as they're thought about at some point!

Yes, I’m glad you brought them up!

···

On Dec 29, 2015, at 6:10 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

--
Brent Royal-Gordon
Architechies

44

Looking forward to it!

-Dave

···

On Dec 31, 2015, at 11:42 AM, Matthew Johnson <matthew@anandabits.com> wrote:

I will continue to work on motivating examples, several of which will take advantage of this relaxed requirement.