Introduce flattened unapplied method references (SE-0042 revisited)

[marcusrossel]

Introduction

In SE-0042 @Joe_Groff introduced the idea of flattening unapplied method references (UMRs), from the curried form to a function with an additional Self parameter (the details can be found in the proposal - it's a quick and easy read). This proposal was accepted, yet never implemented.

---

A benefit resulting from this change would be a massive improvement when passing around methods as function parameters. As Joe Groff himself puts it:

For instance, though you can pass the global + operator readily to reduce to sum a sequence of numbers:

func sumOfInts(ints: [Int]) -> Int {
  return ints.reduce(0, combine: +)
}

you can't do the same with a binary method, such as Set.union:

func unionOfSets<T>(sets: [Set<T>]) -> Set<T> {
  // Error: `combine` expects (Set<T>, Set<T>) -> Set<T>, but
  // `Set.union` has type (Set<T>) -> (Set<T>) -> Set<T>
  return sets.reduce([], combine: Set.union)
}

---

Changing UMRs to be flat would fix this problem, as Set.union would then produce a function of type (Set<T>, Set<T>) -> Set<T>. So if you've ever written a closure that contains exactly one statement which is just the call to a method... this would not be necessary anymore.

The Problem

As @Slava_Pestov has pointed out in a forum post, changing the curried form to a flattened form would require quite a large amount of work, as the compiler depends on the curried form. Also, there would have to be some sort of backwards compatibility mechanism to avoid breaking source compatibility.
So it would appear that implementing the change as proposed in SE-0042, would not yield a great return on investment anymore.

Just to add a personal opinion, I also find the curried function to be quite a natural result from an UMR. It basically conveys "I want to have a reference to this method, without having chosen upon which instance of Self it acts."
Consequently, passing a specific instance of Self to this "unbound" method enacts the step of conveying "I have chosen the instance upon which the method should act."

The Solution?

According to the aforementioned forum post, Joe Groff has put forth the idea of using key path syntax. You would write \MyType.method to form an uncurried reference.
This would solve the two problems:

1. (I'm assuming) No massive compiler changes would be required.
2. Source compatibility would not be an issue, as this would be an additive change.

This syntax would also seem to integrate well into the current concept of key paths. Key paths can be used to reference properties in a way that was not possible before, to allow for easier access. Flattened UMRs via key paths would do the same, just for methods.
On the other hand it might be unexpected to users that a key path expression would then produce a function type, instead of the usual KeyPath.

As @MutatingFunk mentioned, there is also the problem that mutating methods lead to undefined behavior, when using the current UMRs. Which entails that simply adding a new syntax for the flattened version would not be enough. The current UMRs also need to be fixed.

---

This post is just a first step to gauge interest in such a feature. If you see any complications that may arise from this proposal, or have a better idea on how to realize this feature, please share your thoughts in the comments.
As an average Swift user I would also very much welcome any knowledge on what would actually be required to implement this kind of feature.

Thanks,
Marcus

[MutatingFunk (James Froggatt)]

From the proposal:

This currying is also incompatible with mutating methods due to the semantics of inout parameters. In a chained call such as f(&x)(y), the mutation window for x only lasts as long as the first call. The second application of y is no longer allowed to mutate x. We currently miscompile unapplied references to mutating methods, capturing a dangling pointer when the reference is partially applied and leading to undefined behavior when the full application occurs.

Should we at least aim to remove the current feature for now, even if we don't have a replacement ready?

I'd rather we free up this syntax and reuse it for the flattened form, than overload the KeyPath syntax. There are many use-cases where the flattened form would be useful, and it's my understanding (though I don't have a source) that key paths are moving more in the direction of resembling currying:

\UIView.subviews[0].superview //Planned subscript keypaths?
\[Int].remove(at: 0).magnitude //The logical extension to methods

Either way, I'm very much eager to see progress in this area. Both the flattened form and the above reverse-curried form would be useful in forEach-type methods, while the current curried form doesn't have much in favour of it.

[marcusrossel]

I'm not sure that removing the current version of unapplied method references (UMR), without providing a replacement, is the way to go. There is still value in the feature, and I for one know that I have used it. Of course any occurrence of a UMR could be replaced with a closure though.

The problem arising from mutating methods kind of removes the option of simply introducing a new syntax, while also keeping the current one for the curried form. Undefined behavior isn't really anything we want to ignore.

Implementing the feature as proposed in SE-0042 therefore seems like the cleanest option. It would remove the undefined behavior, provide the flattened function type, and avoid new syntax.
That brings us back to the problems of source compatibility though.

Perhaps one could at least raise a warning, when a UMR is used on a mutating method, and later on make it a hard error.

[DevAndArtist (Adrian Meister)]

To be honest regardless the source compatibility I'd rather would want the original accepted version despite the complexity it requires to be implemented instead of hacking it into key paths. Don't get me wrong, but we had a full evolution process for this one already and agreed on the best solution. It makes me sad if we're starting to come up with different solutions that are good enough, just because we had no implementation resources available to this day. But who am I to speak like that about the complexity and implementation resources.

The standard library was designed with a lot of missing generic features in mind until they were implemented after years. Labels for function types were removed from the language to simplify the typesystem and to restore the ergonomics of the lost labels for closures we kept the door for compound names open. In that sense I wish for the best accepted solution or an alternative rotational solution for now which will provide a warning and a fix-it for a temporary workaround in Swift 4.2 and make it an error in Swift 5 until we can introduce SE-0042 as it meant to be.

[Paul_Cantrell]

I’ve wanted one or the other of these features — either 0042 or method keypaths — for the purpose of not strongly retaining the received as foo.someMethod does. (I can provide the context if it’s useful.)

Where I’ve wanted it, leaving the type on the left of the dot implicit would be an important part of the ergonomics of the feature. I assume from the current behavior of keypaths that extending them to methods would answer my wish:

// Method keypaths with implicit type left of the dot
// (assuming here that reduce already has a variant that takes keyPaths)
func unionOfSets<T>(sets: [Set<T>]) -> Set<T> {
  return sets.reduce([], \.union)  // instead of \Set.union (ha, italic backslashes look funny)
}

Would an implementation of 0042 allow the same? Would there be yet more serious implementation hurdles to make it so?

// SE-0042 with implicit type left of the dot?
func unionOfSets<T>(sets: [Set<T>]) -> Set<T> {
  return sets.reduce([], .union)  // instead of Set.union
}

[anandabits (Matthew Johnson)]

I have also been eagerly awaiting the implementation of 0042. IIRC there was some discussion around the time of that proposal of enhancing the dot shorthand syntax in the direction you're asking about @Paul_Cantrell.

However, that discussion was prior to key paths. I suspect that at this point it makes most sense to move in the direction of key path syntax for all unapplied member references. I would enthusiastically support a revised 0042 that does while deprecating the existing unapplied method reference syntax.

[Paul_Cantrell]

@anandabits

I would enthusiastically support a revised 0042 that does while deprecating the existing unapplied method reference syntax.

Yes, likewise. If as discussed on other threads we had either a “keypath to function” operator:

names.map(*\.uppercased)  // `*` is a placeholder

…or implicit keypath → closure promotion:

names.map(\.uppercased)

…then the two approach just about identical. I would enthusiastically support either if the ergonomics worked out. I defer to those who would have to implement it on which one is better for the health of the language.

[anandabits (Matthew Johnson)]

@Paul_Cantrell

…or implicit keypath → closure promotion:

names.map(.uppercased)

Right, I should have mentioned this as well - thanks for bringing it up. Some kind of implicit promotion is essential to making this approach workable. A "key path to function" operator would already require special compiler support (at least in the absence of variadic generics) and would add noise without increasing the clarity of usage sites.

As is often the case, I believe @Joe_Groff had the idea that made the most sense to me. Unfortunately I don't recall the details or where I saw it.

[Paul_Cantrell]

@anandabits

A “key path to function” operator would already require special compiler support (at least in the absence of variadic generics) and would add noise without increasing the clarity of usage sites.

How so? This works today:

prefix operator *

prefix func *<T,U>(_ keyPath: KeyPath<T,U>) -> (T) -> U {
    return { $0[keyPath: keyPath] }
}

["foo","bar"].map(*\.capitalized)

…and the operator defined above does still work if U is a function type:

extension String {
    var appendingAsProperty: (String) -> String {
        return self.appending
    }
}
["foo","bar"].map(*\.appendingAsProperty).map { $0("bar") }

… so presumably if the method key path \String.appending were allowed, the operator above would handle it too.

Is there a thornier case that this misses?

---

I tend to prefer implicit promotion too as long as it doesn’t wreak havoc. But if it does, the operator seems reasonable enough. That leaves a nice menu of options to scratch the itch of this thread.

[Karl]

Personally, I don't really like currying in Swift.

If we introduce a new unapplied method thing, I would really like full partial application.

[anandabits (Matthew Johnson)]

This example is not at all what I would expect. Remember that the central thrust of 0042 is to flatten the signature of unapplied method references. That means the signature would not be (String) -> (String) - String as in the above example, but would instead be (String, String) -> String. Also keep in mind that this is not the same as a function that takes a single tuple argument ((String, String)) -> String.

With this in mind, read-only key paths* are roughly equivalent to methods that do not accept any arguments (other than self of course!). If the same syntax is used to create unapplied method references in the future it will need to be capable of creating values that have "holes" for arguments other than self. In fact, to be a fully general feature it would need to be capable of reflecting the complete feature set supported by argument lists.

We can leave most of that complexity aside in answering your question about variadic generics. The operator would need to support an arbitrary number of arguments to the function which were determined by the "key path" representing an unbound method. The best we could do without compiler support would be something along the lines of a family of unbound method types UnboundMethod0<S, R>, UnboundMethod1<S, R, A1>, UnboundMethod2<S, R, A1, A2>.

However, this would not properly account for the annotations that can be applied to parameters. Following the syntax of key paths for unbound method references makes a lot of sense to me as it unifies the syntax for unbound member references. Identifying the right design for the representation of unbound method references is much more difficult. I'm curious if @Joe_Groff has given much thought to what this would look like.

• Note: Swift does not support returning a writable storage location from a method so we can ignore writable key paths in the context of the present discussion.

[marcusrossel]

@DevAndArtist I checked the mailing list archives (2016.03.14, 2016.03.21) and it seems that there hasn't really been much discussion on the topic. It seems Joe Groff put forth the proposal and it was accepted as is. So perhaps now would be the time to have the community reevaluate the feature, also taking into account - as you mentioned - the changes that have been made to Swift since then.

---

If I understand @anandabits correctly, variadic generics would be one possible solution to implement the flattened UMRs via actual KeyPaths (with some operator that converts them to a function type), as they would work around the "problem" of not having tuple splat.
As with current tuple-equality functions, I would guess we could work around this limitation using gyb-files, until variadic generics are introduced. This would also have the benefit of not causing source compatibility issues, once variadic generics arrive.

A remaining problem is:

However, this would not properly account for the annotations that can be applied to parameters.

---

TL;DR;

People seem to be happy with the proposed syntax. The part we still have to figure out is the representation which this syntax would produce. (@Joe_Groff, @Slava_Pestov )

[Paul_Cantrell]

@anandabits

With this in mind, read-only key paths* are roughly equivalent to methods that do not accept any arguments (other than self of course!). If the same syntax is used to create unapplied method references in the future it will need to be capable of creating values that have “holes” for arguments other than self. In fact, to be a fully general feature it would need to be capable of reflecting the complete feature set supported by argument lists.

Ah, I’m imagining something much simpler. It should be possible to get methods with key paths exactly as one can with the current foo.method and foo.method(arg:arg:) syntaxes. This parallel structure should hold:

let words = ["foo", "bar"]

words                .count   // Using a property directly...
words[keyPath: \Array.count]  // ...is equivalent to applying it as a key path.

words                .prefix(upTo:) (1)  // The same pattern with methods...
words[keyPath: \Array.prefix(upTo:)](1)  // ...should also hold for key paths.

In order words, this key path:

\Array<String>.prefix(upTo:)

…should have this type:

KeyPath<Array<String>, (Int) -> ArraySlice<String>>

What I’m describing there is completely orthogonal to 0042. We don’t have to worry about unbound method refs, because the method’s receiver (array, in this case) is baked into the first type param of the key path. The parallel structure would be between method key paths and bound method refs.

Having key paths support this is thus an alternative to using unbound methods; it could exist independently alongside 0042 — or replace it entirely for my purposes.

---

Given all that, I’m not sure I understand the feature you have in mind. Reading this:

@anandabits

If the same syntax is used to create unapplied method references in the future it will need to be capable of creating values that have “holes” for arguments other than self.

…it sounds here like you are imagining … some sort of partial application? Could you elaborate on what you’re imagining, perhaps with an example?

[anandabits (Matthew Johnson)]

We are definitely thinking about this differently.

Your example reminds me of previous discussion of allowing method calls in key paths such that arguments would be bound by the key path (exactly as subscript arguments are). In that possible future Swift would support something like words[keyPath: \Array.prefix(upTo: 1)]. This is different than what both of us have been discussing.

Regardless of how a final design relates to (or doesn't relate to) key paths, I strongly desire improved support for unbound method refs along the lines originally proposed by 0042. The change was very well motivated.

I think a good example use case to consider is the new overload of reduce introduced by 0171 that takes the accumulator as inout. It would be really nice to be able to pass an unbound method reference as the combine argument without having to manually uncurry first.

More generally, the line of argument is that in Swift we usually prefer to write methods over standalone functions. However, it is very rare to see higher-order functions that accept function arguments in curried form. This introduces an unfortunate and significant impedance mismatch that 0042 would eliminate.

@Paul_Cantrell

…it sounds here like you are imagining … some sort of partial application? Could you elaborate on what you’re imagining, perhaps with an example?

Sorry, I am not really imagining anything specific as it isn't clear to me what the right design would be. I can say that it is definitely not some sort of partial application as I am interested in exposing uncurried, unbound methods directly with ergonomic syntax.

[Slava_Pestov]

@marcusrossel

According to the aforementioned forum post, Joe Groff has put forth the idea of using key path syntax. You would write \MyType.method to form an uncurried reference.

This would solve the two problems:

(I’m assuming) No massive compiler changes would be required.
Source compatibility would not be an issue, as this would be an additive change.

The compiler uses curried method types internally quite a fair bit. All method declarations have a type of the form (Self) -> (Args...) -> Result. In the AST, a call like base.method(args) is modeled as ``(apply (apply method base) args)`.

If SE-0042 is ever implemented in any form, we should do the full refactoring to the AST to change both the type of methods and the expression nodes for a method application. The backward compatibility can be implemented with a special thunk. The idea would be then we'd be able to rip out the old curried type logic completely once the backward compatibility is not needed anymore. Otherwise, having curried method types internally but not exposed in a user-visible form would constitute a huge blob of technical debt.

[marcusrossel]

If the changes that @Slava_Pestov described should/need to be implemented anyway, it seems that @Joe_Groff's original SE-0042 still precisely portrays what should happen.

Repitching the proposal therefore seems a bit redundant.
Implementation of the old proposal should be sufficient.

[Joe_Groff]

We have strict source compatibility constraints now, and since SE-42 was not implemented as proposed originally and the Swift 2 behavior was never removed, we can't accept the proposal as is anymore. I would support a re-pitched proposal to provide the behavior SE-42 proposed with different syntax, such as the keypath-style \Type.method syntax. We could then potentially deprecate the old syntax and behavior.

[DevAndArtist (Adrian Meister)]

@Joe_Groff

We could then potentially deprecate the old syntax and behavior.

I'm not against shifting the syntax for that particular feature for the whole language, I'm just worried that the potential removement of the old syntax is not a guarantee. (Sorry if I'm a little pessimistic here.)

[Joe_Groff]

Evidence seems to suggest that Swift developers are pretty diligent at following deprecation warnings (if anything, they work too well, and our current deprecation approach introducing warnings for the current language version has come off as too aggressive for many developers). I don't think we need to be aggressive about completely removing the old behavior; warnings will discourage most developers from writing new code using it, and keeping the old behavior around will maintain compatibility with existing, working libraries that use it.

[marcusrossel]

So I guess the changes to the compiler that @Slava_Pestov described would therefore still have to be implemented, but would then be exposed via the keypath syntax?