Higher order functions and function types in Swift

(Jens Persson) #5

Some more examples of the same issue(s):

func takesTwoIntsFn(_ twoIntsFn: (Int, Int) -> Int) {
    print("twoIntsFn:", type(of: twoIntsFn))
}
func takesIntPairFn(_ intPairFn: ((Int, Int)) -> Int) {
    print("intPairFn:", type(of: intPairFn))
}

func test() {
    let twoIntsFn:  (Int, Int)  -> Int =  { $0   + $1   }
    let intPairFn: ((Int, Int)) -> Int =  { $0.0 + $0.1 }

    print(type(of: twoIntsFn)) // (Int, Int) -> Int
    print(type(of: intPairFn)) // (Int, Int) -> Int   (bug?)
    print(type(of: twoIntsFn) == type(of: intPairFn)) // false (as expected)

    takesTwoIntsFn(twoIntsFn) // (Int, Int) -> Int
    takesIntPairFn(intPairFn) // (Int, Int) -> Int    (bug?)
    
    // Note that the following compiles, but I assume it shouldn't?
    takesTwoIntsFn(intPairFn) // (Int, Int) -> Int    (bug?)
    takesIntPairFn(twoIntsFn) // (Int, Int) -> Int    (bug?)
    
    // While the following line results in an expected error:
    // let a: [(Int, Int) -> Int] = [twoIntsFn, intPairFn] // ERROR: "Cannot
    // convert value of type '((Int, Int)) -> Int' to expected element type
    // '(Int, Int) -> Int'"
    // note that the following compiles without errors:
    var arrayOfTwoIntsFns = [(Int, Int) -> Int]()
    arrayOfTwoIntsFns.append(twoIntsFn)
    arrayOfTwoIntsFns.append(intPairFn) // (bug?)

    var arrayOfIntPairFns = [((Int, Int)) -> Int]()
    // arrayOfIntPairFns = arrayOfTwoIntsFns // ERROR as expected
    arrayOfIntPairFns.append(arrayOfTwoIntsFns[0]) // (bug?)
    arrayOfIntPairFns.append(arrayOfTwoIntsFns[1]) // (bug?)
    // arrayOfIntPairFns[0] = arrayOfTwoIntsFns[0] // ERROR as expected
    // arrayOfIntPairFns.append(contentsOf: arrayOfTwoIntsFns) // ERROR as
    // expected but with a nonsensical error message.
}

test()

And, I think the following is a bit puzzling to think about:

func test() {
    let twoIntsFn:  (Int, Int)  -> Int =  { $0   + $1   }
    let intPairFn: ((Int, Int)) -> Int =  { $0.0 + $0.1 }

    // Some inputs of type (Int, Int):
    let inputs: [(Int, Int)] = [(1, 2), (3, 4), (5, 6)]

    // Both function types can be used with this (pair) input: 
    print(inputs.map(twoIntsFn)) // [3, 7, 11]
    print(inputs.map(intPairFn)) // [3, 7, 11]

    // I guess this have to be expected behavior (unless Swift introduced
    // some other way than tuples to define arg-lists), but it raises some
    // questions about in what sense the two functions have the same or
    // different type ...
}

// And, if (1) the above should continue to be expected behavior, is that
// really consistent with (2) treating the following (or the overloaded hmm
// in the OP) as different functions? I think 1 and 2 might be mutually
// exclusive ...

func foo<A, B, R>(_ fn: (_ arg1: A, _ arg2: B) -> R) {
    print("A two-args-function!")
}

func foo<A, B, R>(_ fn: (_ pair: (A, B)) -> R) {
    print("A one-pair-arg-function!")
}

test()

(I get the same result with default toolchain of Xcode 10.1 (10B61) and dev snapshot 2018-12-06.)

This area of Swift, ie essentially the whole conflation-of-parentheses-complexity-of-problems, including single-element tuples etc, has been broken in various different ways for years.

Any chance of these things ever getting fixed, or is it too complex and too late? (serious question)

Can anything be said about the priority/severity, relation to ABI stability and future language features, etc?

(Jens Persson) #6

Filed SR-9446 for the issue of higher order function accepting invalid arguments, although I'm not sure what fixing that would mean for the above example, repeated here:

func test() {
    let twoIntsFn:  (Int, Int)  -> Int =  { $0   + $1   }
    let intPairFn: ((Int, Int)) -> Int =  { $0.0 + $0.1 }

    // Some inputs of type (Int, Int):
    let inputs: [(Int, Int)] = [(1, 2), (3, 4), (5, 6)]

    // Both function types can be used with this (pair) input: 
    print(inputs.map(twoIntsFn)) // [3, 7, 11]
    print(inputs.map(intPairFn)) // [3, 7, 11]

    // I guess this have to be expected behavior (unless Swift introduced
    // some other way than tuples to define arg-lists), but it raises some
    // questions about in what sense the two functions have the same or
    // different type ...
}

Seems to me there is a situation where we have to choose between two seemingly buggy behaviors, or there is some other more satisfying solution.

(Slava Pestov) #7

The implementation of function types has gotten a lot better and most of the technical debt of the old representation is gone.

However there is an intentional implicit conversion between a function taking multiple arguments and a function taking a tuple argument in some narrow cases. This was introduced so that you could, for example, pass a closure taking two arguments to map() on a dictionary (where the elements are logically tuples).

Single-element tuples are a separate issue not related to the function type representation. It's a matter of banning single-element tuple expressions, and I really should just fix that.

2 Likes
(Slava Pestov) #8

We can't remove these conversions, because it would break source compatibility.

Note that gmm(pairFn) is ambiguous only with -swift-version 4, because we allow the splatting conversion in both directions. If I recall, this was due to an oversight in the old implementation.

In -swift-version 5, only one direction is allowed.

If you look at CSSimplify.cpp you'll see that now both conversions are explicitly codified because the new representation is quite different between the two arguments and single argument of tuple form.

It would be nice if ranking would prefer the version without the conversion if possible. Ranking is itself a bit of a mess right now, and needs some focused redesign.

2 Likes
How to set Swift version 5 (for recent dev snapshots) in Xcode build settings?
(Slava Pestov) #9

I explained what's going on with the implicit conversion in the bug.

I'm not sure why the behavior is buggy. Note that the two functions have different types, ie

print(type(of: intPairFn) == type(of: twoIntsFn))

Will print false.

(Jens Persson) #10

Huh, I just realized that my command line target's Swift Language Version has been set to 4.2 while I've been switching between the default toolchain and the recent master dev snapshot (which I thoughtlessly just expected to use Swift 5) ... So I got the wrong impression about the Swift 5 behavior ... Happy to see that I made a stupid mistake and things have been moving in the right direction.

However, the option for 5 is lacking from Xcode's UI, there's only 3, 4, 4.2 and unspecified, I've asked about this in a separate thread.


I just figured the overloaded-parentheses-and-single-element-tuples-thing might have something to do with things like the following printed function type representation SR-8235:

func f(_ a: Int, _ b: Int) {}
func g(_ pair: (Int, Int)) {}
func h(_ fn:  (Int, Int)  -> Void) {}
func i(_ fn: ((Int, Int)) -> Void) {}
func test() {
    print(type(of: f)) // Prints (Int, Int) -> ()
    print(type(of: g)) // Prints (Int, Int) -> ()    <--- bug?
    print(type(of: h)) // Prints ((Int, Int) -> ()) -> ()
    print(type(of: i)) // Prints ((Int, Int) -> ()) -> ()  <--- bug?
}
test()
I'm really sure this is the Swift 5 behavior this time.
$ cat > test.swift
func f(_ a: Int, _ b: Int) {}
func g(_ pair: (Int, Int)) {}
func h(_ fn:  (Int, Int)  -> Void) {}
func i(_ fn: ((Int, Int)) -> Void) {}
func test() {
    print(type(of: f))
    print(type(of: g))
    print(type(of: h))
    print(type(of: i))
}
test()
$ /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2018-12-06-a.xctoolchain/usr/bin/swiftc --version
Apple Swift version 5.0-dev (LLVM 491b123d06, Clang 9bb9e9884e, Swift 00d2acd809)
Target: x86_64-apple-darwin18.2.0
$ /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2018-12-06-a.xctoolchain/usr/bin/swiftc -swift-version 5 test.swift && ./test
(Int, Int) -> ()
(Int, Int) -> ()
((Int, Int) -> ()) -> ()
((Int, Int) -> ()) -> ()
$ 

How to set Swift version 5 (for recent dev snapshots) in Xcode build settings?
(Sebastian) #11

Why was the distinction not generally dropped for the sake of consistency, so that the second hmm would count as an invalid redeclaration? I feel this would also avoid some tuple destructuring in closures and weird "double" parentheses ...

(Jens Persson) #12

I think the current Swift 5 behavior is still clearly buggy:

func hmm(_ fn:  (Int, Int)  -> Int) { print("two ints") }
func hmm(_ fn: ((Int, Int)) -> Int) { print("pair") }

func test() {
  let twoIntsFn: (Int, Int) -> Int = { $0 + $1 }
  let pairFn: ((Int, Int)) -> Int =  { $0.0 + $0.1 }

  hmm(twoIntsFn) // error: ambiguous use of `hmm` <--- buggy behavior
  hmm(pairFn) // prints pair, as expected
}

test()

The reason behind that error is either very counterintuitive or a compiler bug.

(Slava Pestov) #13

I actually noticed the behavior you describe last night and pushed a fix. Glad to see there was a bug for it already, not sure why it slipped through the cracks but its fixed now. The problem was that the code to construct a mangling from a type metadata had not been updated for the new representation yet.

4 Likes
(Slava Pestov) #14

There are two overloads of hmm. When you call hmm with twoIntsFn, both overloads apply; the first one as-is, and the second with the implicit conversion that I describe. There's no general rule that says an overload is always preferred if it has fewer conversions or anything like that. So in the absence of a tie-breaker, the type checker concludes that neither solution is better than the other and diagnoses an error.

I agree this is a bit confusing, and in general the solution ranking needs an overhaul, but that's going to be difficult without breaking source compatibility.

2 Likes
(Jens Persson) #15

As far as I can see, the down sides (complexity, inconsistency, limited expressibility) of allowing that implicit conversion outweigh the upside (an inconsistent "convenience") that motivates it.

For example, it makes it impossible to write a function that can tell a single-(N-tuple)-argument function from an N-argument function in Swift. Or put differently: Higher order functions in Swift can't help but mixing up certain function types. I'm color blind and Swift's functions are tuple-argument blind.

(Xiaodi Wu) #16

Allowing that implicit conversion was specifically requested by many users after SE-0110 was implemented:

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170619/037616.html

Without explicit tuple splatting I don't see how this feature can be removed.

1 Like
(Slava Pestov) #17

This was the case before Swift 3 -- all functions logically took a single argument, which could be a tuple, and this assumption was baked in throughout the compiler. In Swift 3 a series of proposals were accepted and implemented:

(The last one is marked as 'deferred' but it was partially implemented; we went back and added the implicit conversion I described as an escape hatch. Note that in the old implementation, this conversion was free because both were literally the same type; when we added it back, we had to write explicit code to thunk between the two representations).

A big source of confusion in Swift 3 and 4 was that these proposals were only partially implemented because internally the compiler still modeled function types as taking a single argument. We finally mostly completed the transition away from this model on the master branch a few months ago.

4 Likes
(Jens Persson) #18

The following is the behavior of both Swift 4.2 and 5 with recent snapshots:

func f(_ x: Int, _ y: Int) { print("I was given two args:", x, "and", y) }
func g(_ pair: (Int, Int)) { print("I was given one pair:", pair) }

func test() {
    
    let a = f
    var b = g
    
    a(1, 2)    // I was given two args: 1 and 2
    b((3, 4))  // I was given one pair: (3, 4)

    // b = a // expected error: cannot assign value of type …  
    b = { $0 }(a) // 😎
    
    a(1, 2)    // I was given two args: 1 and 2
    b((3, 4))  // I was given two args: 3 and 4 🤔

    print(type(of: a) == type(of: b)) // false 
}

test()

Is this working as intended?

(Slava Pestov) #19

I think b = { $0 }(a) is not really intended to work, but I guess because of how the check is written we accept it...

Slava

(Xiaodi Wu) #20

On the other hand, I don't know that it's supposed to not work, though. It is using the closure to get the implicit conversion escape hatch that closures are supposed to provide.

I guess we've unintentionally created a horrible syntax for explicit tuple splatting... :crazy_face:

func f(_ x: Int, _ y: Int, _ z: Int) {
  print(x * y * z)
}

let x = (1, 2, 3)
({ $0 }(f) as ((Int, Int, Int)) -> Void)(x)
3 Likes
(Jens Persson) #21

A simple [x].map(f) is enough:

func f(_ x: Int, _ y: Int, _ z: Int) {
    print(x * y * z)
}

let x = (1, 2, 3)

_ = [x].map(f) // Prints 6

Or we can make a tuple splat operator:

infix operator •
func •<I, O>(lhs: (I) -> O, rhs: I) -> O { return lhs(rhs) }

func f(_ x: Int, _ y: Int, _ z: Int) {
    print(x * y * z)
}

let x = (1, 2, 3)

f • x // Prints 6

I would say it's more of an inconsistency than a feature. And, now that we have explicit tuple splatting, can we please remove the implicit conversion? :wink:

(Slava Pestov) #22

Your "tuple splat operator" relies on this implicit conversion. Otherwise you would not be able to pass a (Int, Int, Int) -> () where a (U) -> () is expected (try it in a context where you cannot convert types, for example as a generic argument; it won't work).

(Jens Persson) #23

Yes, I know, hence the wink smiley.

The point I was trying to make is that this implicit conversion is a lot more than the little “convenience/feature” it was accepted as.

It's a source of confusion and leads to further inconsistencies.

(Patrick Goley) #24

Would a change to the solution ranking really cause source compatibility issues here? In my case, I've had to distinguish between generic function args with a different function name or param labels.

// need "testN" (not just "test") function names or this example is currently ambiguous
func test1<T, U>(_ fn: (T) -> U) {
    
}

func test2<T, U, V>(_ fn: (T, U) -> V) {
    
}

func add(x: Int, y: Int) -> Int {
    return x + y
}

test2(add)

If we update the ranking to prefer the function that is not implicitly being converted to having a tuple arg, the ambiguity between unary and n-arity generic function args disappears but the code above still works fine. Basically, that change would allow the compiler to disambiguate where users had to manually disambiguate before (as I've done in my example). Is there another example where that isn't true?

1 Like