Confusing behaviour of optionals

The more I learn with (Swift) optionals, the more confused I become. The examples that come to mind are

  1. Why does this block compile? More specifically, which == am I using or is this result due to syntactic sugar or special syntax for optionals
let a: Int? = 1
let b: Int = 1
print(a == b)
// => true
  1. The behaviour of flatMap on Optionals and Arrays is somewhat confusing. More specifically, flatMap frequently behaves like map for many closures despite the different type signatures:
let x: Int? = 1
let f: (Int) -> Int = { $0 + 1 }
x.map(f) // 2 : Int?
x.flatMap(f)
// 2 : Int?
// but according to the type signature, this line should not compile

However, when there is nested structure in the return argument, they return different results. I would expect them to also return different result if it should compile:

let y: String = "1"
y.map(Int.init) // 1 :: Int??
y.flatMap(Int.init) // 1 :: Int?
let array = [1, 2, 3]
array.map { [$0] } // [[1], [2], [3]]
array.flatMap { [$0] } // [1,2,3]

My understanding is that map is related to the functor instance and flatMap is the monad instance. Is this belief correct?

  1. I believe 2 might be a result of the following behaviour:
let f = { (x: Int) -> Int in return x * 10 }
let g = f as? ((Int) -> Int?)
if let h = g { print(h(10)) }
// => 70

How does this above work?

Unrelated - is the following behaviour expected:

$ swift
23> let f = Optional<Int>.flatMap 
error: repl.swift:23:23: error: generic parameter 'U' could not be inferred

23> let f = Optional<Int>.flatMap<Int?>
error: repl.swift:23:23: error: cannot explicitly specialize a generic function
let f = Optional<Int>.flatMap<Int?>
                      ^

repl.swift:23:30: note: while parsing this '<' as a type parameter bracket
let f = Optional<Int>.flatMap<Int?>
1 Like
  1. This one. Swift provides coercion of non-optional types to optional types, so it's more or less:
print(a == (b as Int?))
  1. I suspect that coercion makes the non-optional type subclass of optional type, considering this works:
let f: (Int) -> Int = { $0 + 1 }
let g: (Int) -> Int? = f

This is similar to

protocol Base { }
extension Int: Base { }

let a: (Base) -> Int = { _ in 0 }
let b: (Int) -> Int = a
let c: (Base) -> Base = a
let d: (Int) -> Base = a
  1. Explicitly writing type out should make it clearer:
let f: (Int) -> Int = { x in x * 10 }
let g: ((Int) -> Int?)? = f as? ((Int) -> Int?)
if let h = g {
    // let h: (Int) -> Int?
    print(h(10))
}
1 Like

The type signature looks something like this,

let f: (Int?) -> ((Int) throws -> U?) throws -> U?  = Optional<Int>.flatMap 

You didn't specify U in the first line.
Second line already said that you cannot explicitly specialize a generic function, this is expected.

Yes, it's expected if pretty confusing.

Like the error says, you can't explicitly specialize a generic function (unlike a generic type), and you can't have an variable hold an unspecialized generic function. That is, you can't write this:

func f<T>(_ t: T) { } 

// can't specialize f explicitly
let g = f<Int>
// error: cannot explicitly specialize a generic function

// but you need to specify what T is somehow
let g = f
// error: 'T' could not be inferred

The way you specify this specialization instead is via type context. Normally the type can be inferred from the context (like when you're passing in an argument to a function that expects a particular type), but you can also supply type context manually:

// T must be of type Int here:
let g: (Int)->Void = f

This is what's going on with your unapplied functions map and flatMap, except it's way more complicated because unapplied methods are weird (they're a function from a potential self to a function that you can call on self), and the generic argument U is the return type of a function passed as an argument to map (because you can map to a different type to the value in the optional). But once you figure out the signature you need, it works:

// fixing 'U' to be `Double`
let f: (Int?) -> ((Int) throws -> Double?) throws -> Double? = Optional<Int>.flatMap

This seems consistent to me:

map returns a doubly wrapped type, such as Optional<Optional<Int>> aka Int?? or Array<Array<Int>> aka [[Int]].

flatMap “flattens” the result into a singly wrapped type, such as Optional<Int> and Array<Int>

What would type coercion be for flatMap on an [T] be? The declaration

func flatMap<SegmentOfResult>(_ transform: (Element) throws -> SegmentOfResult) rethrows -> [SegmentOfResult.Element] where SegmentOfResult : Sequence

indicates it should be a Sequence but

[1,2,3].flatMap { $0 + 1 }

would have a closure of type (Int) -> Int. It's likely [Int] but it's difficult to determine or discover (at least for me)

I was not clear as I thought. I understand what the types are for g and h. I was confused as to how we (or the compiler) can change the return type of function / method. For example, [1,2,3].flatMap { $0 + 1 } seems to convert { $0 + 1 } to return a Sequence based on its declaration

func flatMap<SegmentOfResult>(_ transform: (Element) throws -> SegmentOfResult) rethrows -> [SegmentOfResult.Element] where SegmentOfResult : Sequence

but

let f = { $0 + 1 } // : (Int) -> Int
let g = f as? (Int) -> [Int] // nil
let h = f as? (Int) -> CollectionOfOne<Int> // nil

This behaviour is what I'm expecting. My confusion is from the behaviour when you use some other return type that doesn't return the nested structure.

There is an implicit promotional to optional that is happening automagically. It is unfortunate not well documented officially.

That's not quite what's happening.

Assuming you're using a newer version of the compiler, you ought to be getting a warning for that usage: "'flatMap' is deprecated: Please use compactMap(_:) for the case where closure returns an optional value"

The history is, Sequence.flatMap used to be overloaded to mean two things:

  • a "sequence-flattening" map that took a mapping from (T) -> [U] and produced a [U] (not a [[U]] like map would)
  • a "nil-discarding" map that took (T)->U? and produced a [U] by discarding nil values.

Having the same name for both caused immense confusion. As you've seen, the compiler will happily convert (T)->U to (T)->U? if it'll make an expression type check. So what happens here:

let y: [Double] = [1,2,3].compactMap { Double($0) }

is that the compiler is taking { Double($0) }, a function (Int) -> Double, and converting it to a function (Int) -> Double? by wrapping it in a closure that converts the value to an optional. Then compactMap goes through all those optional return values, unwrapping them, discarding the nils (in this case, there won't be any). It's a lot of extra work for the same result as calling map. We saw loads of people misusing flatMap like this, hence the rename of the compacting version.

That makes more sense. Is there a way to turn on warnings inside of the Swift interpreter? I wasn't getting any of these warning in the interpreter:

$ swift
Welcome to Apple Swift version 5.0 (swiftlang-1001.0.69.3 clang-1001.0.47).
Type :help for assistance.
  1> let f = { $0 + 1 }
f: (Int) -> Int = 0x00000001000e9030 $__lldb_expr2`closure #1 (Swift.Int) -> Swift.Int in __lldb_expr_1 at repl.swift:1
  2> let g = f as? (Int) -> CollectionOfOne<Int>
g: ((Int) -> CollectionOfOne<Int>)? = nil
  3> [1,2,3].flatMap { $0 + 1 }
$R0: [Int] = 3 values {
  [0] = 2
  [1] = 3
  [2] = 4
}
  4>  

What is the best way about adding official documentation (around this behaviour)? It can be confusing for this behaviour, especially since all subtypes don't get promoted. For example, an argument could be made for T promoting to Result<T, Swift.Error>. I wouldn't make it, but this behaviour doesn't (currently) occur. There's another example but I can't think of it right now.

@Ben_Cohen or @nnnnnnnn would know better.

This example was what I was thinking about before.

The difference between A?.flatMap and [A].flatMap is causing some confusion as well. If f: (A) -> B, then [A].flatMap(f) will complain but A?.flatMap(f) will not. Is it possible to include a similar warning for the incorrect application of A?.flatMap(f) that we say with [A].flatMap?

Optional.flatMap is correct. A flatmap takes what would be a Foo<Foo<T>> if you used map and collapses it to Foo<T> instead. compactMap is specific to sequences of optionals, i.e. there's two different types involved.

I believe @atfelix is referring to what happens when you call Optional.flatmap with a function which returns a non-optional:

func foo(_ x: Int) -> Int { return x }
let a: Int? = 4
let b = a.flatMap(foo)
print(b)    // Optional(4)

In particular, should we make the compiler reject this entirely, since the behavior is identical to that of map?

Ah, I see. Yes, that would be nice…but only because we know there's an alternative available. I got confused because the same applies to Sequence.compactMap, which will also not warn.

Perhaps we ought to prevent compactMap from accepting a function which returns a non-optional as well.

Yes, I was referring to Optional<A>.flatMap behaviour when the return type of the closure is non-optional.

I would be in favour of both

  • a.flatMap { (x: Int) in return x } failing compilation when a: A?
  • [1].compactMap { (x: Int) in return x }` failing compilation

I believe that would lead to less confusion.