[draft] Introduce Sequence.filteredMap(_:)

moiseev · October 24, 2017, 5:35pm

+1 good idea.

Re. the naming I would suggest `mapFilterNil` since it says what it does and filter, nil, and map are all understood already in Swift. (I have sympathy for people wanting `mapFilteringNil`, but Swift chose `filter`.)

`nil` is an overloaded term, since there is ExpressibleByNilLiteral, and pretty much any type can conform to it.

···

On Oct 23, 2017, at 8:38 PM, Howard Lovatt <howard.lovatt@gmail.com> wrote:

The problems I see with `filterMap` are that:

 1. It sounds like it is a merged `filter` and `map` and therefore you would expect it to have two arguments, one to filter and one to map, i.e. `filterMap<R>(filter: (T) -> Bool, map: (T) -> R) -> [R]`.
 2. It sounds like it will filter the incoming values (for `nil`, but see 1 above) and then map, i.e. `filterMap<R>(map: (T?) -> R) -> [R]`, note `T?` *not* `R?`.

 -- Howard.

On 24 October 2017 at 11:56, BJ Homer via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
I agree with Xiaodi; I like ‘filterMap’ more than ‘filteredMap’. But both are superior to ‘flatMap’ in this context.

-BJ

On Oct 23, 2017, at 5:22 PM, Max Moiseev <moiseev@apple.com <mailto:moiseev@apple.com>> wrote:

It occurred to me that filteringMap(_:) should be even more descriptive, still conform to the guidelines, although similarly unprecedented and un-googlable.

Max

On Oct 23, 2017, at 3:52 PM, Xiaodi Wu <xiaodi.wu@gmail.com <mailto:xiaodi.wu@gmail.com>> wrote:

+1 in general. As to the name: since 'map' is used as a term of art, 'filterMap' seems superior to 'filteredMap', which half follows naming guidelines and half is a term of art; neither is immediately comprehensible but 'filterMap' can be googled and has precedents in other languages.
On Mon, Oct 23, 2017 at 17:24 BJ Homer via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
I strongly agree! In fact, I just started writing up a similar proposal the other day, but hadn’t had time to finish it yet.

The current name for this particular filtering variant is not particularly descriptive. It’s certainly not obvious to newcomers that ‘flatMap’ will filter out results. And it’s not true to the existing usage of ‘flatMap' from other languages; you have to really squint at it to see how any “flattening” is happening at all.

So yes, a big +1 from me. Thanks!

-BJ Homer

On Oct 23, 2017, at 4:15 PM, Max Moiseev via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi swift-evolution!

I would like to propose the following change to the standard library:

deprecate `Sequence.flatMap(_: (Element) -> U?) -> [U]` and make this functionality available under a new name `Sequence.filteredMap(_:)`.

The draft is available at 0000-introduce-filteredmap.md · GitHub and is included below for your convenience.

Max

Introduce Sequence.filteredMap(_:)

Proposal: SE-NNNN <https://gist.github.com/moiseev/NNNN-filename.md>
Authors: Max Moiseev <https://github.com/moiseev>
Review Manager: TBD
Status: Awaiting implementation
<0000-introduce-filteredmap.md · GitHub

We propose to deprecate the controversial version of a Sequence.flatMap method and provide the same functionality under a different, and potentially more descriptive, name.

<0000-introduce-filteredmap.md · GitHub

The Swift standard library currently defines 3 distinct overloads for flatMap:

Sequence.flatMap<S>(_: (Element) -> S) -> [S.Element]
 where S : Sequence
Optional.flatMap(_: (Wrapped) -> U?) -> U?
Sequence.flatMap(_: (Element) -> U?) -> [U]
The last one, despite being useful in certain situations, can be (and often is) misused. Consider the following snippet:

struct Person {
 var age: Int
 var name: String
}

func getAges(people: [Person]) -> [Int] {
 return people.flatMap { $0.age }
}
What happens inside getNames is: thanks to the implicit promotion to Optional, the result of the closure gets wrapped into a .some, then immediately unwrapped by the implementation of flatMap, and appended to the result array. All this unnecessary wrapping and unwrapping can be easily avoided by just using map instead.

func getAges(people: [Person]) -> [Int] {
 return people.map { $0.age }
}
It gets even worse when we consider future code modifications, like the one where Swift 4 introduced a Stringconformance to the Collection protocol. The following code used to compile (due to the flatMap overload in question).

func getNames(people: [Person]) -> [String] {
 return people.flatMap { $0.name }
}
But it no longer does, because now there is a better overload that does not involve implicit promotion. In this particular case, the compiler error would be obvious, as it would point at the same line where flatMap is used. Imagine however if it was just a let names = people.flatMap { $0.name <http://0.name/> } statement, and the names variable were used elsewhere. The compiler error would be misleading.

<0000-introduce-filteredmap.md · GitHub solution

We propose to deprecate the controversial overload of flatMap and re-introduce the same functionality under a new name. The name being filteredMap(_:) as we believe it best describes the intent of this function.

For reference, here are the alternative names from other languages:

Haskell, Idris >>>> mapMaybe :: (a -> Maybe b) -> [a] -> [b]
Ocaml (Core and Batteries) >>>> filter_map : 'a t -> f:('a -> 'b option) -> 'b t
F# >>>> List.choose : ('T -> 'U option) -> 'T list -> 'U list
Rust >>>> fn filter_map<B, F>(self, f: F) -> FilterMap<Self, F> >>>> where F: FnMut(Self::Item) -> Option
Scala >>>> def collect[B](pf: PartialFunction[A, B]): List[B]
<0000-introduce-filteredmap.md · GitHub compatibility

Since the old function will still be available (although deprecated) all the existing code will compile, producing a deprecation warning and a fix-it.

<0000-introduce-filteredmap.md · GitHub on ABI stability

This is an additive API change, and does not affect ABI stability.

<0000-introduce-filteredmap.md · GitHub on API resilience

Ideally, the deprecated flatMap overload would not exist at the time when ABI stability is declared, but in the worst case, it will be available in a deprecated form from a library post-ABI stability.

<0000-introduce-filteredmap.md · GitHub considered

It was attempted in the past to warn about this kind of misuse and do the right thing instead by means of a deprecated overload with a non-optional-returning closure. The attempt failed due to another implicit promotion (this time to Any).

The following alternative names for this function were considered:

mapNonNil(_:) >>>> Does not communicate what happens to nil’s
mapSome(_:) >>>> Reads more like «map some elements of the sequence, but not the others» rather than «process only the ones that produce an Optional.some»
filterMap(_:) >>>> Does not really follow the naming guidelines and doesn’t seem to be common enough to be considered a term of art.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

moiseev · October 24, 2017, 6:45pm

If the biggest problem of flatMap is that people who don’t understand it write code that isn’t optimal (but still does the right thing!), I don’t think there is any change needed. I’m not even sure that that wrapping/unwrapping is actually done, because it should be discoverable by the compiler.

It can be seen that the generated SIL is more complicated for a flatMap case, the microbenchmark also shows that it’s about 3x slower than the map. But even if that can be optimized away, don’t you find a String example from the proposal even a little convincing?

It’s unfortunate when a piece of code gets a different meaning — but as you said:
It’s a misuse, and when you write code that is wrong, the results might be wrong as well.

And this is exactly that we’re trying to address. Not the impurity of the function, not the performance, but the potential to misuse.

Maybe Scala chose better name for the concept, but the Swift definition is consistent as it is now.

collect does not say anything about the transformation (A => B) that happens along the way…

So, if you remove the third overload, the second one imho should be renamed, too.

Honestly, I don’t see why. Can you elaborate? The second one is perfectly consistent, it does not mix two different functor instances in the same function.

···

On Oct 24, 2017, at 11:36 AM, Tino <2th@gmx.de> wrote:

Tino · October 25, 2017, 9:15am

It’s unfortunate when a piece of code gets a different meaning — but as you said:
It’s a misuse, and when you write code that is wrong, the results might be wrong as well.

And this is exactly that we’re trying to address. Not the impurity of the function, not the performance, but the potential to misuse.

But do you think filterMap can’t be misused? I don’t know why people choose flatMap instead of map, but they might as well just switch to filterMap.

Maybe Scala chose better name for the concept, but the Swift definition is consistent as it is now.

collect does not say anything about the transformation (A => B) that happens along the way…

„collectResults“ is probably clearer… but I guess Scala-folks choose to prefer a single word.
But when it’s about clear naming, we probably should talk about „applyFunctionAndCollectUnwrappedResults“, because filtering has no obvious connection to nil-values.
Even worse, I think filter is the wrong word in this context, because that’s what I’d expect from such a function:
Sequence.filterMap(_: (Element) -> U?) -> [U?]

In this case, the result would only contain those Optionals that contain a value — or there could be a parameter for the filter, defaulted to a check for nil.

So, if you remove the third overload, the second one imho should be renamed, too.

Honestly, I don’t see why. Can you elaborate? The second one is perfectly consistent, it does not mix two different functor instances in the same function.

As it is now, both Sequence and Optional are very similar — they are container types that may be empty.
When you say that one incarnation of flatMap that deals with Optionals is wrong, imho it’s cleaner to break that connection completely, and make it a Collection-only thing.
Overload 2 is kind of a special case of the third one.

moiseev · October 25, 2017, 6:05pm

It’s unfortunate when a piece of code gets a different meaning — but as you said:
It’s a misuse, and when you write code that is wrong, the results might be wrong as well.

And this is exactly that we’re trying to address. Not the impurity of the function, not the performance, but the potential to misuse.

But do you think filterMap can’t be misused? I don’t know why people choose flatMap instead of map, but they might as well just switch to filterMap.

Oh I’m sure they will. But I hope it will be harder, because we will have a deprecated overload of a flatMap that will provide a helpful message and guide people to use map instead.

Maybe Scala chose better name for the concept, but the Swift definition is consistent as it is now.

collect does not say anything about the transformation (A => B) that happens along the way…

„collectResults“ is probably clearer… but I guess Scala-folks choose to prefer a single word.

Let’s not forget about the types. Function type is an essential piece of information that should accompany it’s name. With a function type being (LIst[A], PartialFunction[A, B]) => List[B] the name is almost not needed, there is pretty much one way of implementing this function (if we forget about trivial and wrong implementations that, for example, return an empty list all the time, of perform arbitrary side-effects).

But when it’s about clear naming, we probably should talk about „applyFunctionAndCollectUnwrappedResults“, because filtering has no obvious connection to nil-values.
Even worse, I think filter is the wrong word in this context, because that’s what I’d expect from such a function:
Sequence.filterMap(_: (Element) -> U?) -> [U?]

In this case, the result would only contain those Optionals that contain a value — or there could be a parameter for the filter, defaulted to a check for nil.

So, if you remove the third overload, the second one imho should be renamed, too.

Honestly, I don’t see why. Can you elaborate? The second one is perfectly consistent, it does not mix two different functor instances in the same function.

As it is now, both Sequence and Optional are very similar — they are container types that may be empty.
When you say that one incarnation of flatMap that deals with Optionals is wrong, imho it’s cleaner to break that connection completely, and make it a Collection-only thing.
Overload 2 is kind of a special case of the third one.

I was actually thinking in the opposite direction ;-) As in, adding more functions from Collection to Optional. filter can be useful, count, isEmpty, forEach… I’m not saying that it will conform to the Collection protocol, but I can see how sometimes `opt.filter { $0 > 0 }` is a better option than `if let v = opt, x > 0 { … } else { … }

···

On Oct 25, 2017, at 2:15 AM, Tino <2th@gmx.de> wrote:

Alejandro_Martinez · October 26, 2017, 7:40pm

I wrote about this in 2015
The unexpected but convenience case of flatMap in Swift | Alexito's World and forgot about
it.
I just get used to it now but it's still hard to explain to newcomers
why flatMap has this "special" behaviour on Swift.
I'm not an expert on functional programming but if this doesn't make
sense in the theory and it's just for convenience I agree we could
rename it. Still, if it has some reason to work like this (higher
kinded types?) and we expect to have them in an infinite time scale...
maybe we should hold on.

···

On Wed, Oct 25, 2017 at 7:06 PM, Max Moiseev via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 25, 2017, at 2:15 AM, Tino <2th@gmx.de> wrote:

It’s unfortunate when a piece of code gets a different meaning — but as you
said:
It’s a misuse, and when you write code that is wrong, the results might be
wrong as well.

And this is exactly that we’re trying to address. Not the impurity of the
function, not the performance, but the potential to misuse.

But do you think filterMap can’t be misused? I don’t know why people choose
flatMap instead of map, but they might as well just switch to filterMap.

Oh I’m sure they will. But I hope it will be harder, because we will have a
deprecated overload of a flatMap that will provide a helpful message and
guide people to use map instead.

Maybe Scala chose better name for the concept, but the Swift definition is
consistent as it is now.

collect does not say anything about the transformation (A => B) that happens
along the way…

„collectResults“ is probably clearer… but I guess Scala-folks choose to
prefer a single word.

Let’s not forget about the types. Function type is an essential piece of
information that should accompany it’s name. With a function type being
(LIst[A], PartialFunction[A, B]) => List[B] the name is almost not needed,
there is pretty much one way of implementing this function (if we forget
about trivial and wrong implementations that, for example, return an empty
list all the time, of perform arbitrary side-effects).

But when it’s about clear naming, we probably should talk about
„applyFunctionAndCollectUnwrappedResults“, because filtering has no obvious
connection to nil-values.
Even worse, I think filter is the wrong word in this context, because that’s
what I’d expect from such a function:

Sequence.filterMap(_: (Element) -> U?) -> [U?]

In this case, the result would only contain those Optionals that contain a
value — or there could be a parameter for the filter, defaulted to a check
for nil.

So, if you remove the third overload, the second one imho should be renamed,
too.

Honestly, I don’t see why. Can you elaborate? The second one is perfectly
consistent, it does not mix two different functor instances in the same
function.

As it is now, both Sequence and Optional are very similar — they are
container types that may be empty.
When you say that one incarnation of flatMap that deals with Optionals is
wrong, imho it’s cleaner to break that connection completely, and make it a
Collection-only thing.
Overload 2 is kind of a special case of the third one.

I was actually thinking in the opposite direction ;-) As in, adding more
functions from Collection to Optional. filter can be useful, count, isEmpty,
forEach… I’m not saying that it will conform to the Collection protocol, but
I can see how sometimes `opt.filter { $0 > 0 }` is a better option than `if
let v = opt, x > 0 { … } else { … }

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Alejandro Martinez

Tino1 · October 27, 2017, 11:21am

I wrote about this in 2015
The unexpected but convenience case of flatMap in Swift | Alexito's World and forgot about
it.

That text has a line which illustrates the source of dissent:

It’s kind of combining map (to apply the function), flatten (to unwrap the optionals) and filter (to remove the nils)

I’m saying this isn’t true, because after the flatten step, there are no Optionals left, and thus, there can be no nils at all (unless we startet with Optional<Optional<T>>…)

Tino · October 27, 2017, 4:15pm

I wrote about this in 2015
The unexpected but convenience case of flatMap in Swift | Alexito's World and forgot about
it.

That text has a line which illustrates the source of dissent:

It’s kind of combining map (to apply the function), flatten (to unwrap the optionals) and filter (to remove the nils)

I’m saying this isn’t true, because after the flatten step, there are no Optionals left, and thus, there can be no nils at all (unless we startet with Optional<Optional<T>>…)

(hope this doesn’t arrive twice — imho Mail.app really should be retired for this kind of discussion ;-)