Proposal: Add scan, takeWhile, dropWhile, and iterate to the stdlib

When the proposal is "we have a bunch of functions that match functions used in other languages, lets add a few more from the same category of functions that we missed", does there really need to be much explanation beyond "they're useful in the other languages that have them, they'd be useful for the same reasons in Swift”?

I agree with what you’re saying, but the flip-side is: how do we scope what we accept into the standard library? Just existing in some other language doesn’t mean that we should (automatically) accept new standard library functionality.

I’m not arguing for or against your proposal, just trying to say that this rationale isn’t enough to justify adding things to the Swift standard library.

I agree.

Something I would like to see in the longer term is a set of libraries outside the standard library, but still going through a community review process for their evolution. They would be given an “official” stamp of approval and would be hosted in community repos, but accessed via SPM rather than part of the core install. This would allow us to keep the core library lean, but still offer a “batteries included” approach.

Matthew

Naming conventions would suggest that something returning a new collection should be named with a noun phrase describing what it returns. Unfortunately, most of the ones I can think of off the top of my head are fairly clunky. "suffixFromFirstNonMatchingElement" describes what it does, but I haven't thought of a non-painful way to say that yet. "suffixExcluding" is almost right, but it incorrectly implies (to my eye at least) that the returned collection excludes all elements matching the predicate, rather than just matching prefixes. Hm, what about flipping the predicate and getting a "suffixFrom" overload that takes a predicate for the first matching element to be included, rather than the last matching element to be excluded?

  David

Rob Rix pointed out that "suffixAfter" would meet all my original criteria. Not sure if keeping the original "match the stuff to drop rather than the stuff to keep" semantics are critical, but this gives us an option for either way :blush:

    David

Oops, you're right on the skipWhile(), that snuck in there because I was looking at the Rust documentation (and Rust uses the name skipWhile()).

As for dropUntil, all of the precedent I know of behaves like a dropWhile, not a dropUntil. And dropUntil sounds weird to me; every time I've wanted to use this functionality, I do in fact want to drop
the stuff I don't want, rather than dropping until some predicate becomes true.

Language precedent:

Rust: skip_while() and take_while()
Ruby: drop_while() and take_while()
Python: dropwhile() and takewhile()
Haskell: dropWhile and takeWhile

suffixAfter sounds like the equivalent of dropFirst(_:), i.e. it sounds like it should take a count of elements to skip. Similarly, actually trying an expression that takes a predicate looks weird:

seq.suffixAfter({ isspace($0) })

Even knowing what it's supposed to do, it's hard for me to read that expression in any sensible fashion.

Also, I'm not sure the "noun phrase" convention really makes sense for SequenceType methods. They're not technically mutating methods, but single-pass collections are in fact destructively mutated by Array-returning sequence methods (and the methods that return SubSequence also destructively mutate upon any access to the returned subsequence). Which is to say, despite not being marked as mutating, they do in fact behave like mutating methods for single-pass sequences. Which suggests that verb phrases are perfectly fine.

I agree with both of you: that these are useful additions, but also that it’s important to know where to draw the line.

Sorry if this is not the place for this discussion (my first post in this mailing list):
I feel that one way of moving this discussion forward is by looking more generally at what justifies adding new functionality that is similar to existing functionality.

One part of that discussion is how the new additions would serve the broader goals of the language.

- Is this a style that the standard library wants to encourage?
- Why (to serve what goal) was the existing functionality added in the first place?
- Does this addition continue to serve some of the goals of that addition?
For this proposal it would be useful to know a little bit more about the original discussion about woking with sequences in a functional manner and adding `map`, `filter`, `reduce` etc. to the standard library in the first place, since the proposed additions also relate to woking with sequences in a functional manner
For this proposal it might also be helpful knowing some of the reasoning behind adding `prefix`, `suffix`, `dropFirst`, and `dropLast`, since the proposed additions have similar functionality.

Another part of that discussion is how the new functionality relates to the existing.

If the new is more is more general than the existing, then what other useful additions could be implemented with this new? It might be interesting to reverse the question and ask if the existing would be added if the new proposed function already existed.

If the new is a specialization of something existing, then is it something that many developers will end up implementing themselves? I would see this as a good reason to implement it in the standard library. Even more so if there is a high risk that individual implementations will have subtle differences in behavior or a high risk of bugs in the individual implementations.

David

>
>
>>
>> When the proposal is "we have a bunch of functions that match functions
used in other languages, lets add a few more from the same category of
functions that we missed", does there really need to be much explanation
beyond "they're useful in the other languages that have them, they'd be
useful for the same reasons in Swift”?
>
> I agree with what you’re saying, but the flip-side is: how do we scope
what we accept into the standard library? Just existing in some other
language doesn’t mean that we should (automatically) accept new standard
library functionality.
>
> I’m not arguing for or against your proposal, just trying to say that
this rationale isn’t enough to justify adding things to the Swift standard
library.

I agree.

Something I would like to see in the longer term is a set of libraries
outside the standard library, but still going through a community review
process for their evolution. They would be given an “official” stamp of
approval and would be hosted in community repos, but accessed via SPM
rather than part of the core install. This would allow us to keep the core
library lean, but still offer a “batteries included” approach.

This might be a good place for things like Either, Deque, etc.

>
>
>>
>> When the proposal is "we have a bunch of functions that match functions used in other languages, lets add a few more from the same category of functions that we missed", does there really need to be much explanation beyond "they're useful in the other languages that have them, they'd be useful for the same reasons in Swift”?
>
> I agree with what you’re saying, but the flip-side is: how do we scope what we accept into the standard library? Just existing in some other language doesn’t mean that we should (automatically) accept new standard library functionality.
>
> I’m not arguing for or against your proposal, just trying to say that this rationale isn’t enough to justify adding things to the Swift standard library.

I agree.

Something I would like to see in the longer term is a set of libraries outside the standard library, but still going through a community review process for their evolution. They would be given an “official” stamp of approval and would be hosted in community repos, but accessed via SPM rather than part of the core install. This would allow us to keep the core library lean, but still offer a “batteries included” approach.

This might be a good place for things like Either, Deque, etc.

Deque sure and maybe Either. But if we’re going to get an Either that is really a Result I would prefer to see that in the standard library and integrated with the error handling system (i.e. allow us to call a throwing function without `try` and receive a Result wrapping the return value, etc).

The main problem with that reasoning is that every Collection is a
Sequence, and the vast majority of Sequences are in fact Collections (or
at least multi-pass). So you end up with lots of verb phrases for
things that in reality are non-mutating.

To be honest these all seem equally "weird" looking to me, so my hope is that it's just a matter of familiarity :blush: My personal inclination is still that overloading suffixFrom with a predicate-taking variant is the way to go. That way it occupies the same mental slot as the existing suffixFrom method, and all you have to decide when using it is "do I want to specify where to suffix from, or how to find where to suffix from?". e.g. it basically becomes sugar that turns

let idx = foo.indexOf { … }
let suffix = foo.suffixFrom(idx)

into

let suffix = foo.suffixFrom { … }

with the bonus feature that it works on single-pass sequences.

  David

I'm rather opposed to the idea of taking the methods that take indexes and overloading them to take predicates instead. They're extremely different functionality. An index is a value that, once generated, is
constant-time to use, and therefore the functions that take them are typically O(1). For example, CollectionType.suffixFrom() is documented explicitly as being O(1).

But functions taking a predicate must be O(N), because they have to evaluate the predicate on every element in turn until it returns true (or false, depending on the method in question).

So if we overload suffixFrom() to take a predicate, then we have one function with one overload that's always O(1), and one overload that's always O(N), and that is a great way to confuse people and hide performance issues.

This is also why I'm particularly fond of the takeWhile / dropWhile terminology. Besides the rather extensive precedent, the inclusion of the word "while" makes it clear that it's iterating over the
sequence/collection, which means it's intuitively O(N).

+1 to iterate and scan.

I'm familiar with takeWhile from other languages, but I think it doesn't quite fit with existing library methods. I don't like the phrasing of dropFirst already in the library, I'd prefer something more familiar to dropWhile.

Stops when the block returns true, doesn't include that element:

prefixUpTo {...}`

Starts when the block returns true, includes that element.

suffixFrom { ... }

So:

array.prefixUpTo(filterFunction) + array.suffixFrom(filterFunction) == array

Great point Lily, you've convinced me. I'm also happy with dropWhile for this reason. My issues with "drop" should be resolved in a separate proposal.

I’m tempted to say we should rename dropFirst() / dropWhile() to skipFirst() / skipWhile(), i.e. use Rust’s name, because “skip” doesn’t sound like a mutating verb but I do see why people say “drop” does. But yeah, that would be a different proposal entirely.