[Pitch] Retiring `where` from for-in loops


(Erica Sadun) #1

Upon accepting SE-0099, the core team is removing `where` clauses from condition clauses, writing "the 'where' keyword can be retired from its purpose as a boolean condition introducer."

Inspiried by Xiaodi Wu, I now propose removing `where` clauses from `for in` loops, where they are better expressed (and read) as guard conditions.

Guard conditions can `continue` (mimicking the current use of `where`) or `break` (introducing the recently pitched `while` behavior). This limits the current situation where people new to the language expect `while` behavior and expect termination rather than sequence filtering. Removing `where` from for-in loops benefits these new users, reduces cognitive burden for all users, and enhances readability and predictability.

I do not believe the same benefit would accrue in retiring `where` from `catch` clauses and `switch` statement cases. One can argue that there are inherent flaws in both situations: unlike generic constraints, nothing prevents semantic disjunction in their `where` clauses. That said, both measurably benefit from their `where` clauses in the current grammar:

case_item_list : pattern where_clause? | pattern where_clause? ',' case_item_list
catch_clause : 'catch' pattern? where_clause? code_block

Case item lists allow comma-separated patterns in a single case statement. The only way to express a related Boolean assertion is through `where`.
Catch clauses do not allow multiple patterns but I cannot think of an improved way to associate an assertion than `where`.

-- E


(Brent Royal-Gordon) #2

Upon accepting SE-0099, the core team is removing `where` clauses from condition clauses, writing "the 'where' keyword can be retired from its purpose as a boolean condition introducer."

Inspiried by Xiaodi Wu, I now propose removing `where` clauses from `for in` loops, where they are better expressed (and read) as guard conditions.

Do you propose to remove `for case` as well? That can equally be handled by a `guard case` in the loop body.

Alternate proposal: Move `where` clauses to be adjacent to the pattern‚ÄĒrather than the sequence expression‚ÄĒin a `for` loop, just as they are in these other syntaxes.

  for n where n.isOdd in 1...1_000 { … }

This makes them more consistent with the syntax in `switch` cases and `catch` statements, while also IMHO clarifying the role of the `where` clause as a filter on the elements seen by the loop.

···

--
Brent Royal-Gordon
Architechies


(Erica Sadun) #3

I saw your post on that *after* I finished sending this. Moving `where` next to the pattern, like you'd find in `catch` and switch `case`, the code would look like this:

for i where i % 2 == 0 in sequence {
    // do stuff
}

I agree that's really clever and an improvement but after coming up with all the points about wrong expectations about termination vs filtering, the better use of guard, and fetishes about vertical compactness, I think (call it +0.6) I'm going to stick to my guns on this one - and for `for case` too. I've been wuxxed.

* New users might expect the sequence to terminate as soon as i % 2 is 1, rather than the correct interpretation which is "this is a filtering operation"
* The code can be expressed less ambiguously as

for i in sequence.filter({ return i % 2 == 0 }) {
    // do stuff
}

* The while version can be expressed as

for i in sequence.prefix(while: { return $0 % 2 == 0 } ) {
    // do stuff
}

* The code can also use `guard` statements as needed with `break` and `continue`

(And yes, I should have pointed out filter and prefix as well as guard in my first email)

-- E

···

On Jun 8, 2016, at 9:36 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

Upon accepting SE-0099, the core team is removing `where` clauses from condition clauses, writing "the 'where' keyword can be retired from its purpose as a boolean condition introducer."

Inspiried by Xiaodi Wu, I now propose removing `where` clauses from `for in` loops, where they are better expressed (and read) as guard conditions.

Do you propose to remove `for case` as well? That can equally be handled by a `guard case` in the loop body.

Alternate proposal: Move `where` clauses to be adjacent to the pattern‚ÄĒrather than the sequence expression‚ÄĒin a `for` loop, just as they are in these other syntaxes.

  for n where n.isOdd in 1...1_000 { … }

This makes them more consistent with the syntax in `switch` cases and `catch` statements, while also IMHO clarifying the role of the `where` clause as a filter on the elements seen by the loop.


(Sean Heber) #4

Upon accepting SE-0099, the core team is removing `where` clauses from condition clauses, writing "the 'where' keyword can be retired from its purpose as a boolean condition introducer."

Inspiried by Xiaodi Wu, I now propose removing `where` clauses from `for in` loops, where they are better expressed (and read) as guard conditions.

Do you propose to remove `for case` as well? That can equally be handled by a `guard case` in the loop body.

Alternate proposal: Move `where` clauses to be adjacent to the pattern‚ÄĒrather than the sequence expression‚ÄĒin a `for` loop, just as they are in these other syntaxes.

  for n where n.isOdd in 1...1_000 { … }

This makes them more consistent with the syntax in `switch` cases and `catch` statements, while also IMHO clarifying the role of the `where` clause as a filter on the elements seen by the loop.

I saw your post on that *after* I finished sending this. Moving `where` next to the pattern, like you'd find in `catch` and switch `case`, the code would look like this:

for i where i % 2 == 0 in sequence {
    // do stuff
}

This is the best version yet - the placement of 'where' makes total sense and I really like it there.

I agree that's really clever and an improvement but after coming up with all the points about wrong expectations about termination vs filtering, the better use of guard, and fetishes about vertical compactness, I think (call it +0.6) I'm going to stick to my guns on this one - and for `for case` too. I've been wuxxed.

* New users might expect the sequence to terminate as soon as i % 2 is 1, rather than the correct interpretation which is "this is a filtering operation"
* The code can be expressed less ambiguously as

for i in sequence.filter({ return i % 2 == 0 }) {
    // do stuff
}

This seems to trade what was a very declarative syntax about the intent of some code (especially with 'where' in the middle of the statement) for one that injects its own specialized vocabulary into the context (knowing what filter does, a function call, a closure with a return keyword and a pair of extra braces and parenthesis!) which means, to me anyway, significant cognitive overhead. It will also be a lot slower without optimization enabled due to the intermediate array. (I've found *significant* speed ups switching .forEach() with for loops in debug builds, for example.)

* The while version can be expressed as

for i in sequence.prefix(while: { return $0 % 2 == 0 } ) {
    // do stuff
}

And now we've gone from, again, what is likely a very simple and declarative style using a for/while kind of statement and turned it in to something that has *even more* cognitive overhead to figure out what it does because now I have to reason about what "prefix" means here (normally I only think of prefix in the context of strings) and if there's a special variation of it using the "while" argument that I need to also be aware of...

Maybe it's just me, but.. I don't get it. I want to be able to quickly understand a piece of code's intent, not wade through fancy constructions for their own sake.

l8r
Sean - who might be too tired to be emailing responsibly

···

On Jun 8, 2016, at 10:51 PM, Erica Sadun via swift-evolution <swift-evolution@swift.org> wrote:

On Jun 8, 2016, at 9:36 PM, Brent Royal-Gordon <brent@architechies.com> wrote:


(Charlie Monroe) #5

Upon accepting SE-0099, the core team is removing `where` clauses from condition clauses, writing "the 'where' keyword can be retired from its purpose as a boolean condition introducer."

Inspiried by Xiaodi Wu, I now propose removing `where` clauses from `for in` loops, where they are better expressed (and read) as guard conditions.

Do you propose to remove `for case` as well? That can equally be handled by a `guard case` in the loop body.

Alternate proposal: Move `where` clauses to be adjacent to the pattern‚ÄĒrather than the sequence expression‚ÄĒin a `for` loop, just as they are in these other syntaxes.

  for n where n.isOdd in 1...1_000 { … }

This makes them more consistent with the syntax in `switch` cases and `catch` statements, while also IMHO clarifying the role of the `where` clause as a filter on the elements seen by the loop.

I saw your post on that *after* I finished sending this. Moving `where` next to the pattern, like you'd find in `catch` and switch `case`, the code would look like this:

for i where i % 2 == 0 in sequence {
    // do stuff
}

I agree that's really clever and an improvement but after coming up with all the points about wrong expectations about termination vs filtering, the better use of guard, and fetishes about vertical compactness, I think (call it +0.6) I'm going to stick to my guns on this one - and for `for case` too. I've been wuxxed.

* New users might expect the sequence to terminate as soon as i % 2 is 1, rather than the correct interpretation which is "this is a filtering operation"
* The code can be expressed less ambiguously as

for i in sequence.filter({ return i % 2 == 0 }) {
    // do stuff
}

It's important to keep in mind that .filter without using .lazy copies the array. So you need to keep using sequence.lazy.filter({ return i %2 == 0 }), unless you're OK with giving up some performance, which a) adds boilerplate, b) not many people will remember to do.

I've taken the time to run a test, going through milion numbers (several times) using:

for i in arr { if i % 2 == 0 { continue } }
for i in arr where i % 2 == 0 { }
for i in arr.filter({ $0 % 2 == 0 }) { }
for i in arr.lazy.filter({ $0 % 2 == 0 }) { }

Results:

- plain for loop with if-continue: 27.19 seconds (+1.76%)
- with where: 26.72 seconds (+0.00%)
- .filter: 44.73 seconds (+67.40%)
- .lazy.filter: 31.66 seconds (+18.48%)

Yes, 100 milion numbers is an extreme, but it demonstrates that any of the suggested expressions will be slower, mainly if the caller doesn't use .lazy (67% !!!). The only comparable solution is adding additional lines of code into the body of the for loop by adding an if statement.

···

On Jun 9, 2016, at 5:51 AM, Erica Sadun via swift-evolution <swift-evolution@swift.org> wrote:

On Jun 8, 2016, at 9:36 PM, Brent Royal-Gordon <brent@architechies.com <mailto:brent@architechies.com>> wrote:

* The while version can be expressed as

for i in sequence.prefix(while: { return $0 % 2 == 0 } ) {
    // do stuff
}

* The code can also use `guard` statements as needed with `break` and `continue`

(And yes, I should have pointed out filter and prefix as well as guard in my first email)

-- E

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(David Sweeris) #6

What about just this?
for i % 2 == 0 in sequence { ... }

The first new identifier gets to be the index variable and subsequent new identifiers are errors.

- Dave Sweeris

···

On Jun 8, 2016, at 10:51 PM, Erica Sadun via swift-evolution <swift-evolution@swift.org> wrote:

On Jun 8, 2016, at 9:36 PM, Brent Royal-Gordon <brent@architechies.com <mailto:brent@architechies.com>> wrote:

Upon accepting SE-0099, the core team is removing `where` clauses from condition clauses, writing "the 'where' keyword can be retired from its purpose as a boolean condition introducer."

Inspiried by Xiaodi Wu, I now propose removing `where` clauses from `for in` loops, where they are better expressed (and read) as guard conditions.

Do you propose to remove `for case` as well? That can equally be handled by a `guard case` in the loop body.

Alternate proposal: Move `where` clauses to be adjacent to the pattern‚ÄĒrather than the sequence expression‚ÄĒin a `for` loop, just as they are in these other syntaxes.

  for n where n.isOdd in 1...1_000 { … }

This makes them more consistent with the syntax in `switch` cases and `catch` statements, while also IMHO clarifying the role of the `where` clause as a filter on the elements seen by the loop.

I saw your post on that *after* I finished sending this. Moving `where` next to the pattern, like you'd find in `catch` and switch `case`, the code would look like this:

for i where i % 2 == 0 in sequence {
    // do stuff
}


(Xiaodi Wu) #7

Upon accepting SE-0099, the core team is removing `where` clauses from
condition clauses, writing "the 'where' keyword can be retired from its
purpose as a boolean condition introducer."

Inspiried by Xiaodi Wu, I now propose removing `where` clauses from `for
in` loops, where they are better expressed (and read) as guard conditions.

Do you propose to remove `for case` as well? That can equally be handled
by a `guard case` in the loop body.

Alternate proposal: Move `where` clauses to be adjacent to the
pattern‚ÄĒrather than the sequence expression‚ÄĒin a `for` loop, just as they
are in these other syntaxes.

for n where n.isOdd in 1...1_000 { … }

This makes them more consistent with the syntax in `switch` cases and
`catch` statements, while also IMHO clarifying the role of the `where`
clause as a filter on the elements seen by the loop.

I saw your post on that *after* I finished sending this. Moving `where`
next to the pattern, like you'd find in `catch` and switch `case`, the code
would look like this:

for i where i % 2 == 0 in sequence {
    // do stuff
}

This is the best version yet - the placement of 'where' makes total sense
and I really like it there.

I agree that's really clever and an improvement but after coming up with
all the points about wrong expectations about termination vs filtering, the
better use of guard, and fetishes about vertical compactness, I think (call
it +0.6) I'm going to stick to my guns on this one - and for `for case`
too. I've been wuxxed.

Maybe it's the late hour and staring at this too much. For the moment I
think I could live with either not having `where` like Erica proposes or
having it moved like Brent proposes. Perhaps later I'll form a considered
preference.

Brent's idea is so new, yet I have to admit it does feel somehow--this is a
squishy evaluation--satisfying? One thing about it that I like over
previous proposals--that's if we're going to go down this route rather than
taking out `while` altogether--is that the word `in` seems to instinctively
encourage concision. It just feels weird to stuff too much between `for i`
and `in`, so I think people will tend to use it in a more reasonable way
(with nothing to prove this intuition at all, of course).

Then again, it should come as no surprise that I agree with Erica that
removing `while` altogether has the benefit of definitively eliminating any
kind of misinterpretation as to termination vs. filtering. That's a win.

···

On Wed, Jun 8, 2016 at 11:17 PM, Sean Heber via swift-evolution < swift-evolution@swift.org> wrote:

On Jun 8, 2016, at 10:51 PM, Erica Sadun via swift-evolution < > swift-evolution@swift.org> wrote:
On Jun 8, 2016, at 9:36 PM, Brent Royal-Gordon <brent@architechies.com> > wrote:

* New users might expect the sequence to terminate as soon as i % 2 is 1,
rather than the correct interpretation which is "this is a filtering
operation"
* The code can be expressed less ambiguously as

for i in sequence.filter({ return i % 2 == 0 }) {
    // do stuff
}

This seems to trade what was a very declarative syntax about the intent of
some code (especially with 'where' in the middle of the statement) for one
that injects its own specialized vocabulary into the context (knowing what
filter does, a function call, a closure with a return keyword and a pair of
extra braces and parenthesis!) which means, to me anyway, significant
cognitive overhead. It will also be a lot slower without optimization
enabled due to the intermediate array. (I've found *significant* speed ups
switching .forEach() with for loops in debug builds, for example.)

* The while version can be expressed as

for i in sequence.prefix(while: { return $0 % 2 == 0 } ) {
    // do stuff
}

And now we've gone from, again, what is likely a very simple and
declarative style using a for/while kind of statement and turned it in to
something that has *even more* cognitive overhead to figure out what it
does because now I have to reason about what "prefix" means here (normally
I only think of prefix in the context of strings) and if there's a special
variation of it using the "while" argument that I need to also be aware
of...

Maybe it's just me, but.. I don't get it. I want to be able to quickly
understand a piece of code's intent, not wade through fancy constructions
for their own sake.

l8r
Sean - who might be too tired to be emailing responsibly

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Brent Royal-Gordon) #8

I've taken the time to run a test, going through milion numbers (several times) using:

for i in arr { if i % 2 == 0 { continue } }
for i in arr where i % 2 == 0 { }
for i in arr.filter({ $0 % 2 == 0 }) { }
for i in arr.lazy.filter({ $0 % 2 == 0 }) { }

Results:

- plain for loop with if-continue: 27.19 seconds (+1.76%)
- with where: 26.72 seconds (+0.00%)
- .filter: 44.73 seconds (+67.40%)
- .lazy.filter: 31.66 seconds (+18.48%)

This is great data. I have a hard time imagining a little compiler work couldn't make if-continue as fast as for-where, but lazy.filter might be a taller order for it, and optimizing plain filter could actually change behavior.

A month or two ago, I actually fell into the "just use the higher-order functions" camp on this question, but I've been rethinking that more and more lately. Between the trailing closure incompatibility, the need to remember to use `lazy` to get decent performance, and now the noticeable speed difference even *with* lazy, I'm no longer convinced that answer is good enough.

(Though I do think `while` is probably too niche to bother with as a first-class feature, and I am open to if-continue on the `where` clause.)

···

--
Brent Royal-Gordon
Architechies


(Jordan Rose) #9

Just to double-check, was this with optimizations on? Because -Onone numbers aren’t nearly as motivating, but I would expect -O to remove the loop entirely in the simple case.

Jordan

···

On Jun 8, 2016, at 22:19, Charlie Monroe via swift-evolution <swift-evolution@swift.org> wrote:

On Jun 9, 2016, at 5:51 AM, Erica Sadun via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Jun 8, 2016, at 9:36 PM, Brent Royal-Gordon <brent@architechies.com <mailto:brent@architechies.com>> wrote:

Upon accepting SE-0099, the core team is removing `where` clauses from condition clauses, writing "the 'where' keyword can be retired from its purpose as a boolean condition introducer."

Inspiried by Xiaodi Wu, I now propose removing `where` clauses from `for in` loops, where they are better expressed (and read) as guard conditions.

Do you propose to remove `for case` as well? That can equally be handled by a `guard case` in the loop body.

Alternate proposal: Move `where` clauses to be adjacent to the pattern‚ÄĒrather than the sequence expression‚ÄĒin a `for` loop, just as they are in these other syntaxes.

  for n where n.isOdd in 1...1_000 { … }

This makes them more consistent with the syntax in `switch` cases and `catch` statements, while also IMHO clarifying the role of the `where` clause as a filter on the elements seen by the loop.

I saw your post on that *after* I finished sending this. Moving `where` next to the pattern, like you'd find in `catch` and switch `case`, the code would look like this:

for i where i % 2 == 0 in sequence {
    // do stuff
}

I agree that's really clever and an improvement but after coming up with all the points about wrong expectations about termination vs filtering, the better use of guard, and fetishes about vertical compactness, I think (call it +0.6) I'm going to stick to my guns on this one - and for `for case` too. I've been wuxxed.

* New users might expect the sequence to terminate as soon as i % 2 is 1, rather than the correct interpretation which is "this is a filtering operation"
* The code can be expressed less ambiguously as

for i in sequence.filter({ return i % 2 == 0 }) {
    // do stuff
}

It's important to keep in mind that .filter without using .lazy copies the array. So you need to keep using sequence.lazy.filter({ return i %2 == 0 }), unless you're OK with giving up some performance, which a) adds boilerplate, b) not many people will remember to do.

I've taken the time to run a test, going through milion numbers (several times) using:

for i in arr { if i % 2 == 0 { continue } }
for i in arr where i % 2 == 0 { }
for i in arr.filter({ $0 % 2 == 0 }) { }
for i in arr.lazy.filter({ $0 % 2 == 0 }) { }

Results:

- plain for loop with if-continue: 27.19 seconds (+1.76%)
- with where: 26.72 seconds (+0.00%)
- .filter: 44.73 seconds (+67.40%)
- .lazy.filter: 31.66 seconds (+18.48%)

Yes, 100 milion numbers is an extreme, but it demonstrates that any of the suggested expressions will be slower, mainly if the caller doesn't use .lazy (67% !!!). The only comparable solution is adding additional lines of code into the body of the for loop by adding an if statement.


(Erica Sadun) #10

I really like how it aligns with case statements better. BUT I think using guard is still better because
there's still user confusion between "does this terminate" and "does this filter".

Guard use is unambiguous.

-- E

···

On Jun 8, 2016, at 10:17 PM, Sean Heber <sean@fifthace.com> wrote:

On Jun 8, 2016, at 10:51 PM, Erica Sadun via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I saw your post on that *after* I finished sending this. Moving `where` next to the pattern, like you'd find in `catch` and switch `case`, the code would look like this:

for i where i % 2 == 0 in sequence {
    // do stuff
}

This is the best version yet - the placement of 'where' makes total sense and I really like it there.


(Charlie Monroe) #11

I've taken the time to run a test, going through milion numbers (several times) using:

for i in arr { if i % 2 == 0 { continue } }
for i in arr where i % 2 == 0 { }
for i in arr.filter({ $0 % 2 == 0 }) { }
for i in arr.lazy.filter({ $0 % 2 == 0 }) { }

Results:

- plain for loop with if-continue: 27.19 seconds (+1.76%)
- with where: 26.72 seconds (+0.00%)
- .filter: 44.73 seconds (+67.40%)
- .lazy.filter: 31.66 seconds (+18.48%)

This is great data. I have a hard time imagining a little compiler work couldn't make if-continue as fast as for-where, but lazy.filter might be a taller order for it, and optimizing plain filter could actually change behavior.

A month or two ago, I actually fell into the "just use the higher-order functions" camp on this question, but I've been rethinking that more and more lately. Between the trailing closure incompatibility, the need to remember to use `lazy` to get decent performance, and now the noticeable speed difference even *with* lazy, I'm no longer convinced that answer is good enough.

There will IMHO always be noticeable overhead since you're calling a function which is then invoking a closure. When you look at what that means:

- thunks generated around the invocation, which are a few instructions
- new stack frame for each call (correct me if I'm wrong).

So instead of a single `i % 2 == 0` (which is just 2-3 instructions, depending on the architecture and optimization settings), it will invoke the closure milion times, if the array contains a milion members.

Maybe I'm over-optimizing, but 18% seemed like a lot to me.

···

On Jun 9, 2016, at 10:29 AM, Brent Royal-Gordon <brent@architechies.com> wrote:

(Though I do think `while` is probably too niche to bother with as a first-class feature, and I am open to if-continue on the `where` clause.)

--
Brent Royal-Gordon
Architechies


(Brent Royal-Gordon) #12

What about just this?
for i % 2 == 0 in sequence { ... }

The first new identifier gets to be the index variable and subsequent new identifiers are errors.

If that weren't a syntax error, I would expect it to loop over `[true, false, true, false, …]`.

···

--
Brent Royal-Gordon
Architechies


(Xiaodi Wu) #13

Upon accepting SE-0099, the core team is removing `where` clauses from
condition clauses, writing "the 'where' keyword can be retired from its
purpose as a boolean condition introducer."

Inspiried by Xiaodi Wu, I now propose removing `where` clauses from `for
in` loops, where they are better expressed (and read) as guard conditions.

Do you propose to remove `for case` as well? That can equally be handled
by a `guard case` in the loop body.

Alternate proposal: Move `where` clauses to be adjacent to the
pattern‚ÄĒrather than the sequence expression‚ÄĒin a `for` loop, just as they
are in these other syntaxes.

for n where n.isOdd in 1...1_000 { … }

This makes them more consistent with the syntax in `switch` cases and
`catch` statements, while also IMHO clarifying the role of the `where`
clause as a filter on the elements seen by the loop.

I saw your post on that *after* I finished sending this. Moving `where`
next to the pattern, like you'd find in `catch` and switch `case`, the code
would look like this:

for i where i % 2 == 0 in sequence {
    // do stuff
}

This is the best version yet - the placement of 'where' makes total sense
and I really like it there.

I agree that's really clever and an improvement but after coming up with
all the points about wrong expectations about termination vs filtering, the
better use of guard, and fetishes about vertical compactness, I think (call
it +0.6) I'm going to stick to my guns on this one - and for `for case`
too. I've been wuxxed.

Maybe it's the late hour and staring at this too much. For the moment I
think I could live with either not having `where` like Erica proposes or
having it moved like Brent proposes. Perhaps later I'll form a considered
preference.

Brent's idea is so new, yet I have to admit it does feel somehow--this is
a squishy evaluation--satisfying? One thing about it that I like over
previous proposals--that's if we're going to go down this route rather than
taking out `while` altogether--is that the word `in` seems to instinctively
encourage concision. It just feels weird to stuff too much between `for i`
and `in`, so I think people will tend to use it in a more reasonable way
(with nothing to prove this intuition at all, of course).

Then again, it should come as no surprise that I agree with Erica that
removing `while` altogether has the benefit of definitively eliminating any
kind of misinterpretation as to termination vs. filtering. That's a win.

Yikes: s/while/where. That's my queue to quit for the day.

···

On Wed, Jun 8, 2016 at 11:34 PM, Xiaodi Wu <xiaodi.wu@gmail.com> wrote:

On Wed, Jun 8, 2016 at 11:17 PM, Sean Heber via swift-evolution < > swift-evolution@swift.org> wrote:

On Jun 8, 2016, at 10:51 PM, Erica Sadun via swift-evolution < >> swift-evolution@swift.org> wrote:
On Jun 8, 2016, at 9:36 PM, Brent Royal-Gordon <brent@architechies.com> >> wrote:

* New users might expect the sequence to terminate as soon as i % 2 is 1,
rather than the correct interpretation which is "this is a filtering
operation"
* The code can be expressed less ambiguously as

for i in sequence.filter({ return i % 2 == 0 }) {
    // do stuff
}

This seems to trade what was a very declarative syntax about the intent
of some code (especially with 'where' in the middle of the statement) for
one that injects its own specialized vocabulary into the context (knowing
what filter does, a function call, a closure with a return keyword and a
pair of extra braces and parenthesis!) which means, to me anyway,
significant cognitive overhead. It will also be a lot slower without
optimization enabled due to the intermediate array. (I've found
*significant* speed ups switching .forEach() with for loops in debug
builds, for example.)

* The while version can be expressed as

for i in sequence.prefix(while: { return $0 % 2 == 0 } ) {
    // do stuff
}

And now we've gone from, again, what is likely a very simple and
declarative style using a for/while kind of statement and turned it in to
something that has *even more* cognitive overhead to figure out what it
does because now I have to reason about what "prefix" means here (normally
I only think of prefix in the context of strings) and if there's a special
variation of it using the "while" argument that I need to also be aware
of...

Maybe it's just me, but.. I don't get it. I want to be able to quickly
understand a piece of code's intent, not wade through fancy constructions
for their own sake.

l8r
Sean - who might be too tired to be emailing responsibly

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Erica Sadun) #14

So how did the guard versions perform?

···

Sent from my iPad

On Jun 9, 2016, at 4:27 AM, Charlie Monroe <charlie@charliemonroe.net> wrote:

On Jun 9, 2016, at 10:29 AM, Brent Royal-Gordon <brent@architechies.com> wrote:

I've taken the time to run a test, going through milion numbers (several times) using:

for i in arr { if i % 2 == 0 { continue } }
for i in arr where i % 2 == 0 { }
for i in arr.filter({ $0 % 2 == 0 }) { }
for i in arr.lazy.filter({ $0 % 2 == 0 }) { }

Results:

- plain for loop with if-continue: 27.19 seconds (+1.76%)
- with where: 26.72 seconds (+0.00%)
- .filter: 44.73 seconds (+67.40%)
- .lazy.filter: 31.66 seconds (+18.48%)

This is great data. I have a hard time imagining a little compiler work couldn't make if-continue as fast as for-where, but lazy.filter might be a taller order for it, and optimizing plain filter could actually change behavior.

A month or two ago, I actually fell into the "just use the higher-order functions" camp on this question, but I've been rethinking that more and more lately. Between the trailing closure incompatibility, the need to remember to use `lazy` to get decent performance, and now the noticeable speed difference even *with* lazy, I'm no longer convinced that answer is good enough.

There will IMHO always be noticeable overhead since you're calling a function which is then invoking a closure. When you look at what that means:

- thunks generated around the invocation, which are a few instructions
- new stack frame for each call (correct me if I'm wrong).

So instead of a single `i % 2 == 0` (which is just 2-3 instructions, depending on the architecture and optimization settings), it will invoke the closure milion times, if the array contains a milion members.

Maybe I'm over-optimizing, but 18% seemed like a lot to me.

(Though I do think `while` is probably too niche to bother with as a first-class feature, and I am open to if-continue on the `where` clause.)

--
Brent Royal-Gordon
Architechies


(Erica Sadun) #15

My results:

-Onone (None)

plain for loop with guard
Elapsed time: 0.0563530325889587
plain for loop with if
Elapsed time: 0.0631130337715149
where test
Elapsed time: 0.0661619901657104
eager filter test
Elapsed time: 0.684610962867737
lazy filter test
Elapsed time: 0.640420973300934
Program ended with exit code: 0

-O (Fast)

plain for loop with guard
Elapsed time: 0.00411999225616455
plain for loop with if
Elapsed time: 0.00422400236129761
where test
Elapsed time: 0.00419700145721436
eager filter test
Elapsed time: 0.033439040184021
lazy filter test
Elapsed time: 0.00690501928329468
Program ended with exit code: 0

Code:

public func timetest(_ note: String, block: () -> Void) {
    let date = NSDate()
    block()
    let timeInterval = NSDate().timeIntervalSince(date)
    print(note); print("Elapsed time: \(timeInterval)")
}

let count = 4_000_000
let range = 1...count

timetest("plain for loop with guard") {
    for i in range {
        guard i % 2 != 0 else { continue }
        doSomething()
    }
}

timetest("plain for loop with if") {
    for i in range {
        if i % 2 == 0 { continue }
        doSomething()
    }
}

timetest("where test") {
    for i in range where i % 2 == 0 {
        doSomething()
    }
}

timetest("eager filter test") {
    for i in range.filter({ $0 % 2 == 0 }) {
        doSomething()
    }
}

timetest("lazy filter test") {
    for i in range.lazy.filter({ $0 % 2 == 0 }) {
        doSomething()
    }
}

···

On Jun 9, 2016, at 4:27 AM, Charlie Monroe <charlie@charliemonroe.net> wrote:

On Jun 9, 2016, at 10:29 AM, Brent Royal-Gordon <brent@architechies.com> wrote:

I've taken the time to run a test, going through milion numbers (several times) using:

for i in arr { if i % 2 == 0 { continue } }
for i in arr where i % 2 == 0 { }
for i in arr.filter({ $0 % 2 == 0 }) { }
for i in arr.lazy.filter({ $0 % 2 == 0 }) { }

Results:

- plain for loop with if-continue: 27.19 seconds (+1.76%)
- with where: 26.72 seconds (+0.00%)
- .filter: 44.73 seconds (+67.40%)
- .lazy.filter: 31.66 seconds (+18.48%)

This is great data. I have a hard time imagining a little compiler work couldn't make if-continue as fast as for-where, but lazy.filter might be a taller order for it, and optimizing plain filter could actually change behavior.

A month or two ago, I actually fell into the "just use the higher-order functions" camp on this question, but I've been rethinking that more and more lately. Between the trailing closure incompatibility, the need to remember to use `lazy` to get decent performance, and now the noticeable speed difference even *with* lazy, I'm no longer convinced that answer is good enough.

There will IMHO always be noticeable overhead since you're calling a function which is then invoking a closure. When you look at what that means:

- thunks generated around the invocation, which are a few instructions
- new stack frame for each call (correct me if I'm wrong).

So instead of a single `i % 2 == 0` (which is just 2-3 instructions, depending on the architecture and optimization settings), it will invoke the closure milion times, if the array contains a milion members.

Maybe I'm over-optimizing, but 18% seemed like a lot to me.

(Though I do think `while` is probably too niche to bother with as a first-class feature, and I am open to if-continue on the `where` clause.)

--
Brent Royal-Gordon
Architechies


(Erica Sadun) #16

Missed pasting this one bit:

var value = 0
func doSomething() {
    // some numeric load
    value += 1
}

···

On Jun 9, 2016, at 9:59 AM, Erica Sadun via swift-evolution <swift-evolution@swift.org> wrote:

My results:

-Onone (None)

plain for loop with guard
Elapsed time: 0.0563530325889587
plain for loop with if
Elapsed time: 0.0631130337715149
where test
Elapsed time: 0.0661619901657104
eager filter test
Elapsed time: 0.684610962867737
lazy filter test
Elapsed time: 0.640420973300934
Program ended with exit code: 0

-O (Fast)

plain for loop with guard
Elapsed time: 0.00411999225616455
plain for loop with if
Elapsed time: 0.00422400236129761
where test
Elapsed time: 0.00419700145721436
eager filter test
Elapsed time: 0.033439040184021
lazy filter test
Elapsed time: 0.00690501928329468
Program ended with exit code: 0

Code:

public func timetest(_ note: String, block: () -> Void) {
    let date = NSDate()
    block()
    let timeInterval = NSDate().timeIntervalSince(date)
    print(note); print("Elapsed time: \(timeInterval)")
}

let count = 4_000_000
let range = 1...count

timetest("plain for loop with guard") {
    for i in range {
        guard i % 2 != 0 else { continue }
        doSomething()
    }
}

timetest("plain for loop with if") {
    for i in range {
        if i % 2 == 0 { continue }
        doSomething()
    }
}

timetest("where test") {
    for i in range where i % 2 == 0 {
        doSomething()
    }
}

timetest("eager filter test") {
    for i in range.filter({ $0 % 2 == 0 }) {
        doSomething()
    }
}

timetest("lazy filter test") {
    for i in range.lazy.filter({ $0 % 2 == 0 }) {
        doSomething()
    }
}

On Jun 9, 2016, at 4:27 AM, Charlie Monroe <charlie@charliemonroe.net <mailto:charlie@charliemonroe.net>> wrote:

On Jun 9, 2016, at 10:29 AM, Brent Royal-Gordon <brent@architechies.com <mailto:brent@architechies.com>> wrote:

I've taken the time to run a test, going through milion numbers (several times) using:

for i in arr { if i % 2 == 0 { continue } }
for i in arr where i % 2 == 0 { }
for i in arr.filter({ $0 % 2 == 0 }) { }
for i in arr.lazy.filter({ $0 % 2 == 0 }) { }

Results:

- plain for loop with if-continue: 27.19 seconds (+1.76%)
- with where: 26.72 seconds (+0.00%)
- .filter: 44.73 seconds (+67.40%)
- .lazy.filter: 31.66 seconds (+18.48%)

This is great data. I have a hard time imagining a little compiler work couldn't make if-continue as fast as for-where, but lazy.filter might be a taller order for it, and optimizing plain filter could actually change behavior.

A month or two ago, I actually fell into the "just use the higher-order functions" camp on this question, but I've been rethinking that more and more lately. Between the trailing closure incompatibility, the need to remember to use `lazy` to get decent performance, and now the noticeable speed difference even *with* lazy, I'm no longer convinced that answer is good enough.

There will IMHO always be noticeable overhead since you're calling a function which is then invoking a closure. When you look at what that means:

- thunks generated around the invocation, which are a few instructions
- new stack frame for each call (correct me if I'm wrong).

So instead of a single `i % 2 == 0` (which is just 2-3 instructions, depending on the architecture and optimization settings), it will invoke the closure milion times, if the array contains a milion members.

Maybe I'm over-optimizing, but 18% seemed like a lot to me.

(Though I do think `while` is probably too niche to bother with as a first-class feature, and I am open to if-continue on the `where` clause.)

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(L Mihalkovic) #17

I've taken the time to run a test, going through milion numbers (several times) using:

for i in arr { if i % 2 == 0 { continue } }
for i in arr where i % 2 == 0 { }
for i in arr.filter({ $0 % 2 == 0 }) { }
for i in arr.lazy.filter({ $0 % 2 == 0 }) { }

Results:

- plain for loop with if-continue: 27.19 seconds (+1.76%)
- with where: 26.72 seconds (+0.00%)
- .filter: 44.73 seconds (+67.40%)
- .lazy.filter: 31.66 seconds (+18.48%)

This is great data. I have a hard time imagining a little compiler work couldn't make if-continue as fast as for-where, but lazy.filter might be a taller order for it, and optimizing plain filter could actually change behavior.

A month or two ago, I actually fell into the "just use the higher-order functions" camp on this question, but I've been rethinking that more and more lately. Between the trailing closure incompatibility, the need to remember to use `lazy` to get decent performance, and now the noticeable speed difference even *with* lazy, I'm no longer convinced that answer is good enough.

There will IMHO always be noticeable overhead since you're calling a function which is then invoking a closure. When you look at what that means:

- thunks generated around the invocation, which are a few instructions
- new stack frame for each call (correct me if I'm wrong).

So instead of a single `i % 2 == 0` (which is just 2-3 instructions, depending on the architecture and optimization settings), it will invoke the closure milion times, if the array contains a milion members.

Maybe I'm over-optimizing, but 18% seemed like a lot to me.

It looks like this should not be fate, and a pattern that could end up generating the same code after proper inlining.

···

On Jun 9, 2016, at 12:27 PM, Charlie Monroe via swift-evolution <swift-evolution@swift.org> wrote:

On Jun 9, 2016, at 10:29 AM, Brent Royal-Gordon <brent@architechies.com> wrote:

(Though I do think `while` is probably too niche to bother with as a first-class feature, and I am open to if-continue on the `where` clause.)

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Charlie Monroe) #18

Didn't originally do it, but added it and here are the results (in seconds):

-O none
guard: 27.8599720001221 (+0.08%)
if-continue: 29.0756570100784 (+4.00%)
where: 27.836905002594 (+0.00%)
filter: 46.8083620071411 (+68.15%)
lazy.filter: 33.3811990022659 (+19.91%)

-O fast

guard: 0.123715996742249 (+4.00%)
if-continue: 0.118164002895355 (+0.00%)
where: 0.118863999843597 (+0.59%)
filter: 0.520934045314789 (+40.85%)
lazy.filter: 0.132100999355316 (+11.79%)

Note that in order to prevent some compiler magic for ranges (not sure if there is any), I've done the following:

let arr = Array(range)

Also, I invoke the block() within timetest 100 times to get larger values, since there can be some minor changes in calling NSDate() - the first time it's called, +initialize may get called as well as the IMP might not be cached and with 0.04 seconds, it can make the result a bit off due to a lock during initialization of NSDate.

Nevertheless my argument against removing where from the for loops is that:

1) Without boilterplate guard/if, the performance is poor.
2) With the boilerplate, you add 3 lines of code. I do not like the one-liner if condition { continue }
3) I'd vote for keeping where next to the variable name as suggested by Brent.

···

On Jun 9, 2016, at 4:22 PM, Erica Sadun <erica@ericasadun.com> wrote:

So how did the guard versions perform?

Sent from my iPad

On Jun 9, 2016, at 4:27 AM, Charlie Monroe <charlie@charliemonroe.net> wrote:

On Jun 9, 2016, at 10:29 AM, Brent Royal-Gordon <brent@architechies.com> wrote:

I've taken the time to run a test, going through milion numbers (several times) using:

for i in arr { if i % 2 == 0 { continue } }
for i in arr where i % 2 == 0 { }
for i in arr.filter({ $0 % 2 == 0 }) { }
for i in arr.lazy.filter({ $0 % 2 == 0 }) { }

Results:

- plain for loop with if-continue: 27.19 seconds (+1.76%)
- with where: 26.72 seconds (+0.00%)
- .filter: 44.73 seconds (+67.40%)
- .lazy.filter: 31.66 seconds (+18.48%)

This is great data. I have a hard time imagining a little compiler work couldn't make if-continue as fast as for-where, but lazy.filter might be a taller order for it, and optimizing plain filter could actually change behavior.

A month or two ago, I actually fell into the "just use the higher-order functions" camp on this question, but I've been rethinking that more and more lately. Between the trailing closure incompatibility, the need to remember to use `lazy` to get decent performance, and now the noticeable speed difference even *with* lazy, I'm no longer convinced that answer is good enough.

There will IMHO always be noticeable overhead since you're calling a function which is then invoking a closure. When you look at what that means:

- thunks generated around the invocation, which are a few instructions
- new stack frame for each call (correct me if I'm wrong).

So instead of a single `i % 2 == 0` (which is just 2-3 instructions, depending on the architecture and optimization settings), it will invoke the closure milion times, if the array contains a milion members.

Maybe I'm over-optimizing, but 18% seemed like a lot to me.

(Though I do think `while` is probably too niche to bother with as a first-class feature, and I am open to if-continue on the `where` clause.)

--
Brent Royal-Gordon
Architechies


(Charlie Monroe) #19

See my latest post - included results with -Ofast. But still, using filter and lazy.filter is 10+% slower, which were the suggested alternatives to `where`.

···

On Jun 9, 2016, at 6:54 PM, Jordan Rose <jordan_rose@apple.com> wrote:

On Jun 8, 2016, at 22:19, Charlie Monroe via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Jun 9, 2016, at 5:51 AM, Erica Sadun via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Jun 8, 2016, at 9:36 PM, Brent Royal-Gordon <brent@architechies.com <mailto:brent@architechies.com>> wrote:

Upon accepting SE-0099, the core team is removing `where` clauses from condition clauses, writing "the 'where' keyword can be retired from its purpose as a boolean condition introducer."

Inspiried by Xiaodi Wu, I now propose removing `where` clauses from `for in` loops, where they are better expressed (and read) as guard conditions.

Do you propose to remove `for case` as well? That can equally be handled by a `guard case` in the loop body.

Alternate proposal: Move `where` clauses to be adjacent to the pattern‚ÄĒrather than the sequence expression‚ÄĒin a `for` loop, just as they are in these other syntaxes.

  for n where n.isOdd in 1...1_000 { … }

This makes them more consistent with the syntax in `switch` cases and `catch` statements, while also IMHO clarifying the role of the `where` clause as a filter on the elements seen by the loop.

I saw your post on that *after* I finished sending this. Moving `where` next to the pattern, like you'd find in `catch` and switch `case`, the code would look like this:

for i where i % 2 == 0 in sequence {
    // do stuff
}

I agree that's really clever and an improvement but after coming up with all the points about wrong expectations about termination vs filtering, the better use of guard, and fetishes about vertical compactness, I think (call it +0.6) I'm going to stick to my guns on this one - and for `for case` too. I've been wuxxed.

* New users might expect the sequence to terminate as soon as i % 2 is 1, rather than the correct interpretation which is "this is a filtering operation"
* The code can be expressed less ambiguously as

for i in sequence.filter({ return i % 2 == 0 }) {
    // do stuff
}

It's important to keep in mind that .filter without using .lazy copies the array. So you need to keep using sequence.lazy.filter({ return i %2 == 0 }), unless you're OK with giving up some performance, which a) adds boilerplate, b) not many people will remember to do.

I've taken the time to run a test, going through milion numbers (several times) using:

for i in arr { if i % 2 == 0 { continue } }
for i in arr where i % 2 == 0 { }
for i in arr.filter({ $0 % 2 == 0 }) { }
for i in arr.lazy.filter({ $0 % 2 == 0 }) { }

Results:

- plain for loop with if-continue: 27.19 seconds (+1.76%)
- with where: 26.72 seconds (+0.00%)
- .filter: 44.73 seconds (+67.40%)
- .lazy.filter: 31.66 seconds (+18.48%)

Yes, 100 milion numbers is an extreme, but it demonstrates that any of the suggested expressions will be slower, mainly if the caller doesn't use .lazy (67% !!!). The only comparable solution is adding additional lines of code into the body of the for loop by adding an if statement.

Just to double-check, was this with optimizations on? Because -Onone numbers aren’t nearly as motivating, but I would expect -O to remove the loop entirely in the simple case.

Jordan


(Ben Rimmington) #20

Erica Sadun wrote:

public func timetest(_ note: String, block: () -> Void) {
    let date = NSDate()
    block()
    let timeInterval = NSDate().timeIntervalSince(date)
    print(note); print("Elapsed time: \(timeInterval)")
}

For performance testing, it might be better to use:

* XCTestCase's `measureBlock {...}`, which uses
* `NSProcessInfo.processInfo().systemUptime`, which uses
* `mach_absolute_time()` or `clock_gettime(CLOCK_MONOTONIC, ...)`

[swift-corelibs-foundation and swift-corelibs-xctest]

-- Ben