[Draft] Clarify interaction between comments & operators

Hi All,

Last month, I started a discussion about resolving some inconsistencies in the way operators interact with comments. I received some good feedback and then the break happened. I’ve since updated the draft proposal in a few small ways and would appreciate any additional comments folks have before submitting it for review.

The proposal is reproduced below and lives here: swift-evolution/0000-clarify-comments-and-operators.md at comments-and-operators · jder/swift-evolution · GitHub

Thanks!
Jesse

Clarify interaction between comments & operators

Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md&gt;
Author(s): Jesse Rusak <https://github.com/jder&gt;
Status: Review
Review manager: TBD
<swift-evolution/0000-clarify-comments-and-operators.md at comments-and-operators · jder/swift-evolution · GitHub

There are several inconsistencies in how comments are treated when determining whether an operator is prefix, postfix, or infix. They are sometimes treated as whitespace, sometimes as non-whitespace, and this differs depending on whether they are to the left or right of an operator, and the contents of the comment itself. This proposal suggests a uniform set of rules for how these cases should be parsed.

Swift-evolution thread: started here <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003780.html&gt; and continued here <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/003913.html&gt; and continued here <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151228/004646.html&gt;\.

<swift-evolution/0000-clarify-comments-and-operators.md at comments-and-operators · jder/swift-evolution · GitHub

At the moment, comments next to operators are usually treated as non-whitespace for the purpose of determining whether an operator is prefix/postfix/binary <Lexical Structure — The Swift Programming Language (Swift 5.7), meaning that this fails to compile:

if /* comment */!foo { ... }
Because the "!" is parsed as binary operator (no whitespace on either side), rather than as a prefix operator, which seems undesirable. This behavior is also not consistently applied. For example, this currently works:

1 +/* comment */2
Because the "+/*" is treated as one token and sees the whitespace to its right and left, and so is parsed as a binary operator.

In order to resolve these and related issues, this proposes a general rule about the expected behavior.

<swift-evolution/0000-clarify-comments-and-operators.md at comments-and-operators · jder/swift-evolution · GitHub solution

Comments should be treated as absent for all of the purposes in the “operators” section of the swift language reference: determining whether an operator is binary, prefix, or postfix, as well as the special rules around the “!” and “?” predefined operators. In other words, operators should "see through" a comment to the characters on the other side.

This means that adding a comment next to an operator (including between it and its operand) should not change whether the operator is treated as prefix, postfix, or binary, regardless of the contents of the comment.

For example, these should all be equivalent:

if !foo { ... }
if /* comment */!foo { ... }
if !/* comment */foo { ... }
As should these:

// whitespace on both sides
1 + 2
1 +/* comment */ 2

// no whitespace on either side
1+/*comment*/2
1+/* comment
comment */2
This is a predictable model, and is intended to be as unsurprising as possible, especially to beginners that are not used to troubleshooting parse errors.

<swift-evolution/0000-clarify-comments-and-operators.md at comments-and-operators · jder/swift-evolution · GitHub design

When parsing an operator character and trying to determine whether it has a whitespace character to either the right or left, we should skip comments (both possibly-nested slash-star comments, and slash-slash comments). The contents of the skipped comments (e.g. whether they include newlines) should have no effect on this determination.

For this purpose, slash-slash comments should be treated as extending up to but not including the trailing newline character (if present). So, for example, this should be treated as a postfix "~" operator:

let a = foo~// comment
bar()
The newline character after the end of the comment means the "~" has whitespace to its right, not bar. On the other hand, this should be parsed as a binary operator:

let a = foo~/* commment
*/bar()
As the newline appears only within the comment. (Note that this only matters for operators other than "!" and "?" because of their special rules.)

The language reference should also be updated to make clear that comments are ignored for these purposes.

<swift-evolution/0000-clarify-comments-and-operators.md at comments-and-operators · jder/swift-evolution · GitHub on existing code

Only code with comments immediately next to operators will be affected. This is not expected to be very common, and could be fixed by adding/removing whitespace or moving the comment outside of the expression. It would probably be possible to produce fix-its for these. Here are some examples of the changes.

Some cases which would previously work will now produce an error (these are breaking changes):

1 /* */+2
1 +/* comment */2
1+/*comment*/ 2
Some cases which were previously errors will now work:

/* */!foo
!/* */foo

1+/* */2
1 /**/+ 2
Examples of things which will continue to be errors:

1/**/+ 2
1 +/*comment*/2
And things which will continue to work:

foo!// comment
foo/* */?.description
foo/* */!

1 +/**/ 2
1/**/+2
1+/**/2
<swift-evolution/0000-clarify-comments-and-operators.md at comments-and-operators · jder/swift-evolution · GitHub considered

<swift-evolution/0000-clarify-comments-and-operators.md at comments-and-operators · jder/swift-evolution · GitHub comments as whitespace

We could instead specify that comments are treated as whitespace. This is a simpler rule, and it avoids some extra complications described above around slash-slash comments and multi-line slash-star comments. It is also easier for both the lexer and a human reader to determine whether an operator is binary or not, since if comments are always whitespace, you don't have to scan to the other side of a long comment to tell whether an operator has whitespace around it. For example:

1 +/* a very long comment */2
With the proposed changes, you can't tell just by looking near the “+” whether it is a binary or prefix operator.

On the other hand, this goes against the common mental model that comments are ignored when parsing. As a result, this is somewhat more surprising rule. This rule is also less flexible; that is, there are places which this completely prohibits comments which are permitted by the above proposal. (e.g. !/* */foo)

<swift-evolution/0000-clarify-comments-and-operators.md at comments-and-operators · jder/swift-evolution · GitHub more general rule

Another alternative is a more general rule about how comments are handled everywhere in the language (e.g. there must be no effect when replacing a comment with a space character). This has the advantage of possibly resolving other ambiguities, but has potentially far-reaching consequences in various edge cases which are hard to fully enumerate (e.g. multi-line comments, comments within interpolated sequences inside of string literals, comments in lines which contain "#" directives, etc).

I think treat the comments as absence is a good idea. As the comments are
only for the programmers not for the compiler.

Besides, I think one rule should be added. The comments that are used to
generate the help documents or #pragma like should always be written in
separated lines and should never mixed in the code lines.

zhaoxin

···

On Sun, Jan 10, 2016 at 12:50 AM, Jesse Rusak via swift-evolution < swift-evolution@swift.org> wrote:

Hi All,

Last month, I started a discussion about resolving some inconsistencies in
the way operators interact with comments. I received some good feedback and
then the break happened. I’ve since updated the draft proposal in a few
small ways and would appreciate any additional comments folks have before
submitting it for review.

The proposal is reproduced below and lives here:
https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md

Thanks!
Jesse

Clarify interaction between comments & operators

   - Proposal: SE-NNNN
   <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md&gt;
   - Author(s): Jesse Rusak <https://github.com/jder&gt;
   - Status: *Review*
   - Review manager: TBD

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#introduction&gt;
Introduction

There are several inconsistencies in how comments are treated when
determining whether an operator is prefix, postfix, or infix. They are
sometimes treated as whitespace, sometimes as non-whitespace, and this
differs depending on whether they are to the left or right of an operator,
and the contents of the comment itself. This proposal suggests a uniform
set of rules for how these cases should be parsed.

Swift-evolution thread: started here
<https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003780.html&gt;
and continued here
<https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/003913.html&gt;
and continued here
<https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151228/004646.html&gt;
.

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#motivation&gt;
Motivation

At the moment, comments next to operators are usually treated as
non-whitespace for the purpose of determining whether an operator is
prefix/postfix/binary
<The Swift Programming Language: Redirect,
meaning that this fails to compile:

if /* comment */!foo { ... }

Because the "!" is parsed as binary operator (no whitespace on either
side), rather than as a prefix operator, which seems undesirable. This
behavior is also not consistently applied. For example, this currently
works:

1 +/* comment */2

Because the "+/*" is treated as one token and sees the whitespace to its
right and left, and so is parsed as a binary operator.

In order to resolve these and related issues, this proposes a general rule
about the expected behavior.

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#proposed-solution&gt;Proposed
solution

Comments should be treated as absent for all of the purposes in the
“operators” section of the swift language reference: determining whether an
operator is binary, prefix, or postfix, as well as the special rules around
the “!” and “?” predefined operators. In other words, operators should "see
through" a comment to the characters on the other side.

This means that adding a comment next to an operator (including between it
and its operand) should not change whether the operator is treated as
prefix, postfix, or binary, regardless of the contents of the comment.

For example, these should all be equivalent:

if !foo { ... }if /* comment */!foo { ... }if !/* comment */foo { ... }

As should these:

// whitespace on both sides1 + 21 +/* comment */ 2
// no whitespace on either side1+/*comment*/21+/* comment comment */2

This is a predictable model, and is intended to be as unsurprising as
possible, especially to beginners that are not used to troubleshooting
parse errors.

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#detailed-design&gt;Detailed
design

When parsing an operator character and trying to determine whether it has
a whitespace character to either the right or left, we should skip comments
(both possibly-nested slash-star comments, and slash-slash comments). The
contents of the skipped comments (e.g. whether they include newlines)
should have no effect on this determination.

For this purpose, slash-slash comments should be treated as extending up
to but not including the trailing newline character (if present). So, for
example, this should be treated as a postfix "~" operator:

let a = foo~// comment
bar()

The newline character after the end of the comment means the "~" has
whitespace to its right, not bar. On the other hand, this should be
parsed as a binary operator:

let a = foo~/* commment*/bar()

As the newline appears only within the comment. (Note that this only
matters for operators other than "!" and "?" because of their special
rules.)

The language reference should also be updated to make clear that comments
are ignored for these purposes.

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#impact-on-existing-code&gt;Impact
on existing code

Only code with comments immediately next to operators will be affected.
This is not expected to be very common, and could be fixed by
adding/removing whitespace or moving the comment outside of the expression.
It would probably be possible to produce fix-its for these. Here are some
examples of the changes.

Some cases which would previously work will now produce an error (these
are breaking changes):

1 /* */+21 +/* comment */21+/*comment*/ 2

Some cases which were previously errors will now work:

/* */!foo
!/* */foo
1+/* */21 /**/+ 2

Examples of things which will continue to be errors:

1/**/+ 21 +/*comment*/2

And things which will continue to work:

foo!// comment
foo/* */?.description
foo/* */!
1 +/**/ 21/**/+21+/**/2

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#alternatives-considered&gt;Alternatives
considered
<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#treat-comments-as-whitespace&gt;Treat
comments as whitespace

We could instead specify that comments are treated as whitespace. This is
a simpler rule, and it avoids some extra complications described above
around slash-slash comments and multi-line slash-star comments. It is also
easier for both the lexer and a human reader to determine whether an
operator is binary or not, since if comments are always whitespace, you
don't have to scan to the other side of a long comment to tell whether an
operator has whitespace around it. For example:

1 +/* a very long comment */2

With the proposed changes, you can't tell just by looking near the “+”
whether it is a binary or prefix operator.

On the other hand, this goes against the common mental model that comments
are ignored when parsing. As a result, this is somewhat more surprising
rule. This rule is also less flexible; that is, there are places which this
completely prohibits comments which are permitted by the above proposal.
(e.g. !/* */foo)

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#a-more-general-rule&gt;A
more general rule
Another alternative is a more general rule about how comments are handled
everywhere in the language (e.g. there must be no effect when replacing a
comment with a space character). This has the advantage of possibly
resolving other ambiguities, but has potentially far-reaching consequences
in various edge cases which are hard to fully enumerate (e.g. multi-line
comments, comments within interpolated sequences inside of string literals,
comments in lines which contain "#" directives, etc).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--

Owen Zhao

I think it's a good idea to clarify how operators and comments "work together".

Félix

···

Le 9 janv. 2016 à 12:25:22, 肇鑫 via swift-evolution <swift-evolution@swift.org> a écrit :

I think treat the comments as absence is a good idea. As the comments are only for the programmers not for the compiler.

Besides, I think one rule should be added. The comments that are used to generate the help documents or #pragma like should always be written in separated lines and should never mixed in the code lines.

zhaoxin

On Sun, Jan 10, 2016 at 12:50 AM, Jesse Rusak via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>>wrote:
Hi All,

Last month, I started a discussion about resolving some inconsistencies in the way operators interact with comments. I received some good feedback and then the break happened. I’ve since updated the draft proposal in a few small ways and would appreciate any additional comments folks have before submitting it for review.

The proposal is reproduced below and lives here: https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md

Thanks!
Jesse

Clarify interaction between comments & operators

Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md&gt;
Author(s): Jesse Rusak <https://github.com/jder&gt;
Status: Review
Review manager: TBD
<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#introduction&gt;Introduction

There are several inconsistencies in how comments are treated when determining whether an operator is prefix, postfix, or infix. They are sometimes treated as whitespace, sometimes as non-whitespace, and this differs depending on whether they are to the left or right of an operator, and the contents of the comment itself. This proposal suggests a uniform set of rules for how these cases should be parsed.

Swift-evolution thread: started here <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003780.html&gt; and continued here <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/003913.html&gt; and continued here <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151228/004646.html&gt;\.

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#motivation&gt;Motivation

At the moment, comments next to operators are usually treated as non-whitespace for the purpose of determining whether an operator is prefix/postfix/binary <The Swift Programming Language: Redirect, meaning that this fails to compile:

if /* comment */!foo { ... }
Because the "!" is parsed as binary operator (no whitespace on either side), rather than as a prefix operator, which seems undesirable. This behavior is also not consistently applied. For example, this currently works:

1 +/* comment */2
Because the "+/*" is treated as one token and sees the whitespace to its right and left, and so is parsed as a binary operator.

In order to resolve these and related issues, this proposes a general rule about the expected behavior.

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#proposed-solution&gt;Proposed solution

Comments should be treated as absent for all of the purposes in the “operators” section of the swift language reference: determining whether an operator is binary, prefix, or postfix, as well as the special rules around the “!” and “?” predefined operators. In other words, operators should "see through" a comment to the characters on the other side.

This means that adding a comment next to an operator (including between it and its operand) should not change whether the operator is treated as prefix, postfix, or binary, regardless of the contents of the comment.

For example, these should all be equivalent:

if !foo { ... }
if /* comment */!foo { ... }
if !/* comment */foo { ... }
As should these:

// whitespace on both sides
1 + 2
1 +/* comment */ 2

// no whitespace on either side
1+/*comment*/2
1+/* comment
comment */2
This is a predictable model, and is intended to be as unsurprising as possible, especially to beginners that are not used to troubleshooting parse errors.

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#detailed-design&gt;Detailed design

When parsing an operator character and trying to determine whether it has a whitespace character to either the right or left, we should skip comments (both possibly-nested slash-star comments, and slash-slash comments). The contents of the skipped comments (e.g. whether they include newlines) should have no effect on this determination.

For this purpose, slash-slash comments should be treated as extending up to but not including the trailing newline character (if present). So, for example, this should be treated as a postfix "~" operator:

let a = foo~// comment
bar()
The newline character after the end of the comment means the "~" has whitespace to its right, not bar. On the other hand, this should be parsed as a binary operator:

let a = foo~/* commment
*/bar()
As the newline appears only within the comment. (Note that this only matters for operators other than "!" and "?" because of their special rules.)

The language reference should also be updated to make clear that comments are ignored for these purposes.

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#impact-on-existing-code&gt;Impact on existing code

Only code with comments immediately next to operators will be affected. This is not expected to be very common, and could be fixed by adding/removing whitespace or moving the comment outside of the expression. It would probably be possible to produce fix-its for these. Here are some examples of the changes.

Some cases which would previously work will now produce an error (these are breaking changes):

1 /* */+2
1 +/* comment */2
1+/*comment*/ 2
Some cases which were previously errors will now work:

/* */!foo
!/* */foo

1+/* */2
1 /**/+ 2
Examples of things which will continue to be errors:

1/**/+ 2
1 +/*comment*/2
And things which will continue to work:

foo!// comment
foo/* */?.description
foo/* */!

1 +/**/ 2
1/**/+2
1+/**/2
<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#alternatives-considered&gt;Alternatives considered

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#treat-comments-as-whitespace&gt;Treat comments as whitespace

We could instead specify that comments are treated as whitespace. This is a simpler rule, and it avoids some extra complications described above around slash-slash comments and multi-line slash-star comments. It is also easier for both the lexer and a human reader to determine whether an operator is binary or not, since if comments are always whitespace, you don't have to scan to the other side of a long comment to tell whether an operator has whitespace around it. For example:

1 +/* a very long comment */2
With the proposed changes, you can't tell just by looking near the “+” whether it is a binary or prefix operator.

On the other hand, this goes against the common mental model that comments are ignored when parsing. As a result, this is somewhat more surprising rule. This rule is also less flexible; that is, there are places which this completely prohibits comments which are permitted by the above proposal. (e.g. !/* */foo)

<https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md#a-more-general-rule&gt;A more general rule

Another alternative is a more general rule about how comments are handled everywhere in the language (e.g. there must be no effect when replacing a comment with a space character). This has the advantage of possibly resolving other ambiguities, but has potentially far-reaching consequences in various edge cases which are hard to fully enumerate (e.g. multi-line comments, comments within interpolated sequences inside of string literals, comments in lines which contain "#" directives, etc).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

--

Owen Zhao
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

Thanks both of you for your positive feedback. One response below:

···

On Jan 9, 2016, at 2:26 PM, Félix Cloutier <felixcca@yahoo.ca> wrote:

I think it's a good idea to clarify how operators and comments "work together".

Félix

Le 9 janv. 2016 à 12:25:22, 肇鑫 via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> a écrit :

I think treat the comments as absence is a good idea. As the comments are only for the programmers not for the compiler.

Besides, I think one rule should be added. The comments that are used to generate the help documents or #pragma like should always be written in separated lines and should never mixed in the code lines.

I think that this is probably a good convention, but I’m not sure you’d want that as part of the language. Maybe something a style-checker would do?

- Jesse