[Pitch] Clarify behavior of comments near operators

Hi all,

Following a discussion on swift-dev <https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20151214/000348.html&gt; about bug SR-186 <Issues · apple/swift-issues · GitHub, I would like to propose a rule about how Swift should handle comments next to operators in order to make it clear how to resolve some existing bugs & inconsistencies.

Please take a look at the pseudo-proposal below. I believe it is in line with the current intention of the language reference, but I’m interested in hearing differing opinions. Also, most of the cases in which this comes up are pretty contrived, so I’d love to hear if anyone has any common/important cases where this makes a difference.

Thanks,
Jesse

Background

At the moment, comments next to operators are generally treated as non-whitespace for the purpose of determining whether an operator is prefix/postfix/binary <Lexical Structure — The Swift Programming Language (Swift 5.7), meaning that this fails to compile:

if /* comment */!foo { … }

Because the “!” is parsed as binary operator (no whitespace on either side), rather than as a prefix operator, which seems undesirable. This behavior is also not consistently applied. For example, this currently works:

1 +/* comment */2

(I believe this is unintentional; the "+/*" is treated as one token and sees the whitespace to its right.)

In order to resolve these issues, we need a general rule about how this is expected to behave.

Proposed changes

Comments should be treated as whitespace for all of the purposes in the “operators” section of the swift language reference: determining whether an operator is binary, prefix, or postfix, as well as the special rules around the “!” and “?” predefined operators.

Impact on existing code

Only code with comments immediately next to operators will be affected. This is not expected to be very common, and could be fixed by adding a space next to the operator in question or moving the comment outside of the expression. It would probably be possible to produce fix-its for these, though I’m not sure it’s worth it. Here are some examples of the changes.

Some cases which were previously errors will now work:

/* */!foo
1/**/+ 2
1 /**/+ 2
1 +/*hi*/2

Some cases which would previously work will now error (these are breaking changes):

foo/* */?.description
foo/* */!
1/**/+2
1+/**/2

Examples of things which will continue to be errors:

!/* */foo
1+/* */2

And things which will continue to work:

foo!// this is dangerous
1 +/**/ 2
1 +/* hi */2

Alternatives considered

We could instead specify that comments are treated as though they are not present. This more-closely matches some people’s mental model of comments. However, it is harder to describe (the characters are not removed entirely as they still separate tokens) and goes against the current general rule in the language reference that comments are whitespace.

This also has the disadvantage that you have to look at the other side of a comment to determine if an operator has whitespace around it. For example:

a = 1 +/* a very long comment */2

You can’t tell just by looking near the “+” whether it is a binary or prefix operator.

Another alternative is a more precise rule about how comments are handled everywhere in the language (e.g. there must be no effect when replacing a comment with a space character). This has the advantage of possibly resolving other ambiguities, but has potentially far-reaching consequences in various edge cases (for example, multi-line comments which span lines are currently treated as end-of-line).

After looking at all your examples, I think I prefer "comments are not there" with a possible "multi-line comments are treated like newlines" extension. But either way I think having an actual model here and sticking to it is an improvement!

Jordan

···

On Dec 20, 2015, at 11:55 , Jesse Rusak via swift-evolution <swift-evolution@swift.org> wrote:

Hi all,

Following a discussion on swift-dev <https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20151214/000348.html&gt; about bug SR-186 <Issues · apple/swift-issues · GitHub, I would like to propose a rule about how Swift should handle comments next to operators in order to make it clear how to resolve some existing bugs & inconsistencies.

Please take a look at the pseudo-proposal below. I believe it is in line with the current intention of the language reference, but I’m interested in hearing differing opinions. Also, most of the cases in which this comes up are pretty contrived, so I’d love to hear if anyone has any common/important cases where this makes a difference.

Thanks,
Jesse

Background

At the moment, comments next to operators are generally treated as non-whitespace for the purpose of determining whether an operator is prefix/postfix/binary <The Swift Programming Language: Redirect, meaning that this fails to compile:

if /* comment */!foo { … }

Because the “!” is parsed as binary operator (no whitespace on either side), rather than as a prefix operator, which seems undesirable. This behavior is also not consistently applied. For example, this currently works:

1 +/* comment */2

(I believe this is unintentional; the "+/*" is treated as one token and sees the whitespace to its right.)

In order to resolve these issues, we need a general rule about how this is expected to behave.

Proposed changes

Comments should be treated as whitespace for all of the purposes in the “operators” section of the swift language reference: determining whether an operator is binary, prefix, or postfix, as well as the special rules around the “!” and “?” predefined operators.

Impact on existing code

Only code with comments immediately next to operators will be affected. This is not expected to be very common, and could be fixed by adding a space next to the operator in question or moving the comment outside of the expression. It would probably be possible to produce fix-its for these, though I’m not sure it’s worth it. Here are some examples of the changes.

Some cases which were previously errors will now work:

/* */!foo
1/**/+ 2
1 /**/+ 2
1 +/*hi*/2

Some cases which would previously work will now error (these are breaking changes):

foo/* */?.description
foo/* */!
1/**/+2
1+/**/2

Examples of things which will continue to be errors:

!/* */foo
1+/* */2

And things which will continue to work:

foo!// this is dangerous
1 +/**/ 2
1 +/* hi */2

Alternatives considered

We could instead specify that comments are treated as though they are not present. This more-closely matches some people’s mental model of comments. However, it is harder to describe (the characters are not removed entirely as they still separate tokens) and goes against the current general rule in the language reference that comments are whitespace.

This also has the disadvantage that you have to look at the other side of a comment to determine if an operator has whitespace around it. For example:

a = 1 +/* a very long comment */2

You can’t tell just by looking near the “+” whether it is a binary or prefix operator.

Another alternative is a more precise rule about how comments are handled everywhere in the language (e.g. there must be no effect when replacing a comment with a space character). This has the advantage of possibly resolving other ambiguities, but has potentially far-reaching consequences in various edge cases (for example, multi-line comments which span lines are currently treated as end-of-line).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

After looking at all your examples, I think I prefer "comments are not there" with a possible "multi-line comments are treated like newlines" extension. But either way I think having an actual model here and sticking to it is an improvement!

Thanks for your thoughts! To be clear, are you suggesting “comments are not there” for the purposes of whitespace near operators, or as part of a more-general rule to be applied everywhere? (i.e. do you mean the first or second of the alternatives I listed?)

- Jesse

···

On Dec 21, 2015, at 3:18 PM, Jordan Rose <jordan_rose@apple.com> wrote:

On Dec 20, 2015, at 11:55 , Jesse Rusak via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi all,

Following a discussion on swift-dev <https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20151214/000348.html&gt; about bug SR-186 <Issues · apple/swift-issues · GitHub, I would like to propose a rule about how Swift should handle comments next to operators in order to make it clear how to resolve some existing bugs & inconsistencies.

Please take a look at the pseudo-proposal below. I believe it is in line with the current intention of the language reference, but I’m interested in hearing differing opinions. Also, most of the cases in which this comes up are pretty contrived, so I’d love to hear if anyone has any common/important cases where this makes a difference.

Thanks,
Jesse

Background

At the moment, comments next to operators are generally treated as non-whitespace for the purpose of determining whether an operator is prefix/postfix/binary <The Swift Programming Language: Redirect, meaning that this fails to compile:

if /* comment */!foo { … }

Because the “!” is parsed as binary operator (no whitespace on either side), rather than as a prefix operator, which seems undesirable. This behavior is also not consistently applied. For example, this currently works:

1 +/* comment */2

(I believe this is unintentional; the "+/*" is treated as one token and sees the whitespace to its right.)

In order to resolve these issues, we need a general rule about how this is expected to behave.

Proposed changes

Comments should be treated as whitespace for all of the purposes in the “operators” section of the swift language reference: determining whether an operator is binary, prefix, or postfix, as well as the special rules around the “!” and “?” predefined operators.

Impact on existing code

Only code with comments immediately next to operators will be affected. This is not expected to be very common, and could be fixed by adding a space next to the operator in question or moving the comment outside of the expression. It would probably be possible to produce fix-its for these, though I’m not sure it’s worth it. Here are some examples of the changes.

Some cases which were previously errors will now work:

/* */!foo
1/**/+ 2
1 /**/+ 2
1 +/*hi*/2

Some cases which would previously work will now error (these are breaking changes):

foo/* */?.description
foo/* */!
1/**/+2
1+/**/2

Examples of things which will continue to be errors:

!/* */foo
1+/* */2

And things which will continue to work:

foo!// this is dangerous
1 +/**/ 2
1 +/* hi */2

Alternatives considered

We could instead specify that comments are treated as though they are not present. This more-closely matches some people’s mental model of comments. However, it is harder to describe (the characters are not removed entirely as they still separate tokens) and goes against the current general rule in the language reference that comments are whitespace.

This also has the disadvantage that you have to look at the other side of a comment to determine if an operator has whitespace around it. For example:

a = 1 +/* a very long comment */2

You can’t tell just by looking near the “+” whether it is a binary or prefix operator.

Another alternative is a more precise rule about how comments are handled everywhere in the language (e.g. there must be no effect when replacing a comment with a space character). This has the advantage of possibly resolving other ambiguities, but has potentially far-reaching consequences in various edge cases (for example, multi-line comments which span lines are currently treated as end-of-line).

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

I'm not sure how your two alternatives map to these options, but I think the model I like has

foo/*abc*/!

as a postfix '!'. I'm not sure if I want to count

foo/*abc
*/+ bar

as an infix '+', as an error, or as a de-facto line continuation mechanism (making a postfix '+'). I do agree it's important to pick something.

Jordan

···

On Dec 21, 2015, at 19:23, Jesse Rusak <me@jesserusak.com> wrote:

On Dec 21, 2015, at 3:18 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:

After looking at all your examples, I think I prefer "comments are not there" with a possible "multi-line comments are treated like newlines" extension. But either way I think having an actual model here and sticking to it is an improvement!

Thanks for your thoughts! To be clear, are you suggesting “comments are not there” for the purposes of whitespace near operators, or as part of a more-general rule to be applied everywhere? (i.e. do you mean the first or second of the alternatives I listed?)

I’ve spent some more time thinking about this. I think that “comments are whitespace” is a simpler model, but I do think it’s more surprising for many people, and (as Chris pointed out earlier) is less flexible, since you can always add whitespace next to a comment if you want it. So, I think I’m coming around to the “comments are not there” behavior.

For your latter example, my feeling is that having the *content* of a comment affect the parsing of the surrounding code (i.e. treating your last example as an error or infix +) is surprising. For example, it would surprise me that this works:

a+/* something */b

but this doesn’t:

a+/* something
       something */b

So, I think I’ll write up a draft proposal with the “comments are not there” behavior and see how it looks.

- Jesse

···

On Dec 22, 2015, at 1:39 PM, Jordan Rose <jordan_rose@apple.com> wrote:

On Dec 21, 2015, at 19:23, Jesse Rusak <me@jesserusak.com <mailto:me@jesserusak.com>> wrote:

On Dec 21, 2015, at 3:18 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:

After looking at all your examples, I think I prefer "comments are not there" with a possible "multi-line comments are treated like newlines" extension. But either way I think having an actual model here and sticking to it is an improvement!

Thanks for your thoughts! To be clear, are you suggesting “comments are not there” for the purposes of whitespace near operators, or as part of a more-general rule to be applied everywhere? (i.e. do you mean the first or second of the alternatives I listed?)

I'm not sure how your two alternatives map to these options, but I think the model I like has

foo/*abc*/!

as a postfix '!'. I'm not sure if I want to count

foo/*abc
*/+ bar

as an infix '+', as an error, or as a de-facto line continuation mechanism (making a postfix '+'). I do agree it's important to pick something.

And here’s the draft proposal: https://github.com/jder/swift-evolution/blob/comments-and-operators/proposals/0000-clarify-comments-and-operators.md

- Jesse

···

On Dec 28, 2015, at 11:49 AM, Jesse Rusak <me@jesserusak.com> wrote:

On Dec 22, 2015, at 1:39 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:

On Dec 21, 2015, at 19:23, Jesse Rusak <me@jesserusak.com <mailto:me@jesserusak.com>> wrote:

On Dec 21, 2015, at 3:18 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:

After looking at all your examples, I think I prefer "comments are not there" with a possible "multi-line comments are treated like newlines" extension. But either way I think having an actual model here and sticking to it is an improvement!

Thanks for your thoughts! To be clear, are you suggesting “comments are not there” for the purposes of whitespace near operators, or as part of a more-general rule to be applied everywhere? (i.e. do you mean the first or second of the alternatives I listed?)

I'm not sure how your two alternatives map to these options, but I think the model I like has

foo/*abc*/!

as a postfix '!'. I'm not sure if I want to count

foo/*abc
*/+ bar

as an infix '+', as an error, or as a de-facto line continuation mechanism (making a postfix '+'). I do agree it's important to pick something.

I’ve spent some more time thinking about this. I think that “comments are whitespace” is a simpler model, but I do think it’s more surprising for many people, and (as Chris pointed out earlier) is less flexible, since you can always add whitespace next to a comment if you want it. So, I think I’m coming around to the “comments are not there” behavior.

For your latter example, my feeling is that having the *content* of a comment affect the parsing of the surrounding code (i.e. treating your last example as an error or infix +) is surprising. For example, it would surprise me that this works:

a+/* something */b

but this doesn’t:

a+/* something
       something */b

So, I think I’ll write up a draft proposal with the “comments are not there” behavior and see how it looks.