Breaking change in lexing operators next to comments


(Jesse Rusak) #1

Hi all,

I’m investigating this bug: https://bugs.swift.org/browse/SR-186

Which appears to be a result of the fact that the logic that determines if an operator is prefix/postfix/binary does not treat comments as whitespace. So, for example:

/* comment */!foo

does not lex as expected because the “!" thinks it has an something on both sides and so is treated as a binary operator.

Fixing this (by treating comments as whitespace here) will break existing code which relies on the current behavior, such as:

foo/* comment */!

which currently treats the “!” as a postfix operator but will change to binary. I expect these cases would be rare (maybe in some generated code?), but the results might be pretty confusing.

Any objections to fixing this or other thoughts?

Thanks,
Jesse


(Chris Lattner) #2

There are two defensible models here:

1) comments should be treated as whitespace.
2) comments should be treated as if they were not present.

The later model seems more ideal to me (because you can put whitespace on either side of the comment after all), but I don’t have a strong opinion about that. What do others think?

-Chris

···

On Dec 14, 2015, at 8:15 PM, Jesse Rusak via swift-dev <swift-dev@swift.org> wrote:

Hi all,

I’m investigating this bug: https://bugs.swift.org/browse/SR-186

Which appears to be a result of the fact that the logic that determines if an operator is prefix/postfix/binary does not treat comments as whitespace. So, for example:

/* comment */!foo

does not lex as expected because the “!" thinks it has an something on both sides and so is treated as a binary operator.

Fixing this (by treating comments as whitespace here) will break existing code which relies on the current behavior, such as:

foo/* comment */!

which currently treats the “!” as a postfix operator but will change to binary. I expect these cases would be rare (maybe in some generated code?), but the results might be pretty confusing.

Any objections to fixing this or other thoughts?


(Simon Pilkington) #3

It seems to make more sense to treat comments as this if they are not present.

As a related question, should the presence/absence of whitespace be important at all? It seems fragile if it is.

-Simon

···

On 14 Dec 2015, at 9:42 PM, Chris Lattner via swift-dev <swift-dev@swift.org> wrote:

On Dec 14, 2015, at 8:15 PM, Jesse Rusak via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hi all,

I’m investigating this bug: https://bugs.swift.org/browse/SR-186

Which appears to be a result of the fact that the logic that determines if an operator is prefix/postfix/binary does not treat comments as whitespace. So, for example:

/* comment */!foo

does not lex as expected because the “!" thinks it has an something on both sides and so is treated as a binary operator.

Fixing this (by treating comments as whitespace here) will break existing code which relies on the current behavior, such as:

foo/* comment */!

which currently treats the “!” as a postfix operator but will change to binary. I expect these cases would be rare (maybe in some generated code?), but the results might be pretty confusing.

Any objections to fixing this or other thoughts?

There are two defensible models here:

1) comments should be treated as whitespace.
2) comments should be treated as if they were not present.

The later model seems more ideal to me (because you can put whitespace on either side of the comment after all), but I don’t have a strong opinion about that. What do others think?

-Chris

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev


(John Calsbeek) #4

If you treat comments as though they are not present, you can no longer reason locally about whitespace on either side of an operator. Straw example:

foo/* insert an
excerpt from War
and Peace here */!

I need to scan to the other side of the comment to determine if ! is preceded by whitespace.

There is already a list of situations in which some token is treated as whitespace for the purpose of operators in The Swift Programming Language:

For the purposes of these rules, the characters (, [, and { before an operator, the characters ), ], and } after an operator, and the characters , ;, and : are also considered whitespace.

There is one caveat to the rules above. If the ! or ? predefined operator has no whitespace on the left, it is treated as a postfix operator, regardless of whether it has whitespace on the right. To use the ? as the optional-chaining operator, it must not have whitespace on the left. To use it in the ternary conditional (? :slight_smile: operator, it must have whitespace around both sides.

Given that, it seems more natural to me to define comments as “treated-as-whitespace” in the same way.

“Treated as not present” is also not quite the right way to word the opposite case, since comments would still separate tokens. Say you had an automated tool that deletes comments (perhaps unlikely, but let’s roll with it). “Treated as not present” says you should completely delete the comment, but that doesn’t actually work since it could still cause two separate tokens to be glued together. “Treated as whitespace” just means that you have to replace the comment with at least one character of whitespace.

-John

···

On Dec 14, 2015, at 9:42 PM, Chris Lattner via swift-dev <swift-dev@swift.org> wrote:

On Dec 14, 2015, at 8:15 PM, Jesse Rusak via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hi all,

I’m investigating this bug: https://bugs.swift.org/browse/SR-186

Which appears to be a result of the fact that the logic that determines if an operator is prefix/postfix/binary does not treat comments as whitespace. So, for example:

/* comment */!foo

does not lex as expected because the “!" thinks it has an something on both sides and so is treated as a binary operator.

Fixing this (by treating comments as whitespace here) will break existing code which relies on the current behavior, such as:

foo/* comment */!

which currently treats the “!” as a postfix operator but will change to binary. I expect these cases would be rare (maybe in some generated code?), but the results might be pretty confusing.

Any objections to fixing this or other thoughts?

There are two defensible models here:

1) comments should be treated as whitespace.
2) comments should be treated as if they were not present.

The later model seems more ideal to me (because you can put whitespace on either side of the comment after all), but I don’t have a strong opinion about that. What do others think?

-Chris

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev


(Chris Lattner) #5

The current design depends on whitespace to decide whether an operator is infix, postfix, or prefix. It can be surprising to some folks when they first encounter it, but IMO it encourages more maintainable code to be written.

The one thing that I’d love to see fixed is to enhance the error produced for things like:

  let x = y*-z

to produce a fixit, which rewrites it as:

  let x = y * -z

-Chris

···

On Dec 14, 2015, at 9:51 PM, Simon Pilkington <simonmpilkington@icloud.com> wrote:

It seems to make more sense to treat comments as this if they are not present.

As a related question, should the presence/absence of whitespace be important at all? It seems fragile if it is.


(Jesse Rusak) #6

FWIW, I agree with John about this. I think either model is reasonable but treating comments as whitespace is better because:

* The Swift language reference already states the general rule that comments are whitespace; I think it’s better to apply this throughout than change it.

* I think it’s easier to explain that "comments are whitespace" than "comments are treated as not present except they separate tokens”.

* The non-local effects John describes are mildly awkward for human readers and in the lexer. (I think we’d have to walk backwards through slash-star comments to determine if we have space to the left of an operator.)

- Jesse

···

On Dec 14, 2015, at 9:42 PM, Chris Lattner via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

There are two defensible models here:

1) comments should be treated as whitespace.
2) comments should be treated as if they were not present.

The later model seems more ideal to me (because you can put whitespace on either side of the comment after all), but I don’t have a strong opinion about that. What do others think?

On Dec 15, 2015, at 1:08 AM, John Calsbeek via swift-dev <swift-dev@swift.org> wrote:

If you treat comments as though they are not present, you can no longer reason locally about whitespace on either side of an operator. Straw example:

foo/* insert an
excerpt from War
and Peace here */!

I need to scan to the other side of the comment to determine if ! is preceded by whitespace.

There is already a list of situations in which some token is treated as whitespace for the purpose of operators in The Swift Programming Language:

For the purposes of these rules, the characters (, [, and { before an operator, the characters ), ], and } after an operator, and the characters , ;, and : are also considered whitespace.

There is one caveat to the rules above. If the ! or ? predefined operator has no whitespace on the left, it is treated as a postfix operator, regardless of whether it has whitespace on the right. To use the ? as the optional-chaining operator, it must not have whitespace on the left. To use it in the ternary conditional (? :slight_smile: operator, it must have whitespace around both sides.

Given that, it seems more natural to me to define comments as “treated-as-whitespace” in the same way.

“Treated as not present” is also not quite the right way to word the opposite case, since comments would still separate tokens. Say you had an automated tool that deletes comments (perhaps unlikely, but let’s roll with it). “Treated as not present” says you should completely delete the comment, but that doesn’t actually work since it could still cause two separate tokens to be glued together. “Treated as whitespace” just means that you have to replace the comment with at least one character of whitespace.


(Jesse Rusak) #7

Any other comments about this? Can someone from the core team make a call on this, or should we ask for comments on swift-evolution?

···

On Dec 15, 2015, at 7:15 PM, Jesse Rusak <me@jesserusak.com> wrote:

On Dec 14, 2015, at 9:42 PM, Chris Lattner via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

There are two defensible models here:

1) comments should be treated as whitespace.
2) comments should be treated as if they were not present.

The later model seems more ideal to me (because you can put whitespace on either side of the comment after all), but I don’t have a strong opinion about that. What do others think?

On Dec 15, 2015, at 1:08 AM, John Calsbeek via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

If you treat comments as though they are not present, you can no longer reason locally about whitespace on either side of an operator. Straw example:

foo/* insert an
excerpt from War
and Peace here */!

I need to scan to the other side of the comment to determine if ! is preceded by whitespace.

There is already a list of situations in which some token is treated as whitespace for the purpose of operators in The Swift Programming Language:

For the purposes of these rules, the characters (, [, and { before an operator, the characters ), ], and } after an operator, and the characters , ;, and : are also considered whitespace.

There is one caveat to the rules above. If the ! or ? predefined operator has no whitespace on the left, it is treated as a postfix operator, regardless of whether it has whitespace on the right. To use the ? as the optional-chaining operator, it must not have whitespace on the left. To use it in the ternary conditional (? :slight_smile: operator, it must have whitespace around both sides.

Given that, it seems more natural to me to define comments as “treated-as-whitespace” in the same way.

“Treated as not present” is also not quite the right way to word the opposite case, since comments would still separate tokens. Say you had an automated tool that deletes comments (perhaps unlikely, but let’s roll with it). “Treated as not present” says you should completely delete the comment, but that doesn’t actually work since it could still cause two separate tokens to be glued together. “Treated as whitespace” just means that you have to replace the comment with at least one character of whitespace.

FWIW, I agree with John about this. I think either model is reasonable but treating comments as whitespace is better because:

* The Swift language reference already states the general rule that comments are whitespace; I think it’s better to apply this throughout than change it.

* I think it’s easier to explain that "comments are whitespace" than "comments are treated as not present except they separate tokens”.

* The non-local effects John describes are mildly awkward for human readers and in the lexer. (I think we’d have to walk backwards through slash-star comments to determine if we have space to the left of an operator.)

- Jesse


(Chris Lattner) #8

Any other comments about this? Can someone from the core team make a call on this, or should we ask for comments on swift-evolution?

I think it makes sense to bring it up on swift-evolution. I tend to agree with your argument that treating them as whitespace is better, but I don’t feel strongly about it.

-Chris

···

On Dec 17, 2015, at 6:32 AM, Jesse Rusak via swift-dev <swift-dev@swift.org> wrote:

On Dec 15, 2015, at 7:15 PM, Jesse Rusak <me@jesserusak.com <mailto:me@jesserusak.com>> wrote:

On Dec 14, 2015, at 9:42 PM, Chris Lattner via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

There are two defensible models here:

1) comments should be treated as whitespace.
2) comments should be treated as if they were not present.

The later model seems more ideal to me (because you can put whitespace on either side of the comment after all), but I don’t have a strong opinion about that. What do others think?

On Dec 15, 2015, at 1:08 AM, John Calsbeek via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

If you treat comments as though they are not present, you can no longer reason locally about whitespace on either side of an operator. Straw example:

foo/* insert an
excerpt from War
and Peace here */!

I need to scan to the other side of the comment to determine if ! is preceded by whitespace.

There is already a list of situations in which some token is treated as whitespace for the purpose of operators in The Swift Programming Language:

For the purposes of these rules, the characters (, [, and { before an operator, the characters ), ], and } after an operator, and the characters , ;, and : are also considered whitespace.

There is one caveat to the rules above. If the ! or ? predefined operator has no whitespace on the left, it is treated as a postfix operator, regardless of whether it has whitespace on the right. To use the ? as the optional-chaining operator, it must not have whitespace on the left. To use it in the ternary conditional (? :) operator, it must have whitespace around both sides.

Given that, it seems more natural to me to define comments as “treated-as-whitespace” in the same way.

“Treated as not present” is also not quite the right way to word the opposite case, since comments would still separate tokens. Say you had an automated tool that deletes comments (perhaps unlikely, but let’s roll with it). “Treated as not present” says you should completely delete the comment, but that doesn’t actually work since it could still cause two separate tokens to be glued together. “Treated as whitespace” just means that you have to replace the comment with at least one character of whitespace.

FWIW, I agree with John about this. I think either model is reasonable but treating comments as whitespace is better because:

* The Swift language reference already states the general rule that comments are whitespace; I think it’s better to apply this throughout than change it.

* I think it’s easier to explain that "comments are whitespace" than "comments are treated as not present except they separate tokens”.

* The non-local effects John describes are mildly awkward for human readers and in the lexer. (I think we’d have to walk backwards through slash-star comments to determine if we have space to the left of an operator.)

- Jesse

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(John McCall) #9

Any other comments about this? Can someone from the core team make a call on this, or should we ask for comments on swift-evolution?

It has to go through swift-evolution anyway, but yes, I completely agree that comments should be treated as whitespace for the purposes of the spacing rules.

John.

···

On Dec 17, 2015, at 6:32 AM, Jesse Rusak via swift-dev <swift-dev@swift.org> wrote:

On Dec 15, 2015, at 7:15 PM, Jesse Rusak <me@jesserusak.com <mailto:me@jesserusak.com>> wrote:

On Dec 14, 2015, at 9:42 PM, Chris Lattner via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

There are two defensible models here:

1) comments should be treated as whitespace.
2) comments should be treated as if they were not present.

The later model seems more ideal to me (because you can put whitespace on either side of the comment after all), but I don’t have a strong opinion about that. What do others think?

On Dec 15, 2015, at 1:08 AM, John Calsbeek via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

If you treat comments as though they are not present, you can no longer reason locally about whitespace on either side of an operator. Straw example:

foo/* insert an
excerpt from War
and Peace here */!

I need to scan to the other side of the comment to determine if ! is preceded by whitespace.

There is already a list of situations in which some token is treated as whitespace for the purpose of operators in The Swift Programming Language:

For the purposes of these rules, the characters (, [, and { before an operator, the characters ), ], and } after an operator, and the characters , ;, and : are also considered whitespace.

There is one caveat to the rules above. If the ! or ? predefined operator has no whitespace on the left, it is treated as a postfix operator, regardless of whether it has whitespace on the right. To use the ? as the optional-chaining operator, it must not have whitespace on the left. To use it in the ternary conditional (? :slight_smile: operator, it must have whitespace around both sides.

Given that, it seems more natural to me to define comments as “treated-as-whitespace” in the same way.

“Treated as not present” is also not quite the right way to word the opposite case, since comments would still separate tokens. Say you had an automated tool that deletes comments (perhaps unlikely, but let’s roll with it). “Treated as not present” says you should completely delete the comment, but that doesn’t actually work since it could still cause two separate tokens to be glued together. “Treated as whitespace” just means that you have to replace the comment with at least one character of whitespace.

FWIW, I agree with John about this. I think either model is reasonable but treating comments as whitespace is better because:

* The Swift language reference already states the general rule that comments are whitespace; I think it’s better to apply this throughout than change it.

* I think it’s easier to explain that "comments are whitespace" than "comments are treated as not present except they separate tokens”.

* The non-local effects John describes are mildly awkward for human readers and in the lexer. (I think we’d have to walk backwards through slash-star comments to determine if we have space to the left of an operator.)

- Jesse

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Jesse Rusak) #10

Any other comments about this? Can someone from the core team make a call on this, or should we ask for comments on swift-evolution?

I think it makes sense to bring it up on swift-evolution. I tend to agree with your argument that treating them as whitespace is better, but I don’t feel strongly about it.

I’ve been trying to put together a proposal but have run into several cases in which the current behavior seems inconsistent, making it hard to produce a description of the proposed deltas. In current Swift (and 2.1.1), these parse:

1+/**/2
1+/*hi*/2

but this doesn’t:

1+/* */2 // error: consecutive statements on a line must be separated by ';'

And this parses (say bar is an optional):

bar/* */!

but this doesn’t (say bar is a Bool):

!/* */bar // error: unary operator cannot be separated from its operand

And this parses:

1 +/**/ 2

but this doesn’t:

1 /**/+ 2 // error: consecutive statements on a line must be separated by ';'

Would it make sense to put forward a proposal for the overall behavior of comments, with the intention of aligning these all in a particular direction? Or is there an understanding of the expected behavior today such that some of the above would be bugs to be filed & fixed?

Thanks,
Jesse

···

On Dec 17, 2015, at 1:08 PM, Chris Lattner <clattner@apple.com> wrote:

On Dec 17, 2015, at 6:32 AM, Jesse Rusak via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

On Dec 15, 2015, at 7:15 PM, Jesse Rusak <me@jesserusak.com <mailto:me@jesserusak.com>> wrote:

On Dec 14, 2015, at 9:42 PM, Chris Lattner via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

There are two defensible models here:

1) comments should be treated as whitespace.
2) comments should be treated as if they were not present.

The later model seems more ideal to me (because you can put whitespace on either side of the comment after all), but I don’t have a strong opinion about that. What do others think?

On Dec 15, 2015, at 1:08 AM, John Calsbeek via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

If you treat comments as though they are not present, you can no longer reason locally about whitespace on either side of an operator. Straw example:

foo/* insert an
excerpt from War
and Peace here */!

I need to scan to the other side of the comment to determine if ! is preceded by whitespace.

There is already a list of situations in which some token is treated as whitespace for the purpose of operators in The Swift Programming Language:

For the purposes of these rules, the characters (, [, and { before an operator, the characters ), ], and } after an operator, and the characters , ;, and : are also considered whitespace.

There is one caveat to the rules above. If the ! or ? predefined operator has no whitespace on the left, it is treated as a postfix operator, regardless of whether it has whitespace on the right. To use the ? as the optional-chaining operator, it must not have whitespace on the left. To use it in the ternary conditional (? :slight_smile: operator, it must have whitespace around both sides.

Given that, it seems more natural to me to define comments as “treated-as-whitespace” in the same way.

“Treated as not present” is also not quite the right way to word the opposite case, since comments would still separate tokens. Say you had an automated tool that deletes comments (perhaps unlikely, but let’s roll with it). “Treated as not present” says you should completely delete the comment, but that doesn’t actually work since it could still cause two separate tokens to be glued together. “Treated as whitespace” just means that you have to replace the comment with at least one character of whitespace.

FWIW, I agree with John about this. I think either model is reasonable but treating comments as whitespace is better because:

* The Swift language reference already states the general rule that comments are whitespace; I think it’s better to apply this throughout than change it.

* I think it’s easier to explain that "comments are whitespace" than "comments are treated as not present except they separate tokens”.

* The non-local effects John describes are mildly awkward for human readers and in the lexer. (I think we’d have to walk backwards through slash-star comments to determine if we have space to the left of an operator.)

- Jesse

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev


(Jesse Rusak) #11

For future reference, I’ve started a thread about this on swift-evolution:

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003780.html

···

On Dec 17, 2015, at 10:41 PM, Jesse Rusak <me@jesserusak.com> wrote:

On Dec 17, 2015, at 1:08 PM, Chris Lattner <clattner@apple.com <mailto:clattner@apple.com>> wrote:

On Dec 17, 2015, at 6:32 AM, Jesse Rusak via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Any other comments about this? Can someone from the core team make a call on this, or should we ask for comments on swift-evolution?

I think it makes sense to bring it up on swift-evolution. I tend to agree with your argument that treating them as whitespace is better, but I don’t feel strongly about it.