Merge pull request swiftlang#207 from jder/comments-are-whitespace

Reverse proposal from comments-are-absent to comments-are-whitespace

proposals/0037-clarify-comments-and-operators.md

@@ -2,7 +2,7 @@
 
 * Proposal: [SE-0037](https://github.com/apple/swift-evolution/blob/master/proposals/0037-clarify-comments-and-operators.md)
 * Author(s): [Jesse Rusak](https://github.com/jder)
-* Status: **Scheduled** for March 9...13, 2016
+* Status: **Scheduled** for March 11...15, 2016
 * Review manager: [Chris Lattner](http://github.com/lattner)
 
 ## Introduction
@@ -14,181 +14,148 @@ whether they are to the left or right of an operator, and the contents of
 the comment itself. This proposal suggests a uniform set of rules for how these
 cases should be parsed.
 
-Swift-evolution thread: [started here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003780.html)
-and [continued here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/003913.html)
-and [continued here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151228/004646.html).
-And finally [here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/006030.html).
+Swift-evolution thread: [started here](http://thread.gmane.org/gmane.comp.lang.swift.evolution/605)
+and [continued here](http://thread.gmane.org/gmane.comp.lang.swift.evolution/2855).
 
-A draft implementation is [available here](https://github.com/apple/swift/compare/master...jder:comment-operator-absent).
+A draft implementation is [available here](https://github.com/apple/swift/compare/master...jder:comment-operator-fixes).
 
 ## Motivation
 
 At the moment, comments next to operators are usually treated as
 non-whitespace for the purpose of [determining whether an operator is prefix/postfix/binary](https://developer.apple.com/library/mac/documentation/Swift/Conceptual/Swift_Programming_Language/LexicalStructure.html#//apple_ref/doc/uid/TP40014097-CH30-ID418),
 meaning that this fails to compile ([SR-186](https://bugs.swift.org/browse/SR-186)):
 
-```swift
+```
 if /* comment */!foo { ... }
 ```
 
 Because the "!" is parsed as binary operator (no whitespace on either side),
 rather than as a prefix operator, which seems undesirable. This behavior is also
 not consistently applied. For example, this currently works:
 
-```swift
+```
 1 +/* comment */2
 ```
 
 Because the "`+/*`" is treated as one token and sees the whitespace to its
-right and left, and so is parsed as a binary operator. 
+right and left, and so is parsed as a binary operator.
 
 In order to resolve these and related issues, this proposes a general rule about
 the expected behavior.
 
 ## Proposed solution
 
-Comments should be treated as absent for all of the purposes in the “operators”
-section of the swift language reference: determining whether an operator is
+Comments should be treated as whitespace for all of the purposes in the “operators”
+section of the Swift language reference: determining whether an operator is
 binary, prefix, or postfix, as well as the special rules around the “!” and “?”
-predefined operators. In other words, operators should "see through" a comment
-to the characters on the other side.
+predefined operators.
 
-This means that adding a comment next to an operator (including between
-it and its operand) should not change whether the operator is treated as prefix,
-postfix, or binary, regardless of the contents of the comment.
+This means that swapping a comment with whitespace should
+not change whether an adjacent operator token is treated as prefix, postfix,
+or binary (regardless of the contents of the comment).
+This also includes whitespace/comments between the operator and its operand.
 
-For example, these should all be equivalent:
+For example, these should be equivalent:
 
-```swift
+```
+if/* comment */!foo { ... }
 if !foo { ... }
-if /* comment */!foo { ... }
-if !/* comment */foo { ... }
 ```
 
 As should these:
 
-```swift
+```
 // whitespace on both sides
 1 + 2
-1 +/* comment */ 2
-
-// no whitespace on either side
-1+/*comment*/2
-1+/* comment
- comment */2
+1 +/* comment */2
 ```
 
-This is a predictable model, and is intended to be as unsurprising as
-possible, especially to beginners that are not used to troubleshooting parse
-errors.
+This model is easy to describe, and fits in with the general rule already
+present in the Swift language reference that comments are treated as whitespace.
 
 ## Detailed design
 
-When parsing an operator character and trying to determine whether it has a 
-whitespace character to either the right or left, we should skip
-comments (both possibly-nested slash-star comments, and slash-slash comments). 
-The contents of the skipped comments (e.g. whether they include
-newlines) should have no effect on this determination.
-
-For this purpose, slash-slash comments should be treated as extending up to but
-not including the trailing newline character (if present). So, for example,
-this should be treated as a postfix "~" operator:
-
-```swift
-let a = foo~// comment
-bar()
-```
-
-The newline character after the end of the comment means the "~" has whitespace
-to its right, not `bar`. On the other hand, this should be parsed as a
-binary operator:
-
-```swift
-let a = foo~/* commment
-*/bar()
-```
-
-As the newline appears only within the comment. (Note that this only matters
-for operators other than "!" and "?" because of their special rules.)
+When parsing an operator token and trying to determine whether it has a
+whitespace character to either the right or left, an adjacent slash-slash
+or slash-star comment should be treated as whitespace. The contents of these
+comments (e.g. whether they themselves include whitespace) should have no effect.
 
 The language reference should also be updated to make clear that comments are
-ignored for these purposes.
+treated as whitespace for these purposes.
 
 ## Impact on existing code
 
 Only code with comments immediately next to operators will be affected. This is
-not expected to be very common, and could be fixed by adding/removing whitespace
+not expected to be very common, and could be fixed by adding whitespace
 or moving the comment outside of the expression. It would probably be possible
 to produce fix-its for these. Here are some examples of the changes.
 
-Some cases which would previously work will now produce an error 
+Some cases which would previously work will now produce an error
 (these are breaking changes):
 
-```swift
-1 /* */+2
-1 +/* comment */2
-1+/*comment*/ 2
+```
+foo/* */?.description
+foo/* */!
+1/**/+2
+1+/**/2
 ```
 
 Some cases which were previously errors will now work:
 
-```swift
+```
 /* */!foo
-!/* */foo
-
-1+/* */2
+1/**/+ 2
 1 /**/+ 2
+1 +/**/2
 ```
 
 Examples of things which will continue to be errors:
 
-```swift
-1/**/+ 2
-1 +/*comment*/2
+```
+!/* */foo
+1+/* */2
 ```
 
 And things which will continue to work:
 
-```swift
+```
 foo!// comment
-foo/* */?.description
-foo/* */!
 
 1 +/**/ 2
-1/**/+2
-1+/**/2
+1 +/* */2
 ```
 
 ## Alternatives considered
 
-### Treat comments as whitespace
+### Treat comments as absent
 
-We could instead specify that comments are treated as whitespace. This is a 
-simpler rule, and it avoids some extra complications described above around
-slash-slash comments and multi-line slash-star comments. It is also easier for
-both the lexer and a human reader to determine whether an operator is binary
-or not, since if comments are always whitespace, you don't have to scan to
-the other side of a long comment to tell whether an operator has whitespace
-around it. For example:
+We could instead specify that comments are treated as though they are not present
+(i.e. we would look past comments to see whether whitespace follows).
+This more-closely matches some people’s mental model of comments. It is also more
+flexible; that is, there are places where this rule permits comments which are forbidden
+by the above proposal. (e.g. `!/* */foo`)
 
-```swift
-1 +/* a very long comment */2
-```
+However, this rule is harder to describe (the comments are not ignored entirely as
+they still separate tokens) and goes against the general rule in the
+language reference that comments are treated as whitespace.
+
+This also has the disadvantage that you have to look at the other side of a
+comment to determine if an operator has whitespace around it. For example:
 
-With the proposed changes, you can't tell just by looking near the “+” whether
-it is a binary or prefix operator. 
+```
+a = 1 +/* a very long comment */~x
+```
 
-On the other hand, this goes against the common mental model that comments are
-ignored when parsing. As a result, this is somewhat more surprising rule. This
-rule is also less flexible; that is, there are places which this completely
-prohibits comments which are permitted by the above proposal. (e.g. `!/* */foo`)
+You can’t tell just by looking near the “+” whether it is a binary or prefix
+operator. (And, in fact, this would fail to parse if the comment was simply
+treated as absent.)
 
 ### A more general rule
 
 Another alternative is a more general rule about how comments are handled
 everywhere in the language (e.g. there must be no effect when replacing a
 comment with a space character). This has the advantage of possibly resolving
 other ambiguities, but has potentially far-reaching consequences in various
-edge cases which are hard to fully enumerate (e.g. multi-line comments, 
+edge cases which are hard to fully enumerate (e.g. multi-line comments,
 comments within interpolated sequences inside of string literals, comments
-in lines which contain "#" directives, etc). 
+in lines which contain "#" directives, etc).