Newlines in Text Streams

[Michael_Ilseman]

@SDGGiesbrecht

Given the following quotation from its documentation comment , I imagine a lot of other surprises would be encountered first.

/// Standard input is interpreted as `UTF-8`. Invalid bytes are replaced by
/// Unicode [replacement characters][rc].

Hah, good point!

@benrimmington

I don't know how useful or harmful it might be to recognize E2 80 A9 (U+2029: PARAGRAPH SEPARATOR) as a newline.

Probably doesn't matter for PS. But, if a user has VT in their input, they might expect that to be preserved. I don't think we have to worry too much about these corner cases, just trying to figure out at what level we want our semantics to operate.

@benrimmington

If strippingNewline is true, "field\r\n" would become "field" on all platforms. The recommendations in §5.8 Newline Guidelines are:

Note that even if an implementer knows which characters represent NLF on a particular platform, CR, LF, CRLF, and NEL should be treated the same on input and in interpretation. Only on output is it necessary to distinguish between them.

Good point

@benrimmington

I think it's fine to keep the existing behavior. I'll close the pull request, but can SR-1280 also be closed?

As I said, it was a weakly held opinion and you both brought up good points :-). I was thinking of readLine as being a dual of print, doing a light "text-mode". I could see us changing it to be a basic Unicode rich version. I wouldn't want to throw Character in to the mix right now, grapheme breaking is weird and always changing (e.g. there's been drafts proposing all contiguous whitespace be treated as a single grapheme).

If you want to push for this small behavior change, it should be discussed in some capacity on SE. It might classify as a minor adjustment or behavior-changing bug fix, so maybe not a full proposal. Closing is also fine.

WDYT?