[Pitch] String Input-validating Initializers

[Michael_Ilseman]

@glessard

What I failed to say clearly earlier is that since the throwing version seems to require the input to be Collection rather than Sequence, it isn't strictly more general than the failable initializers proposed here. Given that, I think there is a justification for both kinds of initializers.

I wouldn't assign a very high value to using Sequence over Collection for these inits. Much of Sequence outside of Collection is a historical accident that we didn't have time to fix up before ABI stability. It's also asymmetric with the other initializers we have on String. It's nice when you get if for free, though.

Another side-benefit of constraining over Collection is that I believe all of this can be always-emit-into-client (swift/stdlib/public/core/StringCreate.swift at 9e73dad31110312356469364ba314603d8cff4b7 · swiftlang/swift · GitHub).

---

@Karl

Speaking of which, we could really do with a String initialiser which accepts a sequence/collection of Unicode scalars.

Agreed.

@Alejandro

We've also been thinking about adding conveniences for that initializer like String(utf8: x) etc. I think these shorthand initializers make reaching for the right thing a lot easier, but that's for another proposal.

There's a balance here where on the one hand we want to allow for smaller pitches to the stdlib and on the other we want to make sure the total API is fleshed out.

String(utf8: myBytes) is long-overdue as creating a String from UTF-8 needs better discoverability. If we're adding input-validating convenience overloads, they will show up in code completion and it's possible that common usage would be to use them with a force-unwrap at the end, when I think it would be better for common usage to do input-correction.

This also very quickly opens the door to discussing input-normalizing API.

1. decoding:as: preserves exact scalar values and does input-correction
2. validating:as: preserves exact scalar values and does failable input-validation
3. normalizing:as: normalizes/canonicalizes scalar values, speeding up comparisons and searches, and does input-correction

And a question for String(utf8: myBytes) is whether it has the semantics of 1 or 3.

---

@davedelong

My file format parsers lazily loaded chunks of bytes

This is pitching initializers that, in the process of creating a String, fail if the input is invalidly encoded.

A separate and greatly needed batch of functionality is better Unicode processing, validating, and correcting APIs over code units (particularly when contiguous in memory). Briefly (using straw-person names and [Roadmap] Language support for BufferView):

1. struct BufferView: ~Escapable based functionality and ABI
2. protocol BufferViewable based inlinable (or always-emit-into-client) API
3. Chunking API over [Async]Sequence<CodeUnit> (i.e. create a moving window of in-memory code units and handle truncation)

Validation can throw specific errors in terms of e.g. the nth code unit in the input.

I believe this would also be more appropriate for this kind of use than trying to create a String, especially if the input is incomplete or very large.

@wes1

Would it help to consider the alternative of a static factory or converting function?

Relatedly, there's also the decodeCString static functions which give you more precise info:

@inlinable public static func decodeCString<Encoding: _UnicodeEncoding>(
  _ cString: UnsafePointer<Encoding.CodeUnit>?,
  as encoding: Encoding.Type, 
  repairingInvalidCodeUnits isRepairing: Bool = true
) -> (result: String, repairsMade: Bool)? 

@inlinable public static func decodeCString<Encoding: _UnicodeEncoding>(
  _ cString: [Encoding.CodeUnit],
  as encoding: Encoding.Type,
  repairingInvalidCodeUnits isRepairing: Bool = true
) -> (result: String, repairsMade: Bool)? where Encoding : _UnicodeEncoding

It's possible we could in the future have something like a static func decodeBytes(_: some BufferViewable) throws -> String which gives you much more error information as well as creating the String.