# SE-0363: Unicode for String Processing **URL:** \ **Category:** Proposal Reviews\ **Created:** [June 27, 2022, 9:08pm UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520 "2022-06-27T21:08:20Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![ben-cohen](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ben-cohen/32/296_2.png) [@ben-cohen](https://forums.swift.org/u/ben-cohen)\ **Post date:** [June 27, 2022, 9:08pm UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/1 "2022-06-27T21:08:20Z")
Hello, Swift Community. The review of [SE-0363: Unicode for String Processing](https://github.com/apple/swift-evolution/blob/main/proposals/0363-unicode-for-string-processing.md) begins now and runs through July 11, 2022. Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the [review manager](mailto:ben_cohen@apple.com). If you do email me directly, please put "SE-0351" somewhere in the subject line. ## What goes into a review? The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review: - What is your evaluation of the proposal? - Is the problem being addressed significant enough to warrant a change to Swift? - Does this proposal fit well with the feel and direction of Swift? - If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those? - How much effort did you put into your review? A glance, a quick reading, or an in-depth study? More information about the Swift evolution process is available at: > [https://github.com/apple/swift-evolution/blob/main/process.md](https://github.com/apple/swift-evolution/blob/main/process.md) As always, thank you for contributing to Swift. Ben Cohen Review Manager ---
**Author:** ![paiv](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/paiv/32/4041_2.png) [@paiv](https://forums.swift.org/u/paiv)\ **Post date:** [June 28, 2022, 7:35am UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/2 "2022-06-28T07:35:48Z")
Maybe there should be a shorter direct representation from ```swift /\p{L}/ ``` to ```swift Regex { CharacterClass(.generalCategory(.uppercaseLetter), .generalCategory(.lowercaseLetter), .generalCategory(.titlecaseLetter), .generalCategory(.modifierLetter), .generalCategory(.otherLetter)) } ``` - [swift-experimental-string-processing/GeneralCategory.swift at main · apple/swift-experimental-string-processing · GitHub](https://github.com/apple/swift-experimental-string-processing/blob/main/Utils/gen-unicode-data/Sources/GenScalarProps/GeneralCategory.swift) ---
**Author:** ![paiv](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/paiv/32/4041_2.png) [@paiv](https://forums.swift.org/u/paiv)\ **Post date:** [June 28, 2022, 8:26am UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/3 "2022-06-28T08:26:26Z")
Typo ```diff - Regex syntax (?D): Match only ASCII members for all POSIX properties (including digit, space, and word). + Regex syntax (?P): Match only ASCII members for all POSIX properties (including digit, space, and word). ``` ---
**Author:** ![aerobounce](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/aerobounce/32/14175_2.png) [@aerobounce](https://forums.swift.org/u/aerobounce)\ **Post date:** [June 29, 2022, 8:19pm UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/4 "2022-06-29T20:19:31Z")
Missing link at [L1240](https://github.com/apple/swift-evolution/blob/7b816f811259a39c227470a274573a0d9447daac/proposals/0363-unicode-for-string-processing.md?plain=1#L1240) ```swift ... [Run-time Regex Construction proposal][literals] ``` Extras [at the end of L1321](https://github.com/apple/swift-evolution/blob/7b816f811259a39c227470a274573a0d9447daac/proposals/0363-unicode-for-string-processing.md?plain=1#L1321) ```diff - let regex1 = /\p{name=latin lowercase a}/.extendUnicodeProperty(\.name, by: .firstScalar)`. + let regex1 = /\p{name=latin lowercase a}/.extendUnicodeProperty(\.name, by: .firstScalar) ``` ---
**Author:** ![ksluder](https://avatars.discourse-cdn.com/v4/letter/k/eada6e/32.png) [@ksluder](https://forums.swift.org/u/ksluder)\ **Post date:** [June 29, 2022, 9:46pm UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/5 "2022-06-29T21:46:26Z")
This is an extremely minor point, but I’d appreciate if the proposal title mentioned regex to make it easier to find later. ---
**Author:** ![xwu](https://avatars.discourse-cdn.com/v4/letter/x/b487fb/32.png) [@xwu](https://forums.swift.org/u/xwu)\ **Post date:** [July 3, 2022, 4:35pm UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/6 "2022-07-03T16:35:42Z")
> To support both matching on `String`'s default character-by-character view and more broadly-compatible Unicode scalar-based matching, you can select a matching level for an entire regex or a portion of a regex constructed with the `RegexBuilder` API. [...] > > These specific levels of matching [or to be specific, extended grapheme cluster-based matching rather than Unicode scalar-based matching], and the options to switch between them, are unique to Swift I'm concerned that extended grapheme cluster-based matching is both unique to Swift and also the (silent) default. Compatibility with other regex engines has been a key consideration in this series of proposals, and it is specifically desired with respect to the classical regex syntax. As the proposal acknowledges, extended grapheme cluster-based matching by default decreases that level of broad compatibility. Moreover, since there is no "inside-the-regex" syntax to enable the more compatible Unicode scalar mode and only the rather more verbose `.matchingSemantics(.unicodeScalar)`, I worry about discoverability _even if_ a user knows about this compatibility difference. That said, I do understand the rationale for not including totally new regex syntax, at least at this time, and could probably be on board with that approach were it not for the first consideration above about defaults. In addition, threading these two "matching semantics" modes through the entire proposal also means that we have to review two distinct sets of character classes, one of them unique to Swift. They look very well thought out, and it is certainly possible that the proposed semantics for each character class is optimal as-is. But do we have a _deep_ enough set of Unicode expertise in this community to approximate a "many eyes" approach to evaluating these details? I am skeptical of this. How many people have an informed opinion on halfwidth forms, for instance? It is unclear what we would do down the line (with source compatibility constraints, etc.) if, upon reflection, some definition adopted today turns out to have included or excluded characters in error or conflicts with a future Unicode decision—the Unicode Consortium makes certain backwards compatibility guarantees with respect to its own decisions but isn't under any obligation not to contradict what we decide. Therefore, I wonder if there is another approach possible here—something along the lines below—which would satisfy the overarching (legitimate) need for Unicode correctness by default and consistency with the user-facing representation of `String` as a sequence of extended grapheme clusters as elements, while obviating the need for a distinct extended grapheme cluster-based regex matching scheme: - Support a "Unicode normalization–insensitive" option for regex, enabled by default for regex literals - Where the "Unicode normalization–insensitive" option is enabled, normalize the regex and match against the normalized string representation (i.e., do automatically what the proposal says one must do manually when using regex engines in other languages) - Drop grapheme cluster-level semantics from the proposal; in place of `matchingSemantics`, add an `ignoresNormalizationForm()` option (strawman name) that defaults to being enabled, and possibly an in-regex syntax for toggling the same option * * * I have minor concerns re some spellings (e.g., `dotMatches...` versus `anchorsMatch...`—can't we use the singular for both flags, particularly since it's possible to use only one anchor?) and character class definitions (e.g., the mismatch where `isHexDigit` is more narrowly defined than the `hexDigit` class), which can probably be discussed later. ---
**Author:** ![YOCKOW](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/yockow/32/20636_2.png) [@YOCKOW](https://forums.swift.org/u/YOCKOW)\ **Post date:** [July 5, 2022, 12:32am UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/7 "2022-07-05T00:32:05Z")
> [@xwu](#): > > I'm concerned that extended grapheme cluster-based matching is both unique to Swift and also the (silent) default. Compatibility with other regex engines has been a key consideration in this series of proposals, and it is specifically desired with respect to the classical regex syntax. As the proposal acknowledges, extended grapheme cluster-based matching by default decreases that level of broad compatibility. I agree, while my opinion was [expressed in a different way](https://forums.swift.org/t/pitch-regular-expression-literals/52820/64). > [@xwu](#): > > - Support a "Unicode normalization–insensitive" option for regex, enabled by default for regex literals > - Where the "Unicode normalization–insensitive" option is enabled, normalize the regex and match against the normalized string representation (i.e., do automatically what the proposal says one must do manually when using regex engines in other languages) > - Drop grapheme cluster-level semantics from the proposal; in place of `matchingSemantics`, add an `ignoresNormalizationForm()` option (strawman name) that defaults to being enabled, and possibly an in-regex syntax for toggling the same option I may have read it perversely, but I wonder why you use the term "normalization". 🤔 I mean we can't _ignore_ some normalization forms such as NFKC and NFKD that are not for the purpose of searching. ---
**Author:** ![Alejandro](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/alejandro/32/46197_2.png) [@Alejandro](https://forums.swift.org/u/Alejandro)\ **Post date:** [July 5, 2022, 9:05pm UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/8 "2022-07-05T21:05:20Z")
> [@xwu](#): > > - Support a "Unicode normalization–insensitive" option for regex, enabled by default for regex literals > - Where the "Unicode normalization–insensitive" option is enabled, normalize the regex and match against the normalized string representation (i.e., do automatically what the proposal says one must do manually when using regex engines in other languages) > - Drop grapheme cluster-level semantics from the proposal; in place of `matchingSemantics`, add an `ignoresNormalizationForm()` option (strawman name) that defaults to being enabled, and possibly an in-regex syntax for toggling the same option Correct me if I'm wrong, but it sounds like what you're proposing is a normalize every input string and match by scalar default. This model actually works pretty well until you come across a language whose `Character`s are not represented by a single scalar grapheme. Notoriously, this happens to be emojis, but there are also Indic sequences who follow this rule as well (there might be other cases!). Consider the following: ```swift // This is a single grapheme composed of 5 scalars. let string1 = "क़्‍त" // This is a single grapheme composed of 7 scalars. let string2 = "👨‍👩‍👧‍👦" // Prints: 1 print(string1.count) // Prints: 1 print(string2.count) // Match any single character let regex = /./ // Proposed: grapheme based matching semantics // Prints: क़्‍त print(try regex.firstMatch(in: string1)!) // Prints: 👨‍👩‍👧‍👦 print(try regex.firstMatch(in: string2)!) // ignoresNormalize and scalar based matching semantics // Prints: क print(try regex.firstMatch(in: string1)!) // Prints: 👨 print(try regex.firstMatch(in: string2)!) ``` Without grapheme based matching semantics we fail to match `String`'s single character, instead we get something that doesn't even appear to be in our original input. For someone who has never used regex before, these results seems bizarre! This is pretty inconsistent with the rest of our language model where we want to be Unicode correct. Having different outputs by default between `string.count` and regex `/./` seems like a direction that isn't toward being Unicode correct, even if it means being compatible with classical regex engines. From the get go, our string model is vastly different than that of other language's strings, so we have to diverge from being 100% compatible with other engines to ensure that we're consistent with what model we currently have and to continue trying to be as Unicode correct as possible. ---
**Author:** ![Michael\_Ilseman](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/michael_ilseman/32/44640_2.png) [@Michael\_Ilseman](https://forums.swift.org/u/Michael_Ilseman)\ **Post date:** [July 17, 2022, 6:00pm UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/9 "2022-07-17T18:00:33Z")
I'd like to give a little context to @Alejandro's reply regarding String's philosophy. This is ever-evolving but as we add more API and functionality to String, it starts to become clearer. When it comes to strings and Unicode, there is no universal correctness and strings are messy messy things. Swift uses Unicode out of pragmatism, as it is the very best we have, despite its flaws. Unicode does not directly prescribe or dictate a programming model, it's a collection of some definitions and occasional suggestions. It's an art and a science to eek out some sensible programming model from it. Trying to doggedly assign meaning to the meaningless is a fruitless endeavor at best, and can harm realistic usage at worst. It's better to find some principles. Similarly, these principles are not to be doggedly held to all costs. If the principles were sufficient to define a clear and correct universal programming model, then strings would be easy. ### Swift's default string model: `Character`s Swift's primary, default model of `String` is a collection of `Character`s ("extended" grapheme clusters) equal under Unicode canonical equivalence. In Unicode, grapheme clusters are defined for the sole purpose of having renderers mostly agree with each other. Unicode says little to nothing more about them. Grapheme breaking is designed to be simple enough for renderers, at the cost of allowing all kinds of senseless constructs and corner cases in "unrealistic" usage. Unicode's recommendation (for renderers) is to have some reasonable fall-back behavior for the weird cases that don't arise organically, in order to allow the realistic cases to follow a more consistent model. Swift chose to base its string model on top of grapheme clusters. Swift has to venture forth on its own here. Algorithms against Swift's model of string will be semantically incompatible with those algorithms against a different model of string. String provides ways to explicitly use a different model of string (for example the scalar and code unit views). Weirdness can result from this decision, so we try to interpret Unicode's guidance as we develop some principles and choose between tradeoffs. - Principle: Degenerate cases can be weird in service of making realistic and important usage better For example, `str1.count + str2.count == (str1 + str2).count` does not hold when `str2`'s leading `Character` would combine with `str1`'s trailing `Character`, breaking algebraic reasoning in this situation. But, that would be an example of `str2` being an inorganic case ("degenerate"). Swift made the call that `String`'s `RangeReplaceableCollection` conformance was so important for many practical reasons that an inconsistency under degenerate cases was an acceptable compromise. - Principle: Normal/default API should not create degenerate cases where there originally were none - Corollary: any indices produced by normal/default operations should produce `Character`-aligned indices to avoid degeneracy We should be hesitant to add functionality to String that could produce non-`Character`-aligned indices unless explicitly opted into. I realize this is currently violated by some of the inherited NSString APIs from Objective-C, and it's on-going work to replace them with APIs that are better and more ergonomic. * * * > [@xwu](#): > > I'm concerned that extended grapheme cluster-based matching is both unique to Swift and also the (silent) default. Compatibility with other regex engines has been a key consideration in this series of proposals, and it is specifically desired with respect to the classical regex syntax. As the proposal acknowledges, extended grapheme cluster-based matching by default decreases that level of broad compatibility. Note that there are multiple aspects of `Regex` regarding the concept of "compatibility", and there are many things which `Regex` could aim for compatibility with. There's the syntax of run-time and literal regexes, the behaviors associated with constructs such as repetition, the targeted feature set, and then there's the model of string to which a regex is applied. A regex declares an algorithm over a model of string, and this proposal establishes `String.String` as the default _semantic_ model to be compatible with. E.g. `Regex { CharacterClass.any }` will match `String.first`. > [@xwu](#): > > there is no "inside-the-regex" syntax to enable the more compatible Unicode scalar mode This is [future work](https://github.com/apple/swift-evolution/blob/main/proposals/0363-unicode-for-string-processing.md#regex-syntax-for-matching-level) for a couple reasons. We're a little hesitant to add brand new regex syntax (with few eyeballs) at the same time we're introducing regex syntax in the first place. We'd also like to give more consideration to what a _byte semantics_ of regex could be like. For example, applying a regex to a collection of `UInt8`s under the interpretation of UTF-8, though we'd need to figure out what the encoding validation story is there. > [@xwu](#): > > In addition, threading these two "matching semantics" modes through the entire proposal also means that we have to review two distinct sets of character classes, one of them unique to Swift. > ... > It is unclear what we would do down the line (with source compatibility constraints, etc.) if, upon reflection, some definition adopted today turns out to have included or excluded characters ... Note that the scalar semantics definition is directly prescribed by Unicode in [UTS-18](https://www.unicode.org/reports/tr18/). Unicode doesn't describe grapheme-cluster semantics at all and thus we don't risk incompatibility with Unicode itself; if they one day decide to do so then that's a whole new design area for the future. Many of the common grapheme cluster semantic definitions are equivalent to the [SE-0221](https://github.com/apple/swift-evolution/blob/master/proposals/0221-character-properties.md) definitions, and similar reasoning can apply to the other common properties. [Future work](https://github.com/apple/swift-evolution/blob/main/proposals/0363-unicode-for-string-processing.md#api-for-overriding-unicode-property-mapping) includes the ability to adjust or dictate how properties are extended. For the less common queries, or those that don't have as obvious an extension to grapheme cluster semantics, I think a conservative approach would be to treat the `Extension` column as non-normative. I.e. we pick a suitable default behavior but we're not formally locking that in until there's a clear need to revisit it with more information. There's a decently high chance that they're never revisited anyways, and a developer who cares about obscure Unicode details may want to work in the more precise scalar-semantic mode (which supports `\X` and `\Y` or `anyGraphemeCluster` and `graphemeClusterBoundary`). An implementation strategy could be to throw a compilation error when in grapheme-semantic mode for these fairly obscure corner cases, encouraging the use of the more precise scalar semantics mode. > [@xwu](#): > > Support a "Unicode normalization–insensitive" option for regex, enabled by default for regex literals I believe you are referring to the option to enable matching under Unicode canonical equivalence, such as [in Java](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html#CANON_EQ). I think this could be fine future work. Grapheme-semantic mode enables it by default (which is very natural as normalization segments are always sub-sequences of grapheme clusters) and scalar-semantics disabled it by default. But it could be useful to selectively enable it in scalar semantics (or disable it in grapheme-cluster semantics, though that seems less useful). ---
**Author:** ![scanon](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/scanon/32/2445_2.png) [@scanon](https://forums.swift.org/u/scanon)\ **Post date:** [August 30, 2022, 8:30pm UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/10 "2022-08-30T20:30:20Z")
This proposal has been [accepted](https://forums.swift.org/t/accepted-se-0363-unicode-for-string-processing/59998), thank you everyone! Steve ---
**Author:** ![John\_McCall](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/john_mccall/32/1351_2.png) [@John\_McCall](https://forums.swift.org/u/John_McCall)\ **Post date:** [August 30, 2022, 9:39pm UTC](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520/11 "2022-08-30T21:39:44Z")
Closed on August 30, 2022, 9:39pm UTC