# Adding Unicode properties to UnicodeScalar/Character

**URL:** <https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310>\
**Category:** Pitches\
**Created:** [January 30, 2018, 4:32am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310 "2018-01-30T04:32:26Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [January 30, 2018, 4:32am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/1 "2018-01-30T04:32:26Z")

</div>

Swift's support for Unicode-aware operations on `String`s is quite nice compared to many other modern programming languages, but one area where we're currently lacking is in the ability to query properties of `Unicode.Scalar` and `Character` values (e.g., to classify them).

As one possible example of how these could be implemented, I'll point to the [`UnicodeScalar+BooleanProperties`](https://github.com/allevato/icu-swift/blob/master/Sources/ICU/UnicodeScalar%2BBooleanProperties.swift) extension in my [icu-swift](https://github.com/allevato/icu-swift) project. _I make no claim that this is the ideal API to be adopted by the standard library; rather, I just want it to serve as a starting off point for discussion._

Based on previous discussions here, I think many of us agree that we need to support some subset of these capabilities (at least) in the standard library. The following is a bit of a stream-of-consciousness of what I think the broad goals/questions are. If I've left anything out, please feel free to add it!

# Which properties to support?

The Unicode standard defines a large number of properties that range from "useful in everyday computing" to "typically used only in specialized text processing algorithms." There are also may different types of properties—Boolean-valued, string-valued, numeric-valued, enumerated-type-valued, and so forth.

**A design proposal to improve the state of Unicode properties in the standard library will need to explicitly state which properties will be supported** (and by elimination, which will be omitted, if any). I won't try to be exhaustive here in my first message, but to cherry-pick some examples, properties like `Uppercase`, `Lowercase`, and `White_Space` are obvious candidates for inclusion. Something like `Logical_Order_Exception` that comes up less frequently on its own may not meet the bar that we decide to set.

# How to expose the properties?

How should we design the APIs that expose these properties? This may be motivated by how many properties we decide to support above. If the number we support is small, then individual properties for each probably makes sense (`isLowercase`, `isUppercase`, `isWhitespace`, etc.). But if we end up supporting something closer to the full set of properties, would that bloat the API? Would we want to provide something enum-based like `hasProperty(.lowercase)` instead? That would reduce bloat but it would also be quite less discoverable and less familiar compared to the `is*` APIs provided by Java, C, and others.

We also have to consider the non-Boolean properties; one example, `Numeric_Value`, may be more appropriate to expose as a failable initializer on `Int`. (Or even on `Double`; consider that the `Numeric_Value` of `U+00BD VULGAR FRACTION ONE HALF` is `0.5`.)

# What applies to `Unicode.Scalar` and what applies to `Character`?

By definition, the Unicode standard defines these properties on code points, which Swift represents using `Unicode.Scalar` (except that `Unicode.Scalar` has a "hole" where the surrogate code points live, but I don't believe that will be hugely relevant to our discussion here). Because of that, in my opinion, any properties that we expose should **at a minimum be supported on `Unicode.Scalar`.**

So the question then becomes, what should we **also** support on `Character`? `Character`s are the "default" view on `String` and will be what most users iterate over and operate on unless their use case has performance requirements that make it preferable to use `UnicodeScalarView` instead of paying the cost of calculating grapheme cluster breaks.

When we talk about `Character`s (i.e., grapheme clusters of one or more scalars), I think we can classify the properties in one of three ways:

1. Properties that are well-defined and are derivable solely based on the properties of the `Character`'s constituent `Unicode.Scalar`s. For example, consider `White_Space`. It's reasonable that a user would want to ask if a `Character` is whitespace or not. If we have a "weird" character, like a space followed by a combining accent mark, I think it's sensible to say "no, that's not whitespace". In other words, the `Character` has property X iff all of its scalars have property X.

2. Properties that have sensible definitions for `Character`s but which are not derived solely based on the constituent `Unicode.Scalar`s. For example, we expect users will want to ask "is a character lowercase", but you **can't** say a `Character` is lowercase if all of its scalars have `Lowercase == true` because if you have "a" + a combining accent mark, the combining accent has `Lowercase == false`. In specific cases like this (upper/lower/titlecase), Unicode defines specific algorithms for strings that we can apply to `Character`: a `Character` _c_ is uppercase if `toUpper(c) == c`, and likewise for lower- and titlecase.

3. Properties that make no sense to support on `Character`. One example that comes to mind is `Variation_Selector`. AFAIK, variation selectors always combine with a preceding scalar unless they are in a cluster by themselves, so you don't really gain anything by asking "is this **`Character`** a variant selector?" You should just drop down to the scalars to ask.

* * *

Thoughts?

---

<div class="post-metadata">

**Author:** ![curt](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/curt/32/1961_2.png) [@curt](https://forums.swift.org/u/curt)\
**Post date:** [January 30, 2018, 3:32pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/2 "2018-01-30T15:32:28Z")

</div>

This seems like an excellent direction for extending `UnicodeScalar` and `Character`.

Regarding the way in which the properties are exposed, it seems like we might want a rich `hasProperty()`-style API for all the defined properties. However, that doesn't preclude also having a simpler set of `isFoo` predicates for the more common properties. `CharacterSet`'s static properties are a good jumping off place for what might be considered common.

Correct me if I'm wrong, but it seems like all of this would be purely additive, right?

---

<div class="post-metadata">

**Author:** ![Michael\_Ilseman](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/michael_ilseman/32/44640_2.png) [@Michael\_Ilseman](https://forums.swift.org/u/Michael_Ilseman)\
**Post date:** [January 30, 2018, 5:31pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/3 "2018-01-30T17:31:10Z")

</div>

Thank you for kicking this off! [https://github.com/allevato/icu-swift](http://icu-swift) seems like a great testing ground for this.

> [@allevato](#):
>
> A design proposal to improve the state of Unicode properties in the standard library will need to explicitly state which properties will be supported (and by elimination, which will be omitted, if any). I won’t try to be exhaustive here in my first message, but to cherry-pick some examples, properties like Uppercase, Lowercase, and White\_Space are obvious candidates for inclusion. Something like Logical\_Order\_Exception that comes up less frequently on its own may not meet the bar that we decide to set.

This is a great way to approach the task. We may want to expose all the raw information for sophisticated use cases, and selectively bless some queries for common use.

> [@allevato](#):
>
> How should we design the APIs that expose these properties? This may be motivated by how many properties we decide to support above. If the number we support is small, then individual properties for each probably makes sense (isLowercase, isUppercase, isWhitespace, etc.). But if we end up supporting something closer to the full set of properties, would that bloat the API? Would we want to provide something enum-based like hasProperty(.lowercase) instead? That would reduce bloat but it would also be quite less discoverable and less familiar compared to the is\* APIs provided by Java, C, and others

My vote is for `isLowercase`, etc., to be present directly for common use, semantically equivalent to something built on top of a more general facility. Much like a subset of those provided by icu-swift.

> [@allevato](#):
>
> What applies to Unicode.Scalar and what applies to Character?

I feel like this will end up being a case-by-case (all puns intended) tiny research project.

> [@allevato](#):
>
> Properties that have sensible definitions for Characters but which are not derived solely based on the constituent Unicode.Scalars. For example, we expect users will want to ask “is a character lowercase”, but you can’t say a Character is lowercase if all of its scalars have Lowercase == true because if you have “a” + a combining accent mark, the combining accent has Lowercase == false. In specific cases like this (upper/lower/titlecase), Unicode defines specific algorithms for strings that we can apply to Character: a Character c is uppercase if toUpper(c) == c, and likewise for lower- and titlecase.

Case is pretty tricky. Unicode defines at least 3 different levels of thinking about case, because of course it does.

Old fashioned notions of case (such as in Java or CharacterSet) are based on general category, but that proved to not be very future-proof and skewed towards bicameral alphabets. The second level of case is from derived properties, which is likely what we'll want for `UnicodeScalar` at least. Relevant (trimmed) quote from the State of String thread:

> [@State of String: ABI, Performance, Ergonomics, and You!](https://forums.swift.org/t/state-of-string-abi-performance-ergonomics-and-you/7397/8):
>
> CharacterSet’s builtins model older Unicode semantics than what we probably want to provide. E.g. CharacterSet.lowercaseLetters means general category “Ll”, while modern Unicode defines property Lowercase which includes more scalars. The Lowercase property vs CharacterSet.lowercaseLetters is equivalent to ICU’s u\_isULowercase [1] vs u\_islower [2], where the former is the modern recommended API.
> 
> [1] u\_isULowercase: [ICU-docs - International Components for Unicode Docs | icu-docs](http://icu-project.org/apiref/icu4c/uchar_8h.html#a8321c9ba617ed00787f20c4b23a254bc)
> 
> [2] u\_islower: [ICU-docs - International Components for Unicode Docs | icu-docs](http://icu-project.org/apiref/icu4c/uchar_8h.html#aa26a51b768147fbf34029f8141132815)
> 
> [3] [https://docs.perl6.org/language/regexes#Backslashed,\_predefined\_character\_classes](https://docs.perl6.org/language/regexes#Backslashed,_predefined_character_classes)

Finally, the third level are the String functions such as `isLowercase` and `isUppercase`. I don't know if this level is overkill on Character, but off-the-cuff it seems viable. There might be some decent fast-paths in the implementation we can use for common scenarios.

AFAICT, `isLowercase` would return true for caseless graphemes, such as "7". I don't know what behavior we want to expose, e.g. perhaps a grapheme has to satisfy both Unicode's `isCased` and `isLowercase` for our Swift computed property `isLowercase`.

Or we expose both and let the user sort it out. Really depends on the use.

---

<div class="post-metadata">

**Author:** ![Michael\_Ilseman](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/michael_ilseman/32/44640_2.png) [@Michael\_Ilseman](https://forums.swift.org/u/Michael_Ilseman)\
**Post date:** [January 30, 2018, 5:33pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/4 "2018-01-30T17:33:49Z")

</div>

This is additive, but I think it addresses a very sore spot in String/Character/UnicodeScalar that can be developed in parallel with ABI efforts.

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [January 30, 2018, 7:05pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/5 "2018-01-30T19:05:11Z")

</div>

> [@Michael\_Ilseman](#):
>
> The second level of case is from derived properties, which is likely what we’ll want for UnicodeScalar at least.

Agree completely.

And thanks for mentioning the point about `CharacterSet`'s obsolete logic. IMO we should fix that as well by introducing a new and correct `UnicodeScalarSet` type, but that's best saved for a separate proposal. That being said, if these properties go in, it'll be important to fix because people will wonder why `Unicode.Scalar.isLowercase` isn't consistent with `CharacterSet.lowercaseLetters`.

> [@Michael\_Ilseman](#):
>
> Finally, the third level are the String functions such as isLowercase and isUppercase. I don’t know if this level is overkill on Character, but off-the-cuff it seems viable. There might be some decent fast-paths in the implementation we can use for common scenarios.

I think this is exactly what we want for `Character`. AFAICT, the Unicode standard doesn't distinguish between single grapheme clusters and strings of them for the purposes of case detection above the scalar level. (The exception is titlecase, since that naturally involves multiple scalars and word boundaries, but even in some specific cases, there are scalars that encode multiple "characters" that are inherently titlecased—e.g., `U+01F2: LATIN CAPITAL LETTER D WITH SMALL LETTER Z` (Dz).

So, the same case detection we'd want to use for strings _should_ be correct for characters as well.

> [@Michael\_Ilseman](#):
>
> AFAICT, isLowercase would return true for caseless graphemes, such as “7”. I don’t know what behavior we want to expose, e.g. perhaps a grapheme has to satisfy both Unicode’s isCased and isLowercase for our Swift computed property isLowercase.

That's true—the "Default Case Detection" subsection of 3.13 in the [standard](http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf) shows that uncased code points like "7" are simultaneously uppercase, lowercase, and titlecase. This means we have to make a decision for these:

- `("7" as Unicode.Scalar).isLowercased == true` because it's technically correct according to the Unicode standard, and users have to remember to check `isCased` as well.
- `("7" as Unicode.Scalar).isLowercased == false` because we would internally also factor in `isCased` and do the thing that's most obvious for the user.

I also imagine that nobody wants a world where `("7" as Unicode.Scalar).isLowercased != ("7" as Character).isLowercased`.

Edit: It's also worth noting that if we go with the second option above, there's no way to get _back_ from that result to the raw value; it's not reversible. So if somebody _does_ want the "raw" value of the case property, we'd need to provide that as a separate API.

---

<div class="post-metadata">

**Author:** ![Michael\_Ilseman](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/michael_ilseman/32/44640_2.png) [@Michael\_Ilseman](https://forums.swift.org/u/Michael_Ilseman)\
**Post date:** [January 30, 2018, 10:48pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/6 "2018-01-30T22:48:01Z")

</div>

> [@allevato](#):
>
> ("7" as Unicode.Scalar).isLowercased == false because we would internally also factor in isCased and do the thing that’s most obvious for the user.

I would imagine this would be level 2, that is, based on the `Lowercase` derived property and not the Unicode string function `isLowercase(X)`. "7" does not have that derived property. (See section 4.2 of the standard).

> [@allevato](#):
>
> I also imagine that nobody wants a world where ("7" as Unicode.Scalar).isLowercased != ("7" as Character).isLowercased.

Nano-proposal:

Requirement 1: Canonically equivalent Characters always give the same answer for these queries.

Requirement 2: A Character comprised of a single scalar gives the same answer as querying the scalar directly.

Corollary: Any Character canonically equivalent with a single-scalar Character must give the same answer as that scalar.

Research project: What, then, is a consistent model for multi-scalar graphemes not canonically equivalent to a single-scalar one?

> [@allevato](#):
>
> Edit: It’s also worth noting that if we go with the second option above, there’s no way to get back from that result to the raw value; it’s not reversible. So if somebody does want the “raw” value of the case property, we’d need to provide that as a separate API.

All case folding is irreversible; it may even change the grapheme count.

---

<div class="post-metadata">

**Author:** ![xwu](https://avatars.discourse-cdn.com/v4/letter/x/b487fb/32.png) [@xwu](https://forums.swift.org/u/xwu)\
**Post date:** [January 31, 2018, 1:16am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/7 "2018-01-31T01:16:43Z")

</div>

> [@allevato](#):
>
> How should we design the APIs that expose these properties? This may be motivated by how many properties we decide to support above. If the number we support is small, then individual properties for each probably makes sense (isLowercase, isUppercase, isWhitespace, etc.). But if we end up supporting something closer to the full set of properties, would that bloat the API? Would we want to provide something enum-based like hasProperty(.lowercase) instead? That would reduce bloat but it would also be quite less discoverable and less familiar compared to the is\* APIs provided by Java, C, and others.

Agree with earlier comments that `isLowercase`, etc., should be present directly for common use.

As for more esoteric properties, what of something like a property named `unicodeProperties` that returns an `OptionSet`? I think if feasible it'd afford more flexibility than `hasProperty`, and we're talking about advanced usage anyway.

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [January 31, 2018, 3:30am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/8 "2018-01-31T03:30:11Z")

</div>

> [@Michael\_Ilseman](#):
>
> I would imagine this would be level 2, that is, based on the Lowercase derived property and not the Unicode string function isLowercase(X). “7” does not have that derived property. (See section 4.2 of the standard).

Ah yes, you're right—I was getting wrapped up in the various definitions. In that case, it looks like the derived "Lowercase" and "Uppercase" property would directly give us what we want for scalars and single-scalar characters.

> [@Michael\_Ilseman](#):
>
> Research project: What, then, is a consistent model for multi-scalar graphemes not canonically equivalent to a single-scalar one?

AFAIK, this is where the Default Case Detection rules in 3.13 come in. So to try to restate everything:

1. For `Unicode.Scalar`s and `Character`s consisting of a single `Unicode.Scalar` or consisting of multiple scalars that are canonically equivalent to a single scalar, `isLowercase` equals the value of the single scalar's derived `Lowercase` property.

2. For `Character`s consisting of multiple scalars that are not canonically equivalent to a single scalar, then `isLowercased` is true if and only if `C == toLowercase(C) && isCased(C)`.

Case #1 is really a subset of case #2, but it presents an optimization opportunity for single scalars where we don't have to compute a temporary mapping and test equality. Overall, this behavior is consistent with what's described by 3.13 and produces the correct results for something like "a + several combining accents" (where `isCased` keeps it true) and for emoji sequences where `isCased` would be false, therefore saying the whole cluster is false.

How does that sound?

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [January 31, 2018, 3:41am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/9 "2018-01-31T03:41:05Z")

</div>

> [@xwu](#):
>
> As for more esoteric properties, what of something like a property named unicodeProperties that returns an OptionSet? I think if feasible it’d afford more flexibility than hasProperty, and we’re talking about advanced usage anyway.

This hadn't occurred to me, but I really like it. It moves the "bloat" out of the main `Unicode.Scalar` interface and into its own type for advanced users.

One concern: ICU 60 already defines 64 boolean properties, which would exhaust a `UInt64` that we would use as the raw value of the `OptionSet`. What do we do if Unicode adds another property? We can expand our bit space with `DoubleWidth<>`, but since the underlying integer type is part of the public API of an `OptionSet`, can we safely scale it in the future in a non-breaking way?

---

<div class="post-metadata">

**Author:** ![xwu](https://avatars.discourse-cdn.com/v4/letter/x/b487fb/32.png) [@xwu](https://forums.swift.org/u/xwu)\
**Post date:** [January 31, 2018, 4:10am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/10 "2018-01-31T04:10:30Z")

</div>

> [@allevato](#):
>
> One concern: ICU 60 already defines 64 boolean properties, which would exhaust a UInt64 that we would use as the raw value of the OptionSet. What do we do if Unicode adds another property? We can expand our bit space with DoubleWidth\<\>, but since the underlying integer type is part of the public API of an OptionSet, can we safely scale it in the future in a non-breaking way?

Are any of the 64 properties easily derived from trivial combinations of the others? If so, then it may be feasible. If not, then we might need to look into other designs.

---

<div class="post-metadata">

**Author:** ![jawbroken](https://avatars.discourse-cdn.com/v4/letter/j/cc9497/32.png) [@jawbroken](https://forums.swift.org/u/jawbroken)\
**Post date:** [January 31, 2018, 4:12am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/11 "2018-01-31T04:12:07Z")

</div>

Efficiency and practicality aside, I feel that they would properly be modelled by a `Set` of enum cases and not an `OptionSet`. In reality you probably don't want to have to actually construct such a `Set` by querying all properties at construction, so why not just a custom type that conforms to `SetAlgebra`, with the `Element` being an enum of properties?

---

<div class="post-metadata">

**Author:** ![xwu](https://avatars.discourse-cdn.com/v4/letter/x/b487fb/32.png) [@xwu](https://forums.swift.org/u/xwu)\
**Post date:** [January 31, 2018, 4:15am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/12 "2018-01-31T04:15:28Z")

</div>

> [@jawbroken](#):
>
> Efficiency and practicality aside, I feel that they would properly be modelled by a Set of enum cases and not an OptionSet. In reality you probably don’t want to have to actually construct such a Set by querying all properties at construction, so why not just a custom type that conforms to SetAlgebra, with the Element being an enum of properties?

Each character has multiple properties.  
Edit: I see what you mean--a _set_ of enum cases. That's intriguing, but would we end up having too many types? Is there that much structure among the properties such that we have a deep hierarchy of many mutually exclusive options?

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [January 31, 2018, 4:26am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/13 "2018-01-31T04:26:11Z")

</div>

The more I think about it, the less a "set" (option set or regular set) fits with the API that ICU gives us.

There's no way to query "give me all the boolean properties of this scalar" as a single bitmask. The only function we have AFAICT is [`u_hasBinaryProperty`](http://icu-project.org/apiref/icu4c/uchar_8h.html#a25c5c820d4141e4099acc15ca83572a5), which only lets us query them one at a time. That means that if we want to support a true set type, we have to query all 64 properties any time someone wants just one of them, which seems like a poor implementation strategy.

---

<div class="post-metadata">

**Author:** ![jawbroken](https://avatars.discourse-cdn.com/v4/letter/j/cc9497/32.png) [@jawbroken](https://forums.swift.org/u/jawbroken)\
**Post date:** [January 31, 2018, 4:55am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/14 "2018-01-31T04:55:04Z")

</div>

You wouldn't need to do that if you use a custom type that conforms to `SetAlgebra`. e.g. Very roughly

```swift
enum CharacterProperty {
    case upperCase, lowerCase, deprecated, diacritic // etc, possibly with raw values that match ICUs UProperty
}

struct CharacterProperties: SetAlgebra {
    typealias Element = CharacterProperty
    
    func contains(_ member: CharacterProperty) -> Bool {
        u_hasBinaryProperty(...) // map member to UProperty, call ICU
    }
    // etc
}

```

Some of the `SetAlgebra` functions would require querying all properties, though.

However, this is only if being a set is particularly desirable, which it probably isn't unless you expect people to be e.g. intersecting the properties of all characters in a string to determine what properties they have in common. A function that takes an enum, such as hasProperty mentioned above, makes the most sense to me, with shorthands for common properties.

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [January 31, 2018, 3:42pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/15 "2018-01-31T15:42:02Z")

</div>

> [@jawbroken](#):
>
> Some of the SetAlgebra functions would require querying all properties, though.

Most of them would, if we implement `contains` the most efficient way by calling `u_hasBinaryProperty` directly and not caching anything. So yeah, that would end up being even worse than `OptionSet`.

---

<div class="post-metadata">

**Author:** ![Michael\_Ilseman](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/michael_ilseman/32/44640_2.png) [@Michael\_Ilseman](https://forums.swift.org/u/Michael_Ilseman)\
**Post date:** [January 31, 2018, 9:04pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/16 "2018-01-31T21:04:27Z")

</div>

I'd say defer any approach that cannot be efficiently implemented on top of ICU as future work. We can consider what an ideal future would look like, but let's also separate out something concrete for inclusion in Swift 5.

---

<div class="post-metadata">

**Author:** ![Michael\_Ilseman](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/michael_ilseman/32/44640_2.png) [@Michael\_Ilseman](https://forums.swift.org/u/Michael_Ilseman)\
**Post date:** [January 31, 2018, 9:05pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/17 "2018-01-31T21:05:43Z")

</div>

> [@allevato](#):
>
> Case #1 is really a subset of case #2

Establishing this was where my individual research left off. Do you have a relevant part of the spec, justification, and/or an argument for why this must hold?

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [February 2, 2018, 6:27pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/18 "2018-02-02T18:27:38Z")

</div>

> [@Michael\_Ilseman](#):
>
> Establishing this was where my individual research left off. Do you have a relevant part of the spec, justification, and/or an argument for why this must hold?

I've dug through the spec a bit and can't find specific writing for these assertions but I **believe** they hold:

First, let's take Case 2:

> [@](#):
>
> For `Character`s consisting of multiple scalars that are not canonically equivalent to a single scalar, then `isLowercased` is true if and only if `C == toLowercase(C) && isCased(C)`.

Then, Case 1 was written as such:

> [@](#):
>
> For `Unicode.Scalar`s and `Character`s consisting of a single `Unicode.Scalar` or consisting of multiple scalars that are canonically equivalent to a single scalar, `isLowercase` equals the value of the single scalar’s derived `Lowercase` property.

So let's try to restate it in terms of Case 2. Let's say we have a single scalar `S`. Then we need to show that the derived `Lowercased` property of `S` is always equivalent to `S == toLowercase(S) && isCased(S)`. The cases are:

1. `S` is an un-cased code point. Then its `Lowercased` derived property should be false. Likewise, `isCased(S)` will be false, so the two are equivalent.
2. `S` is a cased code point. Then,  
a. `S` has `Lowercased` property == `true`. Then `S == toLowercase(S)` is true because `S` doesn't get changed, and `isCased(S)` is true, so they're equivalent.  
b. `S` has `Lowercased` property == `false`. Then `S == toLowercase(S)` is false, so the whole expression is false.

Unfortunately I can't find more concrete properties in the spec that clarify this, but I believe this will always hold?

---

<div class="post-metadata">

**Author:** ![Michael\_Ilseman](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/michael_ilseman/32/44640_2.png) [@Michael\_Ilseman](https://forums.swift.org/u/Michael_Ilseman)\
**Post date:** [February 2, 2018, 7:11pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/19 "2018-02-02T19:11:25Z")

</div>

> [@allevato](#):
>
> I **believe** they hold

If I had a nickel for every time I said that! This is Unicode, and thus we throw all intuition out the door and also consult the tables:

> [@](#):
>
> If there are any cases of mismatches between the definition of a derived property as listed in DerivedCoreProperties.txt or similar data files in the UCD, and the definition of a derived property as a set definition rule, the explicit listing in the data file should always be taken as the normative definition of the property.

😓

For lowercase/uppercase, I think we still need more justification based on the defined semantics of these methods. From the spec:

> [@](#):
>
> The following rules specify the default case conversion operations for Unicode strings. These rules use the full case conversion operations, Uppercase\_Mapping(`C`), Lowercase\_Mapping(`C`), and Titlecase\_Mapping(`C`), as well as the context-dependent mappings based on the casing context, as specified in Table 3-14.  
> For a string X:
> 
> - R1 toUppercase(X): Map each character C in X to Uppercase\_Mapping(`C`).
> - R2 toLowercase(X): Map each character C in X to Lowercase\_Mapping(`C`).

Then the next step is to determine what this mapping is, and if the mapping is invariant for all unicode scalars with the relevant property. Basically, we need to prove:

∀(s ∈ UnicodeScalars) isLowercase(s) === Lowercase\_Mapping(s) == s

We can validate this for given a version of Unicode through exhaustive search (it's only a million scalars, so this can be done in about a second). We can do this today!

To reason about whether this will remain valid in future versions of Unicode, we need to reason through how Uppercase\_Mapping(`C`) is defined and whether it's compatibility affordances is similar/equivalent to the derived properties. That is, we also need to prove:

∀(V ∈ FutureUnicodeVersions, sv ∈ V.UnicodeScalars) isLowercase(sv) === Lowercase\_Mapping(sv) == sv

with a fairly high degree of confidence.

From the spec:

> [@](#):
>
> The fact that a given Unicode character property is normative does not mean that the values of the property will never change for particular characters. Corrections and extensions to the standard in the future may require minor changes to normative values, even though the Unicode Technical Committee strives to minimize such changes. See also “Stability of Properties” later in this section.

(Commentary from me and not proven): Note that casing is normative but not fixed, however this is ok so long as the case mappings are always in sync. String provides universal semantics by default and leaves localization to the platform, so we're not interested in `SpecialCasing` properties. Since we're talking about graphemes, and not whole Strings, position-dependent case mapping (such as the Greek sigma) are not relevant. So in this case (all puns intended), we only care about `SimpleCasing`, assuming it aligns with the other derived properties.

So I _think_ the next step is to run the experiment on today's Unicode version and see.

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [February 3, 2018, 9:30pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/20 "2018-02-03T21:30:57Z")

</div>

> [@Michael\_Ilseman](#):
>
> Basically, we need to prove:
> 
> ∀(s ∈ UnicodeScalars) isLowercase(s) === Lowercase\_Mapping(s) == s

Are we conflating strings and scalars? Because AFAICT this statement is tautological (or trivially proven) based on the definitions in the spec:

- `isLowercase(X)` is a function that the Unicode spec defines on _strings_ (3.13, D139) that is true when `toLowercase(X) == X`
- `toLowercase(X)` is also defined on strings (3.13, R2) as mapping each character `C` in `X` to `Lowercase_Mapping(C)`
- `Lowercase_Mapping` is a _string_-valued property of scalars; the corresponding scalar-valued property is `Simple_Lowercase_Mapping`

So I don't think we can evaluate the equation to prove without first treating the scalars first as strings in some contexts. If we do so, we end up with:

> ∀(s ∈ UnicodeScalars) isLowercase(String(s)) === Lowercase\_Mapping(s) == String(s)

Then expanding:

> ∀(s ∈ UnicodeScalars) toLowercase(String(s)) == String(s) === Lowercase\_Mapping(s) == String(s)

And since `String(s)` consists of a single scalar, `toLowercase(String(s))` is equal by definition to `Lowercase_Mapping(s)`.

But that doesn't tell us anything about whether we can use the derived `Lowercase` property of a scalar as an optimization equivalent to performing these string comparisons, does it?

I started running some experiments to test that assertion, and I found some problems already, just in the first 256:

```plaintext
Mapping mismatch for scalar ª (U+aa): isLowercase(s) && isCased(s) == false, but Lowercase property == true
Mapping mismatch for scalar º (U+ba): isLowercase(s) && isCased(s) == false, but Lowercase property == true

```

This means it looks like we can't use the Lowercase property on its own for scalars if we want to be consistent with the definition how it's defined for strings 😔

So, I'll posit that our API should probably be something like:

- `Unicode.Scalar.hasProperty(.lowercase)` – a low-level operation that returns the Unicode property value directly
- `Unicode.Scalar.isLowercase` – promotes the scalar to a `String` and uses the transformation/detection functions defined by the Unicode spec
- `Character.isLowercase` – promotes the character to a `String` and uses the transformation/detection functions defined by the Unicode spec

...and likewise for uppercase (and titlecase?).

[Next page](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310.md?page=2)