Adding Unicode properties to UnicodeScalar/Character

[Michael_Ilseman]

Thank you for kicking this off! https://github.com/allevato/icu-swift seems like a great testing ground for this.

@allevato

A design proposal to improve the state of Unicode properties in the standard library will need to explicitly state which properties will be supported (and by elimination, which will be omitted, if any). I won’t try to be exhaustive here in my first message, but to cherry-pick some examples, properties like Uppercase, Lowercase, and White_Space are obvious candidates for inclusion. Something like Logical_Order_Exception that comes up less frequently on its own may not meet the bar that we decide to set.

This is a great way to approach the task. We may want to expose all the raw information for sophisticated use cases, and selectively bless some queries for common use.

@allevato

How should we design the APIs that expose these properties? This may be motivated by how many properties we decide to support above. If the number we support is small, then individual properties for each probably makes sense (isLowercase, isUppercase, isWhitespace, etc.). But if we end up supporting something closer to the full set of properties, would that bloat the API? Would we want to provide something enum-based like hasProperty(.lowercase) instead? That would reduce bloat but it would also be quite less discoverable and less familiar compared to the is* APIs provided by Java, C, and others

My vote is for isLowercase, etc., to be present directly for common use, semantically equivalent to something built on top of a more general facility. Much like a subset of those provided by icu-swift.

@allevato

What applies to Unicode.Scalar and what applies to Character?

I feel like this will end up being a case-by-case (all puns intended) tiny research project.

@allevato

Properties that have sensible definitions for Characters but which are not derived solely based on the constituent Unicode.Scalars. For example, we expect users will want to ask “is a character lowercase”, but you can’t say a Character is lowercase if all of its scalars have Lowercase == true because if you have “a” + a combining accent mark, the combining accent has Lowercase == false. In specific cases like this (upper/lower/titlecase), Unicode defines specific algorithms for strings that we can apply to Character: a Character c is uppercase if toUpper(c) == c, and likewise for lower- and titlecase.

Case is pretty tricky. Unicode defines at least 3 different levels of thinking about case, because of course it does.

Old fashioned notions of case (such as in Java or CharacterSet) are based on general category, but that proved to not be very future-proof and skewed towards bicameral alphabets. The second level of case is from derived properties, which is likely what we'll want for UnicodeScalar at least. Relevant (trimmed) quote from the State of String thread:

@Michael_Ilseman

CharacterSet’s builtins model older Unicode semantics than what we probably want to provide. E.g. CharacterSet.lowercaseLetters means general category “Ll”, while modern Unicode defines property Lowercase which includes more scalars. The Lowercase property vs CharacterSet.lowercaseLetters is equivalent to ICU’s u_isULowercase [1] vs u_islower [2], where the former is the modern recommended API.

[1] u_isULowercase: ICU-docs - International Components for Unicode Docs | icu-docs

[2] u_islower: ICU-docs - International Components for Unicode Docs | icu-docs

[3] Regexes | Raku Documentation

Finally, the third level are the String functions such as isLowercase and isUppercase. I don't know if this level is overkill on Character, but off-the-cuff it seems viable. There might be some decent fast-paths in the implementation we can use for common scenarios.

AFAICT, isLowercase would return true for caseless graphemes, such as "7". I don't know what behavior we want to expose, e.g. perhaps a grapheme has to satisfy both Unicode's isCased and isLowercase for our Swift computed property isLowercase.

Or we expose both and let the user sort it out. Really depends on the use.