# Adding Unicode properties to UnicodeScalar/Character **URL:** \ **Category:** Pitches\ **Created:** [January 30, 2018, 4:32am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310 "2018-01-30T04:32:26Z")\ **Posts on this page:** 15\ **Page:** 3
**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\ **Post date:** [March 21, 2018, 9:29pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/41 "2018-03-21T21:29:24Z")
> [@Michael\_Ilseman](#): > > For script and block, which are pretty large String enums, it’s a tradeoff against stdlib binary size. Depending on the impact, I’d saw we could keep or defer them for later. Those two were probably the most concerning ones in terms of size, in addition to General\_Category (which I've added to my branch). > [@Michael\_Ilseman](#): > > What other enums were you considering? I was thinking that some like `GraphemeClusterBreakType` might be useful, but since it looks like stdlib uses a grapheme-cluster-based break iterator to determine grapheme boundaries, I'm not sure whether it would be used internally or not. If disk space were no object, my personal goal would be to expose Swift-friendly APIs for all of ICU 🙂. Of course, if all of this is going into `libswiftCore`, then we need to be fairly selective about what we think users will actually want. Throwing this out there: do you think anyone would object to having a new standard library specifically for advanced extensions to `Unicode.Scalar`, `Character`, and `String`, along with some new APIs? It could be distributed with the toolchain but as a separate library from `libswiftCore`, so only those who need it will link it in. The advantage here vs. doing it as a third-party library is that we still resolve much of the pain of linking to ICU for users, on both platforms: Apple OSes, where you have to link your own copy statically or get an App Store rejection; and Linux, where renaming makes using ClangImporter problematic with the system ICU libs. If that's a viable path, we should consider it now before adding a lot of APIs to `libswiftCore`, so that we don't end up with a strange delineation of those APIs. ---
**Author:** ![Michael\_Ilseman](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/michael_ilseman/32/44640_2.png) [@Michael\_Ilseman](https://forums.swift.org/u/Michael_Ilseman)\ **Post date:** [March 21, 2018, 10:00pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/42 "2018-03-21T22:00:44Z")
> [@allevato](#): > > I was thinking that some like GraphemeClusterBreakType might be useful, but since it looks like stdlib uses a grapheme-cluster-based break iterator to determine grapheme boundaries, I’m not sure whether it would be used internally or not. Since the grapheme breaking rules themselves change from version-to-version, we can't have an implementation based on these properties. > [@allevato](#): > > Throwing this out there: do you think anyone would object to having a new standard library specifically for advanced extensions to Unicode.Scalar, Character, and String, along with some new APIs? It could be distributed with the toolchain but as a separate library from libswiftCore, so only those who need it will link it in. The advantage here vs. doing it as a third-party library is that we still resolve much of the pain of linking to ICU for users, on both platforms: Apple OSes, where you have to link your own copy statically or get an App Store rejection; and Linux, where renaming makes using ClangImporter problematic with the system ICU libs. This has been discussed in many different flavors (e.g. [Large Proposal: Non-Standard Libraries - #42 by Michael\_Ilseman](https://forums.swift.org/t/large-proposal-non-standard-libraries/6969/42)), but in general I would expect such a thing to happen first as a package. As far as the Linux ICU pain, we could just ship a modern ICU on Linux with Swift rather than trying to rely on the OS's. Linux has the reverse situation where Swift code is typically newer than the OS's libraries. > [@allevato](#): > > If that’s a viable path, we should consider it now before adding a lot of APIs to libswiftCore, so that we don’t end up with a strange delineation of those APIs. If by all of ICU you also mean the CLDR, then rest assured that there will be _plenty_ of unexposed well-delineated surface area ;-) I think UnicodeScalar properties within the stdlib is sensible. ---
**Author:** ![DeFrenZ](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/defrenz/32/827_2.png) [@DeFrenZ](https://forums.swift.org/u/DeFrenZ)\ **Post date:** [March 22, 2018, 10:00am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/43 "2018-03-22T10:00:12Z")
> [@allevato](#): > > If disk space were no object, my personal goal would be to expose Swift-friendly APIs for all of ICU 🙂. Doesn't this make it fit more as an official but separate library? Something like `SwiftICU` or something like that... This is a very good to have, but not that common to be used. Might this be a good example for the various "separate libraries" discussion? ---
**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\ **Post date:** [March 22, 2018, 2:54pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/44 "2018-03-22T14:54:58Z")
Yes, I agree—the rest of my post above after statement that discusses doing just that. 🙂 But in the interest of not feature creeping this pitch/proposal thread, I'll stick with the basic Unicode properties on scalars for now. ---
**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\ **Post date:** [March 29, 2018, 5:33am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/45 "2018-03-29T05:33:20Z")
I now have the [prototype implementation](https://github.com/apple/swift/pull/15593) and [proposal write-up](https://github.com/apple/swift-evolution/pull/820) posted as pull requests. Anyone who's interested, please take a look! ---
**Author:** ![hartbit](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/hartbit/32/738_2.png) [@hartbit](https://forums.swift.org/u/hartbit)\ **Post date:** [March 29, 2018, 6:29am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/46 "2018-03-29T06:29:22Z")
I don’t know much about Unicode but I’ll give my two cents: I really like this proposal. It’s well designed and described. I only have two concerns. 1. I would prefer the alternative of a `hasProperty` function as I find it as discoverable but I find that is reads better: ```swift // This reads as asking if the scalar properties is alphabetic scalar.properties.isAlphabetic // I find the following clearer: scalar.hasProperty(.alphabetic) ``` Whatever ends up being chosen, I agree that we should not bloat the Scalar API directly. 1. Do we want the Properties struct (or enum if we go with my suggestion) on Scalar? When we look at Character, won’t we reuse it? In that case, shouldn’t it live directly in the Unicode “namespace”? ---
**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\ **Post date:** [March 29, 2018, 6:36am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/47 "2018-03-29T06:36:22Z")
Thanks for the feedback! > [@hartbit](#): > > I would prefer the alternative of a hasProperty function as I find it as discoverable but I find that is reads better: If we went that direction, how would you recommend handling the non-Boolean properties? Just surface them on `Unicode.Scalar` directly? One advantage to the `Properties` struct is that there's a single location where all of the properties defined by the Unicode Standard live, whether they're Boolean, enums, strings, numbers, etc. > [@hartbit](#): > > When we look at Character, won’t we reuse it? The `Properties` struct wouldn't be reusable verbatim for `Character`, since many of the properties defined on `Unicode.Scalar` only apply to single scalars, not to grapheme clusters. For example, `isGraphemeBase` and `isGraphemeExtend` only make sense for a single scalar. ---
**Author:** ![hartbit](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/hartbit/32/738_2.png) [@hartbit](https://forums.swift.org/u/hartbit)\ **Post date:** [March 29, 2018, 6:42am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/48 "2018-03-29T06:42:56Z")
> [@allevato](#): > > If we went that direction, how would you recommend handling the non-Boolean properties? Just surface them on Unicode.Scalar directly? As you say in the proposal, the case mappings would be quite useful to have on Scalar directly. For the other properties, perhaps we could prefix them with Unicode to group them together: unicodeAge unicodeGeneralCategory ... > [@allevato](#): > > One advantage to the Properties struct is that there’s a single location where all of the properties defined by the Unicode Standard live, whether they’re Boolean, enums, strings, numbers, etc. That makes sense, that’s why I don’t strongly disagree with the proposal’s direction. I just feel like some of those properties feel so useful that I would have liked to have them more accessible on Scalar directly instead of always going through properties. But perhaps I’m overthinking this. ---
**Author:** ![ahti](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ahti/32/604_2.png) [@ahti](https://forums.swift.org/u/ahti)\ **Post date:** [March 29, 2018, 9:13am UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/49 "2018-03-29T09:13:40Z")
I’m not sure this is the proper way to go here, but it would be possible to have a `scalar.property(.uppercase)` style method with a return type depending on the property. See this gist for an example of this: [generic\_property.swift · GitHub](https://gist.github.com/ahti/26e28a3df00b5f1e536939459c16afc5) One quirk is that the property members need to be computed properties instead of stored, since those are not supported in generic types, and those don’t seem to be picked up by auto complete right now (at least in Swift Playgrounds for iPad). ---
**Author:** ![Michael\_Ilseman](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/michael_ilseman/32/44640_2.png) [@Michael\_Ilseman](https://forums.swift.org/u/Michael_Ilseman)\ **Post date:** [March 29, 2018, 2:49pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/50 "2018-03-29T14:49:09Z")
> [@allevato](#): > > The Properties struct wouldn’t be reusable verbatim for Character, since many of the properties defined on Unicode.Scalar only apply to single scalars, not to grapheme clusters. For example, isGraphemeBase and isGraphemeExtend only make sense for a single scalar. > [@hartbit](#): > > That makes sense, that’s why I don’t strongly disagree with the proposal’s direction. I just feel like some of those properties feel so useful that I would have liked to have them more accessible on Scalar directly instead of always going through properties. But perhaps I’m overthinking this. It's... a bit more complicated here. Unicode defines properties on scalars but almost nothing on graphemes. Any queries on `Character` would be Swift inventing semantics not part of the Unicode standard. I'm working on a pitch right now to expose some of these queries, but the semantics need careful consideration and need to be more "Swifty" than "Unicody". That is, their naming and usage in common Swift code should adhere to what something really means rather than whether a bit was set in the UCD. These scalar properties have very specific semantics, and are helpful for many expert uses, but surfacing them in a "non-expert" namespace gives a false impression of linguistic correctness. For example, the "isEmoji" property on scalars is set for ASCII numbers, which _might_ be part of an emoji sequence somewhere. Trying to "fix" the name is hard, as their semantics are often nothing more than "is the bit set in the UCD". I think @xwu is right that their names as terms-of-art should reflect Unicode connotations rather than common Swift usage connotations (hence the "expert namespace" of Properties and highly technical documentation): > [@xwu](#): > > Brief comment: for properties, especially the “is\*” ones, it’d be nice to hew closer to the Unicode names—easier to find for those who are experienced in these details, and not really any less clear for those that aren’t, since no name will fully explain. These are definitely all terms of art. ---
**Author:** ![bzamayo](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/bzamayo/32/764_2.png) [@bzamayo](https://forums.swift.org/u/bzamayo)\ **Post date:** [March 30, 2018, 9:18pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/51 "2018-03-30T21:18:43Z")
As someone who is unlikely to ever need to dive to this level of scalar characteristics, I like how these accessors are all neatly contained into the `properties` and do not pollute the namespace of `Unicode.Scalar` itself. ---
**Author:** ![Chris\_Lattner3](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/chris_lattner3/32/348_2.png) [@Chris\_Lattner3](https://forums.swift.org/u/Chris_Lattner3)\ **Post date:** [April 1, 2018, 3:57pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/52 "2018-04-01T15:57:46Z")
Random question, but doesn't swift core already depend on all of ICU? Or would the proposal add an additional dependency? ---
**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\ **Post date:** [April 1, 2018, 5:33pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/53 "2018-04-01T17:33:34Z")
It does, so this proposal doesn’t add new dependencies but instead makes more features from ICU available to Swift users through stdlib without the pain of linking separately to ICU. ---
**Author:** ![AliSoftware](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/alisoftware/32/2764_2.png) [@AliSoftware](https://forums.swift.org/u/AliSoftware)\ **Post date:** [April 24, 2018, 9:18pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/54 "2018-04-24T21:18:34Z")
Love this idea. Just my two cents, I'd make the `Unicode.GeneralCategory` enum `RawRepresentable` (`enum Unicode.GeneralCategory: String`) so that we can access the two-letter codes if needed This could especially be useful for readability and clarity when creating regular expressions referencing those character categories, using `"\\p{\(Unicode.GeneralCategory.lowercaseLetter)}"` for example. ---
**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\ **Post date:** [April 25, 2018, 2:04pm UTC](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310/55 "2018-04-25T14:04:08Z")
Thanks! I agree that this would be an excellent addition to the API; I've called it out in the proposal review thread. [Previous page](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310.md?page=2)