Pitch: Add bidiClass to Unicode.Scalar.Properties

Evolution Pitches
1

Thanks to Chris Chapman (@cjchapman) for driving this!

gist

Add bidiClass to Unicode.Scalar.Properties

Introduction

Unicode.Scalar.Properties exposes many of the scalar properties defined by the Unicode Standard, but not Bidi_Class, the classification that drives the Unicode Bidirectional Algorithm (UAX #9). This proposal adds a bidiClass property and a Unicode.BidiClass enum for its values.

Motivation

Code that lays out or transforms bidirectional text needs each scalar's Bidi_Class. An example task is deciding whether a piece of text must be wrapped in isolate controls (U+2068…U+2069) before being embedded in surrounding text of the opposite direction, so that it does not reorder its neighbors. That decision requires knowing whether the text contains strong-directional or numeric scalars, which is exactly what Bidi_Class reports. The closely related generalCategory is not a substitute: it does not distinguish a strong left-to-right letter from a strong right-to-left one.

Today this value must come from elsewhere, and the most direct source, the system's ICU library, is the same pain point described in SE-0211. Swift's Foundation ran into this when implementing list-item isolate wrapping in ListFormatStyle.

Proposed solution

We add a bidiClass computed property to Unicode.Scalar.Properties, and a Unicode.BidiClass enum, mirroring the existing generalCategory property and Unicode.GeneralCategory enum:

let scalar: Unicode.Scalar = "\u{05D0}"  // HEBREW LETTER ALEF
scalar.properties.bidiClass  // .rightToLeft

Every scalar has a defined Bidi_Class: scalars not explicitly assigned one, including unassigned code points, take a code-point-based default. So bidiClass returns a value for every Unicode.Scalar.

Detailed design

Unicode.BidiClass has one case per Bidi_Class value, with names derived from the property's long value names, each documented with its standard abbreviation (as Unicode.GeneralCategory does with the two-letter codes):

extension Unicode {

  /// The classification of a scalar used by the Unicode Bidirectional
  /// Algorithm.
  @available(SwiftStdlib 6.5, *)
  public enum BidiClass: Hashable, Sendable {
    case leftToRight            // L
    case rightToLeft            // R
    case arabicLetter           // AL
    case europeanNumber         // EN
    case europeanSeparator      // ES
    case europeanTerminator     // ET
    case arabicNumber           // AN
    case commonSeparator        // CS
    case nonspacingMark         // NSM
    case boundaryNeutral        // BN
    case paragraphSeparator     // B
    case segmentSeparator       // S
    case whitespace             // WS
    case otherNeutral           // ON
    case leftToRightEmbedding   // LRE
    case leftToRightOverride    // LRO
    case rightToLeftEmbedding   // RLE
    case rightToLeftOverride    // RLO
    case popDirectionalFormat   // PDF
    case leftToRightIsolate     // LRI
    case rightToLeftIsolate     // RLI
    case firstStrongIsolate     // FSI
    case popDirectionalIsolate  // PDI
  }
}

extension Unicode.Scalar.Properties {

  /// The bidirectional class of the scalar.
  ///
  /// This property corresponds to the "Bidi_Class" property in the
  /// [Unicode Standard](http://www.unicode.org/versions/latest/).
  @available(SwiftStdlib 6.5, *)
  public var bidiClass: Unicode.BidiClass { get }
}

The values follow the Unicode Character Database's DerivedBidiClass.txt, including its @missing default rules.

Source compatibility

This proposal is purely additive and has no source compatibility impact.

Effect on ABI stability

This proposal is purely an extension of the ABI of the standard library. Unicode.BidiClass is a non-frozen enum (like Unicode.GeneralCategory), so future values can be added without breaking ABI. No raw representation is exposed, leaving the storage free to change.

13 Likes
2

Would there be any utility in providing also the 'strength' of each of these classes (i.e., strong, weak, neutral, explicit) per UAX#9?

3

Is there a binary size cost to this (on non-Apple platforms), or is this already included in the data the stdlib has, or is it not included but will be dead-strippable?

4

Bidi class is the normative Unicode property, and strength is just an informal grouping in the UAX #9 prose that can be derived directly from the class. ICU works this way too: its bidi implementation stores only the class and derives the groupings it needs. Since stdlib API is effectively permanent, we'd rather keep the initial surface minimal and not add categories speculatively. If we find a concrete need later, we can add an API for them without shipping any new data.

5 Likes
5

The bidi class is stored as an inversion list that adds about 5k of data to the Unicode.Scalar.Properties. I think the bidi class data should be dead-strippable in a static-link scenario. In that respect it will behave the same as other Unicode.Scalar.Properties like general category.

6

I think we should make this new BidiClass type a frozen struct that wraps your choice of UInt8/UInt16/UInt32/UInt64 (probably UInt64) and provides computed static variables of each "enum case". The performance of resilient enums is not ideal and by making a frozen struct of UInt64 we can shave off a ton of the performance regressions by being resilient. Unfortunately Unicode.Scalar.Properties is also resilient so by simply accessing scalar.properties you've already hit a performance pitfall, but that doesn't mean we can't make this new type be more performant than its predecessors. Ideally in the future we are able to mark old types as frozen after a particular release. So what I'm imaging is something like the following:

public struct BidiClass {
  public static var leftToRight: BidiClass {
    get
  }

  ... etc
}

extension BidiClass: Equatable {}

// Can maybe add Hashable? Not sure how useful, but doable at least.

Of course leaving a bunch of the performance and ABI attributes out because we don't typically include such things in proposals.

7

Can we just...fix this instead of baking in workarounds?

2 Likes
8

We cannot just fix resilient enums. A client making/getting a value of a resilient type needs to ask the Swift runtime for its metadata (which it potentially has to go allocate itself) to know how large the type is at runtime (because the compiler doesn't know how large the type is at compile time) to make dynamic stack allocations. This is a fundamental property of resilient types and ABI compatibility. With a frozen struct wrapping some UInt64 clients know exactly how large the type is at compile time and can eliminate asking the Swift runtime anything.

9

Clearly. I mean (by adding underscored attributes where needed before necessarily formalizing into a public feature) addressing the underlying problem that we don't have a notion for enums of "frozen in size but not the specific number of cases" and have to essentially open code that with structs. Feels like we're leaving ourselves a known, repeatable performance footgun.

2 Likes