String for linguistic processing

[Michael_Ilseman]

@silberz

From your feedback, I understand that it would be possible to get a “composition” operation that follows the NFKC standard to get an array of unicode scalars for a given string. This would allow one to build a robust syntactic parser that could parse texts in any natural language. Ligatures would not be a linguistic problem because NFKC properly separates them into sequences of letters; from a linguistic point of view, emojis should be processed as words rather than letters.

@silberz, as far as whether you can rely on composition for your needs, you would know your domain best. Note that Unicode has several exceptions to the rules (because of course it does). Are you familiar with composition exclusions?

For example, this might affect your support for Tibetan, which had a rocky history in early Unicode where it was added, removed, then re-added. But, I'm not familiar with the details here.

@silberz

I understand that a standard “decomposition” operation such as NFKD could be available as well

ICU provides support for NFC, NFKC, NFKD, NFD, and FCC. So the question for Swift is what is the most "Swifty" way to expose these? Since these are expert-oriented (highly domain-specific), it seems similar to the Unicode Scalar Properties pitch.

@silberz

Another missing feature that is desperately needed: a data <=> string reversibility access that would allow a parser to tell its client at what exact position in the initial data (byte array) the match actually occurred. Without this basic feature, I don’t see how to build serious NLP applications.

Yes, this is a fallout of String not having native UTF-8 storage, where we cannot provide a mapping between a String.Index and a byte offset due to internally transcoding to UTF-16. This is a use case that's increasing in frequency and popping up in different domains (e.g. SwiftSyntax is dealing with this now).

Until then you'll have to maintain a mapping. @Xi_Ge or @harlanhaskins might be able to describe how they're doing so. While annoying, hopefully it's just a Dictionary<String.Index, Int> whose keys are from whatever view you're doing your processing in (likely a scalar view).

@silberz

don’t know how to “start a pitch” to get these three features; are “pitches” Apple official procedure?

Nope, this is Swift procedure: https://github.com/apple/swift-evolution/blob/master/process.md. For example, @allevato started the Unicode Scalar Properties pitch.