What's in the 34 MB of ICU data in lib_FoundationICU.so, and what can be removed

Platform Android
,
1

lib_FoundationICU.so is the largest library in a Swift app on Android: 40.0 MB of the 65.7 MB in a Foundation "Hello World" built with the Swift 6.4.1 SDK (arm64, stripped like Gradle does). Almost all of it is ICU data. Following up on the proposal to filter that data, I measured which parts swift-foundation and swift-corelibs-foundation actually use, and what breaks when each part is removed.

Short version

  • About 1 MB is unused by both libraries and can go for everyone.
  • Another 6–10 MB per ABI is used only by a few corelibs features, so it can only go as an opt-in.
  • Break iterator rules and unit data look unused (nothing calls ubrk_open()), but removing them breaks real APIs and can crash.

What the 34 MB is

The packaged data is ICU 76, 33.98 MB, 4,855 entries:

Data Size Share
Locale and formatting data (coll, zone, curr, lang, region, rbnf, locales) 19.66 MB 58%
Converters (197) 5.67 MB 17%
Break iterator dictionaries (CJK, Thai, Khmer, Lao, Burmese) 3.00 MB 9%
Units 2.96 MB 9%
Transliteration + character names 1.45 MB 4%
Line breaking rules, StringPrep, confusables 0.94 MB 3%
Break iterator rules (word, sentence, title) 0.15 MB <1%

Who uses which data

swift-corelibs-foundation links the same _FoundationICU and needs more of it than swift-foundation does. I checked both: by reading the ICU calls, and by removing each part and diffing the output of ICU-backed APIs.

Data swift-foundation swift-corelibs-foundation Verdict
Locale and formatting data date, number, list formatting NSString compare, DateFormatter, NumberFormatter keep
Units Duration unit formatting, ByteCountFormatStyle Duration unit formatting keep
Break iterator rules capitalized(with:), capitalization contexts, relative dates NSRegularExpression with .useUnicodeWordBoundaries keep
20 converters String.Encoding, 11 encodings part of availableStringEncodings keep
177 other converters — availableStringEncodings (all CF encodings) opt-in
Transliteration + character names — applyingTransform opt-in
Break iterator dictionaries — word-boundary regex on CJK and Southeast Asian text opt-in
Line breaking rules, StringPrep, confusables — — remove

Looks unused, but isn't

  • Break iterator rules. ICU opens break iterators internally for title-casing, capitalization contexts and word-boundary regex. Without them, capitalized(with:) gets Turkish İ and Dutch IJ wrong, capitalization contexts are ignored, word-boundary regex finds no matches even in English, and Date.RelativeFormatStyle with a capitalization context traps on a force unwrap.
  • Units. Without them, Duration prints 1.0 measure-unit/duration-hour and ByteCountFormatStyle prints 1.234567.

Three filters, tested

I rebuilt the data with each filter and ran swift-foundation's tests on macOS (1,754), swift-corelibs-foundation's tests on Linux (1,906), and a probe that dumps the output of ICU-backed APIs (encodings, case mapping, date, number, Duration and byte-count formatting, capitalization contexts, relative dates, transforms, regex, IDNA).

Filter Removes Saves swift-foundation corelibs
safe line breaking rules, StringPrep, confusables 0.94 MB same output same output
high StringPrep, confusables, 177 converters, transliteration, character names 6.69 MB same output 22 of 70 encodings left, applyingTransform returns nil
all high + line breaking rules + break iterator dictionaries 10.38 MB same output as high, plus no word segmentation for CJK and Southeast Asian scripts in regex

Every test in both suites passes with all three filters. The corelibs tests don't cover encodings beyond the common ones, transforms or CJK word boundaries, which is why only the probe shows the breakage.

On Android

A minimal NativeActivity, Swift 6.4.1 SDK, NDK r30, arm64, all libraries stripped with llvm-strip --strip-unneeded as the Android Gradle Plugin does by default. lib_FoundationICU.so rebuilt from swift-foundation-icu (same sources as swift-6.4.1-RELEASE) with each filter, then run on an Android 16 emulator:

Build APK Output
Foundation, SDK as shipped 65.7 MB reference
Foundation, ICU rebuilt, full data 64.3 MB same
safe 63.4 MB same
high 57.7 MB applyingTransform returns nil
all 54.0 MB as high, plus CJK word-boundary regex

Where the rest of the APK goes

What each import costs (same app, stripped, arm64):

App imports APK Added
stdlib only 11.2 MB
FoundationEssentials 17.7 MB +6.5 MB
Foundation 65.7 MB +48.0 MB
Foundation + FoundationNetworking 69.2 MB +3.5 MB

In the 65.7 MB Foundation APK: lib_FoundationICU.so 40.0 MB (34.0 MB data, ~4 MB code), libswiftCore.so 6.9, libFoundationEssentials.so 6.2, libFoundation.so 6.1, libFoundationInternationalization.so 1.7, libc++_shared.so 1.4, Regex 1.6, the rest ~1.5.

The biggest part of the ICU data is per-language: 16.4 MB across 292 languages (median 20 KB, English 220 KB, Chinese 1.7 MB, mostly collation). Locale data for 10 common languages (en, es, de, fr, ru, ja, zh, pt, it, ko), including units, is about 3.8 MB, so per-app locale subsetting would save about 15 MB, more than any filter above.

Unstripped, the same APK is 95.6 MB: 30 MB of symbol tables and debug info, 8 MB of it in the NDK's libc++_shared.so. Gradle strips these by default, but hand-rolled packaging may not.

What I'd propose

  1. Apply the safe filter in ICUExport, so it is reapplied whenever the data is regenerated for a new ICU version. Small, but free.
  2. Add an opt-in profile for apps that don't need legacy encodings, transliteration or CJK word-boundary regex: a CMake option (the Android SDK is built with CMake) and a SwiftPM trait. That's −6.7 to −10.4 MB per ABI.

The bigger lever, per-language locale data, is out of scope for these filters. The work on moving formatting data into Swift with per-locale subsetting (swift-foundation #1991 and #2330) looks like the long-term answer for that.

The Android side of this is tracked in swift-foundation-icu#95. Would a PR for (1) be welcome? And for (2), would SDK vendors ship a slim variant if it existed? I'm happy to share the scripts and probes.

3 Likes
2

My two cents would be: Free is certainly good even if small. But as to (2), from a high level standpoint, are we solving anyone's (very real) issue with Foundation code size by going from 63 MB to 58 MB? Enough to design out another 'flavor' of Foundation which then probably would need its own long-term maintenance? Particularly if there's long-term work for per-locale subsetting which looks set to make a much larger impact? I think your analysis has convinced me that the work already ongoing takes the right approach.

3

@xwu Thanks for reading through it and for the honest take.

Fair point on (2). Saving 5 MB out of 63 doesn’t justify a second configuration to maintain. I’ll drop it and send a PR for the safe filter only.

Moving formatting into Swift looks like the better path: the data can be limited per locale, as the --locale option in swiftlang/swift-foundation#1991 and swiftlang/swift-foundation#2330 already allows. Where I can help there: run the Swift implementations against ICU on Linux and Android with the flags on, across all locales, and report differences, and measure what each step saves in an Android APK.

One thing I couldn’t find an answer to: what’s the long-term picture for swift-foundation-icu? In the earlier thread, Charles described building the ICU data as part of the SwiftPM build, so apps could thin it down to the locales they ship. With formatting moving into Swift, is that still the plan, or is the direction for ICU to become optional, needed only for things like collation or transliteration?

I’m asking because corelibs calls ICU directly (string encodings, CFStringTransform, NSRegularExpression), so on Android import Foundation will keep pulling in lib_FoundationICU.so even after swift-foundation stops needing it.

1 Like
4

Android already has ICU as part of the system, and ICU4C is part of Android's NDK. How feasible is it to let Swift Foundation make use of that, and would it help with reducing the app bundle size?

2 Likes
5

Foundation's implementation assumes that it's using specific versions of the ICU APIs and data tables, which also include a number of extensions/patches that Apple has added on top. My understanding is that the changes needed to make it work against arbitrary versions of whatever Android ships wouldn't be straightforward.

4 Likes
6

I also considered this previously, but I think we can do better than this. Last time I checked (two months ago), Android SDK doesn't expose every ICU functionality Foundation uses. To go with this approach, we will need to guard Foundation's API availability according to the underlying ICU functions.

This leads to a lot of questions that we haven't quite figured out: For example, what does this mean for our promise to provide the same behavior across platforms? How do we structure our code to check for ICU functionalities and versions everywhere inside our implementation? We also want to allow other platforms to benefit from data reduction, so we're looking for a more generic approach.

4 Likes
7

I think that it is confusing because you are treating SCF and SF as separate entities. In reality they are one. The long term roadmap is for corelibs foundation to be entirely replaced with a pure Swift implementation (swift-foundation). In the meantime, Foundation uses corelibs-foundation as an internal implementation detail. When the interface is rewritten, it is removed from corelibs-foundation. It will likely be that the dependencies on ICU shuffle towards swift-foundation. While some things may be able to move away as things evolve, it isn’t guaranteed.

That said, this has come up in the Windows workgroup where it seems that there was someone exploring the ICU data representation and believed that they should be able to generically shrink it significantly.

8

Thanks, that clears it up. I was treating corelibs as a separate consumer of ICU, but if its interfaces move into swift-foundation as they get rewritten, the ICU dependency moves with them, and data reduction is really a swift-foundation question.

Do you know where the ICU data representation work from the Windows workgroup was discussed? I'd be glad to help there, for example by measuring the effect on Android APKs, or by testing a smaller data file against Foundation on Android and Linux.

Also, a follow-up to my earlier reply: after @xwu's point, I didn't send the safe filter PR after all. Without working around ICU's dependency checks it only removes about 0.25 MB, which isn't worth a change to ICUExport. While looking at what else pulls in the full Foundation on Android, I found that packages declaring a privacy manifest link it when cross-compiling from a Mac, and opened swiftlang/swift-package-manager#10694 for that.

9

It was discussed in the windows workgroup meeting, but I don't have anything to share on it.

1 Like