The biggest gain you can make right now is to factor out the ICU data into a single minimal data file that is loaded at runtime. That's because the way ICU has always been built for Swift is to hard-code that data into the library itself. You can see this in the way it was packaged before Swift 6:
> du -sk swift-5.10.1-RELEASE-fedora39/usr/lib/swift/linux/libicu*1
28008 swift-5.10.1-RELEASE-fedora39/usr/lib/swift/linux/libicudataswift.so.69.1
4148 swift-5.10.1-RELEASE-fedora39/usr/lib/swift/linux/libicui18nswift.so.69.1
2428 swift-5.10.1-RELEASE-fedora39/usr/lib/swift/linux/libicuucswift.so.69.1
> readelf -sW swift-5.10.1-RELEASE-fedora39/usr/lib/swift/linux/libicudataswift.so.69.1
Symbol table '.dynsym' contains 2 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000001000 0x1b562d0 OBJECT GLOBAL DEFAULT 4 icudt_swift69_dat
Symbol table '.symtab' contains 3 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000001b58f10 0 OBJECT LOCAL DEFAULT 6 _DYNAMIC
2: 0000000000001000 0x1b562d0 OBJECT GLOBAL DEFAULT 4 icudt_swift69_dat
Most of the size was from libicudata, which appears to just be a single large array of data, icudt_swift69_dat.
Swift 6 brought ICU in-house into a swiftlang package with its own manifest, so it is extremely easy to tinker with now. 
It may have initially supported loading its data from a file, but that was switched over to the old model of hard-coding it all into the single large ICU library, which Marc highlighted last week.
I suggest that you and Marc look into reverting that and have Foundation load the data at runtime instead, so the four shared libraries for each Android architecture can share a single icudata file, instead of shipping multiple large shared libraries that duplicate the same data for each architecture.
As for the further code slimming you mention, they will pale in comparison to such data slimming.