The core Swift libraries aren't all that large. If you just depend on the standard library, it will only add a few megs worth of dependency .so files. It is only once you include Foundation that it does really inflate things, largely because of the chonky FoundationInternationalization (see Android app size and lib_FoundationICU.so) but also because FoundationNetworking has to bundle libcurl and boringssl. That being said, most useful Swift libraries do have a Foundation dependency, so it is something we are actively trying to improve on.
But yes, KMP (and RN, and Flutter) all bring along an entire managed runtime with garbage collection and the works. Contrast with Swift on Android, where it doesn't bring a "runtime" per se: you can call into a Swift function from Android and it will be as efficient as if you had called a C function. There are no GC threads or other overhead that are associated with spinning up an entire managed runtime.
Swift's value as a cross-platform app development language should be judged not just in terms of what it brings to the Android side of your app, but also in terms of what it does not detracted from the iOS side.