How to disable implicit Foundation imports?

I greatly appreciate that you've moved so quickly to improve this situation @xwu, thank you!

2 Likes

this is not at all an unproductive direction, in fact i personally think it is a very productive thing to talk about (at least as a library author) and the only reason it is not talked about more is because someone still has to write the smaller, more modular frameworks to replace Foundation, and we do not always have time to take a 3 week detour to write a pure swift library so more often than not we just go down the Foundation route because it is the path of least resistance.

the issues with Foundation are well known and if anything i think the things you have mentioned are only touching on the tip of the iceberg.

again i do not think anyone disagrees with what you are saying but it is a very big task and “i hate Foundation” is necessary but not sufficient. in particular many of us pure swift library authors are not good at cocoapods or XCode so writing libraries that don’t import Foundation but only compile on amazon linux and only compile on this week’s nightly toolchain is not a step forward it is really just more of the same if we are being honest.

if we want to make progress it is important i think to focus less on “we need more foundationless libraries” because there actually are a lot more than people think but they don’t get adoption because going from linux to macOS/iOS/watchOS/tvOS is just as hard as the other way around and the ecosystem is still fragmented just in a different way.

I would appreciate a mod moving what is essentially a new thread here (starting with my post above) actually into a new thread. I think it could be quite productive to discuss potential solutions.

For me, I'd be happy if I could just import and link, say, the Foundation implementation of JSONDecoder without the kitchen sink coming along for the ride. In the JavaScript world the solution to that is known as tree shaking but, in both languages, this is easier said than done with automated tooling alone. As evidenced in this thread, that doesn't seem to work at all for Wasm, and I also haven't had any luck doing so on Android – i.e. it doesn't work where we need it the most.

I personally would be happy with a solution that modularises small parts of Foundation and reduces interdependencies between those modules; maybe in the future these could be moved out of the library entirely. The difficulty I see is that Foundation seems to be deliberately written in such a way that increases interdependencies between its parts – philosophically it appears to want to be "the platform", and as a codebase appears to implicitly reward dogfooding from within.

In the immediate-term my main concern is about that philosophy itself – Foundation being "the platform" probably makes sense on Darwin because the library is shipped with the OS and has years (decades?) of battle testing, so embracing it there is essentially low cost and all win. The issue is when that mentality leaks, for example via SwiftPM, into platforms (arguably all non-Darwin platforms?) where the opposite is true: the cost of linking Foundation is high, and the implementation itself is incomplete and often incompletely tested (-> buggy).

3 Likes

"At the beginning people complained that Foundation doesn't exist on non Apple platforms, now when it is finally there people doing their best to avoid it..." :sweat_smile:

Considering static linking only: It looks like you would be happy if strip did its job better - and it doesn't (*) - perhaps then this is the strip whom to blame. This is not Foundation specific - the same could be said about any large library that explodes the app binary size even if you use a single function in the library. Splitting things into micro packages to avoid app size issue works but feels like a roundabout / workaround solution.

(*) or maybe strip does work and without it the app size would be much larger? :thinking:

PS. I understand that you'd prefer 3MB to 26MB app, it does fell wrong and I am fully with you on that. However is this just that, i.e. "it feels wrong" or do you have some real practical issue with that, like you are shipping your app to your customers over 3G connection and it takes ages? To be honest I don't even know how many MBs is the app we are shipping, won't be surprised (or care much) if it is 5MB or 50MB.

7 times out of 10 i am installing apps when i am cellular data or spotty wifi or spotty wifi that displaces cellular data because of iOS’s fallback and if it takes too long i give up on the download and my friends are the same. and if it is a website and not an app store app i am even more likely to bounce.

so i totally understand why this is an existential problem for app developers.

2 Likes

Nobody ever said this. We wanted Swift native versions of some of the functionality in Foundation, and for it to be open source. Apple's eventual answer was swift-corelibs-foundation and... well... here we are.

4 Likes

Yes, I would say the vast majority of internet users don't have an Ethernet cable running to their device (especially if it's a phone or a tablet). You don't have to use 3G to experience this, just try downloading something at a hotel, a coffee shop, an airport, or at some residential area with heavy WiFi congestion. And I'm only speaking of 1st world countries that have a privilege of having something better than 3G available.

1 Like

I'm talking about a web app, which has much more ethereal "download" characteristics (i.e. it's not downloaded and then "done"), and is updated more often than a typical mobile app. A typical mobile app may well be ~50MB, but a typical web app tends to be much smaller than that.

Also, users don't make as much of a conscious choice to download a web app – there is no popup asking them if they want to download it –, so it doesn't feel good at an ethical level to regularly spam them with 10s of MBs of downloads that may or may not be cached.

Also, we (essentially have no choice but to) mirror our app in China, which comes tied with endless amounts of performance considerations. The connection between China and the rest of the world is tenuous at best, and people's internet connections within China are also not always as good as they are elsewhere.

I could continue, but in short, it matters a lot :slight_smile:

2 Likes

Interestingly to see people commenting upon the easy "why 3MB is better than 26MB" point yet nobody picked up upon why strip doesn't seem to be doing what it's supposed to do to keep the app size still around "3MB". Is it the case that "String.trimmingCharacters" indirectly touches the whole/most of the Foundation (perhaps unicode tables (1))? Or is strip inherently incapable / unreliable?

(1) and if it is indeed due to inclusion of unicode tables perhaps Foundation could have some alternative that won't explode the app size? bike shedding:

"string".nonUnicodeAwareTrimmingCharacters(in: .nonUnicodeAwareWhitespaces)
Somewhat relevant tangent.

All good reasons above why we need to keep app binaries small. Yet we (the industry as a whole and Apple in particular) have got ourselves in a situation when we increased all binaries by some 30-70% (my estimate, please correct if you have a better figure) by switching all apps to 64 bits, even if 99.99% (another estimate of mine) of apps don't utilise the benefits of being 64 bits. Isn't that a mistake?

2 Likes

As for Unicode tables, I've mentioned this before in other threads: after stdlib switched from ICU to its own tables, the situation became worse in terms of binary size for statically-linked apps. This is caused by Foundation not reusing the tables from stdlib and still relying on ICU. Thus statically-linked apps bundle two separate Unicode tables, one instance from stdlib, the other one from ICU.

5 Likes

I guess it is not yet capable, there's e.g.:

3 Likes

in your opinion, what are a few of the biggest obstacles preventing you from adopting pure-swift alternatives like swift-json, swift-png, or swift-url?

Ignoring those specific libraries, I really don't know. My guess is that Apple's leadership is unwilling to spend resources on such open source work and swift-corelibs-foundation was a political compromise within Apple that technically allowed Apple engineers to develop in the open because it was still controlled by Apple and could be used in the backend environments, even if it wasn't ideal for the language. This bootstrapped some open source functionality into Swift but also took energy away from native replacements, which is compounded by the lack of investment in Foundation from Apple more recently.

On the plus side it seems likely that the existence of swift-corelibs-foundation played some part in the recent open evolution proposals for URL and other Foundation improvements Apple is shipping this year.

But this is all speculation, which many really don't like here, so I'll leave it at that.

I don't think we should even be questioning OP's concerns about binary size - OP isn't using Foundation, so they shouldn't have to pay for it.

Whether OP's concerns apply to anybody else's projects, and how good/bad internet connections around the world may be, is not relevant to the original question.

7 Likes

this isn’t really helpful for those of us who make pure swift libraries because we are not part of Apple and not part of the corelibs project. so we can’t really make better libraries until we know what we specifically are doing wrong and not what Apple is doing wrong.

I've been trying to knock out the dependencies my Swift programs have on corelibs-foundation recently. The issue I've run into with using these foundation-less replacements is that a number of core dependencies already import Foundation. So then what's the point of depending on one of these libraries when I already have to "pay" for Foundation & everyone already knows Foundation? (the specific libraries I've come up against are Soto & async-http-client)

For me that's definitely the major issue. I do think that replacements for major currency types like Date and maybe Data (though some Collection<UInt8> is pretty good for that) would really benefit from inclusion in the standard library just because they become so much more useful when everyone can agree on the same type. Just like Result used to be.

Specific feedback on swift-json

This library looks really awesome and if I ever do manage to banish Foundation from my server components, it will probably be what I use for JSON decoding.

Encoding is obviously an important feature to make it useful for replacing Foundation and I see you have that on your roadmap, which is great.

The other thing I think would really help (just my 2 cents!) would be a very simple entry point into the package. It's a small thing, but just decoding a Decodable type (where I think most Swift programmers would want to start with a package like this) requires seeing the following types:

  • JSON
  • Grammar
  • JSON.Rule
  • JSON.Rule.Root

If I were the author of this package, I would add a convenience API for what I think is the most popular use:

extension JSON {
    static func decode<Data: Collection, Decoded: Decodable>(_ data: Data, as: Decoded.Type) throws -> Decoded { ... }
}

So then your getting started example could be:

import JSON
struct MyType: Codable { var x: Int }

let input = #"{ "x": 1 }"#
let object = try JSON.decode(input.utf8, as: MyType.self)

Then the section after that could get into more advanced things like annotating error messages and fancy grammar things. Basically I think when the community & tools have such a heavy bias towards Foundation you have to make it as simple as possible to use an alternative.

The WebURL getting started section is extremely focused by comparison, even though that library has tons of complex functionality:

import WebURL

var url = WebURL("https://github.com/karwa/swift-url")!
url.scheme   // "https"
url.hostname // "github.com"
url.path     // "/karwa/swift-url"

url.pathComponents.removeLast(2)
// "https://github.com/"

url.pathComponents += ["apple", "swift"]
// "https://github.com/apple/swift"
9 Likes

thank you! i’ve opened #47 to track this.

2 Likes

I would like to see a new checker tool that can list all the SDK-provided libraries. This is critical for cross-platform developers, e.g. they can check easily if it’s easy to support a new platform/

2 Likes

AHC having a Foundation dependency is something we'd like to address going forward. We should absolutely be able to use AHC without requiring Foundation, and the dependency was only added for a few specific API entry points that in principle we ought to be able to haul out.

Unfortunately, those API entry points are public API, so we'd need to mint a new major version to make this change. That's always a bit disruptive. With that said, AHC hasn't had a new major version for three years, so we probably have some breathing room to make that change. However, in a new major version we'll probably want to substantially revamp the older, Future-based API to be a bit less sprawling and a bit more considered. So there's some work to do there.

3 Likes

Perhaps worth mentioning - the WebURL port of async-http-client already supports building without any Foundation dependency.

By default, it builds with Foundation compatibility APIs, so you can continue to make requests using Foundation.URL and they will be converted to WebURL internally. The goal of those compatibility APIs is to demonstrate that projects such as AHC, Soto, and others, can adopt WebURL while retaining compatibility with clients using Foundation URL (of course, that requires a 1.0 release with a stable API).

The only API-breaking change is that HTTPClient.Request.url returns a WebURL - because I thought it would be sad if the WebURL port didn't return a WebURL :sweat_smile: - you can convert it to a Foundation URL if you like, and there's an .nsURL computed property which does it for you.

But if you have a package which uses the WebURL port of AHC, you can also build it using the following command:

NO_FOUNDATION_COMPAT=1 swift build -Xswiftc -DWEBURL_IDNA_USE_STDLIB_NFC

In that case, the compatibility APIs will be left out, and WebURL will use a standard library SPI for Unicode string normalisation (currently it has to get that from Foundation). I think it's accepted that the stdlib should expose this via a proper public API at some point; it is needed for basic operations such as String comparison anyway, so this is a temporary dependency.

One of the advantages of having a pure Swift implementation is that our Unicode processing (for Unicode domain names/IDNA) can use those same data tables and algorithms as the stdlib. String normalisation via the standard library is also 30-40% faster than doing it via Foundation.

As for binary size - I wrote a simple CLI program to make a request to a URL and stream the output to stdout. These are the sizes of the resulting binary, building in release mode with --static-swift-stdlib ("Regular" means using the official AHC repo):

Variant Size Difference
Regular (with Foundation) 77MB
WebURL 44MB -44%
Regular (with Foundation), stripped 48MB
WebURL, stripped 14MB -70%

And there are no missing features; it's everything AHC supports. It also includes IDNA everywhere (part of the URL standard) and it improves support for many URLs and redirects which Foundation does not process correctly.

Those binary sizes could probably go lower still; dead code (and metadata) elimination is a known issue in Swift, as has been mentioned above. Some of WebURL's interfaces might also be overly-generic or there may be other gains to be had; file a GitHub issue or send a PR if anything jumps out at you. It's also something I do keep an eye on and try to improve.

Anyway - I don't want to divert the thread. I just thought this was a relevant data point.

11 Likes