FoundationEssentials on Darwin?

If your Linux or Windows app imports the Foundation library from the Swift toolchain today, you get all of these improvements for free. And if your app is particularly sensitive to binary size, you can now import the FoundationEssentials library, which provides a more targeted subset of Foundation's features that omits internationalization and localization data.

I'm developing a package for both macOS and Windows that is sensitive to binary size, so it sounds like import FoundationEssentials is exatly what I want.

As FoundationEssentials isn't available on macOS, should I explicitly import the swift-foundation package into my project, or should I always check what platform I'm using when importing Foundation?

#if canImport(Darwin)
import Foundation
#else
import FoundationEssentials
import FoundationInternationalization
#endif

It would be rather nice if import FoundationEssentials just worked on Darwin too, the way it does on Windows/Linux.

8 Likes

TL;DR: don’t worry, the Foundation shouldn’t impact your binary size on macOS.

The #if canImport(Darwin) is the solution I’d go with. Yeah, it is cumbersome :dotted_line_face:

On Apple platforms the foundation library is shipped with the OS and is dynamically linked on launch. Unless you are shipping your own version of stdlib, by default, Foundation is not included into your binary.

FoundationEssentials is more tailored for when you are targeting linux, with usually statically linked binary.

If you do want to statically link stdlib on Darwin, then yes, swift-foundation is there. You might have good reasons to. For e.g. if you are targeting an older OS release, but are relying on newer foundation additions.

P.S. you can’t even really make static binaries for Darwin. If you are to make system calls, you are required to dynamically link libSystem.

6 Likes

Thanks. Yeah – I'm not worried about binary size on macOS, but I am worried about binary size on other platforms (Linux, Windows) – that's why it seems like FoundationEssentials is the way to go.

It seems confusing that FoundationEssentials isn't included in macOS to support coding cross platform packages in Xcode. Right now I have to switch to Linux/Windows to confirm that the code I'm writing compiles correctly (because certain symbols are available in Foundation on macOS, but not in FoundationEssentials).

Including FoundationEssentials on Darwin too would make this workflow much more elegant.

4 Likes

I had a discussion with some folks at the conference about this - there are good technical reasons why it is the way it currently is but I agree it's confusing for people trying to write cross-platform code.

There is a quick fix which would allow you to write import FoundationEssentials on Darwin platforms without the need for the #if canImport(Darwin) dance but that would give you the whole of the Foundation API when importing. Great to start with, but problematic when you pull in an API that's not available on Linux and it compiles on macOS but fails when you build it. I do not think this is the right solution.

Marking which APIs are available in FoundationEssentials is a harder task without affecting a lot of other stuff but I'd love to see this eventually solved properly.

cc @Tony_Parker

2 Likes

this is pretty closely related to the cross-platform Foundation story, i’ll also cross-link GRDB 7 beta - #4 by taylorswift . the fix also requires a lot of #if canImport(Darwin) conditionals, so it would be great if this stuff just worked by default.

The pattern that I use is:

#if canImport(FoundationEssentials)
import FoundationEssentials
#else
import Foundation
#endif

This makes this less about Darwin and more about Foundation on the different platforms.

10 Likes

Yeah this still presents the problem that you can end up using an API that's not available on FoundatoinEssentials and not see until it hits Linux (as well as the whole perception thing of "it's not the same on all platforms" blah blah blah). Long term I'd love to see import FoundationEssentials work on Darwin and expose the correct APIs

11 Likes

If this happens we need a new way to detect when we're being built against the open source Foundation. Alamofire uses FoundationNetworking to detect this case because URLSession is still hugely incomplete on those platforms. I don't know what the plan is to bring it into the open, but we need a way to detect when that happens and to differentiate it from older versions.

1 Like

As an example of the noise not having import FoundationEssentials not working on all platforms - the noise it makes in PRs when adopting - e.g. JWTKit 5, FoundationEssentials-only + more by MahdiBM · Pull Request #253 · vapor/penny-bot · GitHub

1 Like

Won't #if canImport(Darwin) work?

I'd like to bump this as it's becoming apparent that dealing with the mismatch is becoming painful. Aside from polluting the code with the #if canImport(FoundationEssentials) dance It's hard to know what's available in FoundationEssentials and what's not. There's no documentation for the different modules and involves you digging through GitHub or trying to decipher the re-exports from CoreFoundation and then hoping you get it right as it goes through CI. This makes developing cross platform libraries on macOS much harder than it should be.

@Tony_Parker I know when we spoke there was a lot of technical issues that would need to be solved for import FoundationEssentials to work on macOS. Is it possible to split Foundation up on these platforms so they're bundled as individual OS libraries and then have Foundation that just re-exports all the components? (I know this is a large undertaking)

14 Likes

Thanks @0xTim - I understand how this can be a challenge. I still don't see any short term answer to this, but we will keep looking for some kind of solution.

3 Likes

We (@Tony_Parker @icharleshu and @jmschonfeld, please correct if I got anything wrong or missed any thing) have been talking about this, but unfortunately we've not been able to come up with a nice solution yet. Here are some ideas we've come up so far.

My understanding is that we want that, ultimately, one should be able to write this without platform guard

import FoundationEssentials

and the same code should builds on all platforms.

Option 1: Introduce a FoundationEssentials module on Darwin for parity.

This would mean splitting Foundation.framework into FoundationEssentials, FoundationInternationalization, and another module for other Darwin-only code that will never make into Linux. For the sake of argument, let's call it FoundationDarwinSupport.

This is challenging because while we want FoundationEssentials to be "the same" on Linux and on Darwin, that's not the current status, and I'm not sure if it's ever possible. Take FileManager as an example. It is an Obj-C imported type, a subclass of NSObject on Darwin, but not on Linux.

Now, what do we do with FileManager?

Option 1.1: It remains available in FoundationEssentials, but we modify it to look exactly like how it is on Linux (so, not a NSObject)

But for Darwin compatibility, it has to be NSObject when needed. Ideally we'd conditionally declare it as a NSObject if FoundationDarwinSupport is linked, or extend the type so it becomes a NSObject subclass in FoundationDarwinSupport. Unfortunately neither is possible now.

Option 1.2: Introduce a FileManager2, reimplemented from bottom up and sharing the same source on two platforms.

This will indeed achieve what we want. However, it also means we'll have two types serving very similar functionality. This strategy doesn't scale well to all types we've been building with either.

Option 2: Annotation support

It's hard to know what's available in FoundationEssentials and what's not.

It's another point that @0xTim mentioned that, if addressed, would greatly improve the experience.

But what can we do now? For example, if we could add @available(linux) to APIs that are available on Linux, and some compiler flag to specify what deployment targets to match against, you'd be able to get compile time check when building on MacOS.

Or, if there's a way to typealias/define your own import that effectively does this

#def EssentialsIfAvailable 
#if canImport(FoundationEssentials)
import FoundationEssentials
#else
import Foundation
#endif
#end_def

// in your code
import EssentialsIfAvailable

But again, I'm not sure if it's possible to do something like this yet.

Short summary

While we haven't been able to reach any solution yet, we're definitely going to continue thinking about this. I'd like to hear if anyone has other ideas too.

2 Likes

If I understand it correctly, Option 1.1 is the “cleanest” option but some language support is needed? Well, if this language support does not “pollute” the language with anything too stupid, I would say go for it, you have done so much other magic to the language, why not. :+1:

even if it were possible to make them “the same”, wouldn’t this be ABI breaking?

Yeah, ABI compatibility is definitely a prerequisite.

I imagine in this world, those who import Foundation will get all
FoundationEssentials , FoundationInternationalization, and the presumed FoundationDarwinSupport. And there should not be any ABI breaking implications.

1 Like

Making FileManager not be a subclass of NSObject might have been possible during the original Great Renaming of Foundation types, when these APIs were being settled on for Swift, but it is far too late now.

Having a type be or not be a subclass of NSObject depending on how it’s imported in a particular file would be astonishing and unprecedented. This is simply not a gap that can be closed in the cross-platform story. But it is also as narrowly scoped as possible to the underlying discrepancy between platforms with respect to Obj-C.

That doesn’t mean that the gaps that can be closed, such as making FoundationEssentials importable on all platforms and making that import bring in as close to the same set of APIs as possible, couldn’t or shouldn’t be done, I think.

I don’t think this is the right formulation: it is impossible by construction given ABI stability and other guarantees we would want to maintain for the types deemed “essential.” It demotivates any incremental progress towards achievable parity, since any fraction of an infinitely unachievable goal is infinitesimal.

Rather, more narrowly, we want that code which is buildable in some way on all platforms not to require one import incantation on one platform and a different one on another.

When it comes to NSObject subclassing, (IIUC) that’s never going to be buildable on Linux no matter what—we’re not going to have an Obj-C runtime to interop with—so it is out of scope of this narrower parity question.

(Since we special-case a lot of this interop anyway, it might be worth having the compiler have diagnostics which automatically treat any NSObject-related use of Foundation types as unavailable on Linux, as though there were some sort of platform availability guard—but to me this is a diagnostics QoL issue that can be separately addressed orthogonal to the Foundation/FoundationEssentials issue.)

1 Like

We should probably enumerate specific ABI-compatibility issues that would be considered blocking, because there are tools to make this easier, and it's not the first Apple framework that has been split and had definitions moved from one to another.

For example, in the OS versions associated with Xcode 16's SDKs, SwiftUI was split into SwiftUI and SwiftUICore, with a large number of types and other declarations moving from the former to the latter. This was achieved with underscored attributes like @_originallyDefinedIn, which IIUC ensures that the linker emits the right symbols depending on the minimum deployment version being targeted. Similar techniques could presumably be used here. (Unfortunately lld.macho still needs to catch up to supporting this, I think...)

That being said, since Foundation is at such a low-level of the tech stack, it's possible that even with those techniques that work with newer frameworks like SwiftUI, there are possibly other issues from folks who are used to using runtime tricks to accomplish things from the Objective-C world.

1 Like

To be clear - the request is the opposite of this. To be able to write code that builds all platforms with the minimal imports required (i.e. FoundationEssentials when only using types in there) and have those imports import the same APIs on all platforms.

So if I write:

import FoundationEssentials

func processDate(_ date: Date) { ... }

That works and compiles fine. Whereas if I write

import FoundationEssentials

func shellOut() {
  let process = Process()
  // ...
}

I get a compiler error on the platform I'm building on, even if I'm targeting a different platform at the end of the day.

This still wouldn't solve the issue of not knowing what APIs are available in FoundationEssentials when building on macOS

This makes the most sense to me. Split up Foundation on Darwin and have Foundation export Essentials, Networking, Internationalisation etc. This provides the best platform parity and best experience for Swift developers no matter the platform they're on.

With regards to types like FileManager I think pulling in the new FileManager when compiling for non-Darwin platforms and pulling in the legacy version when compiling for Darwin platforms is an acceptable trade-off.

This seem incredible to me that the long-term goal (talking years here) is not to completely align the codebases on all platforms. I think option 1.2 with a new name is probably the eventual way to go for this.

Isn't this the entire goal of swift-foundation? To share the same source on all platforms, one codebase to maintain and one API that's the same on all platforms?

6 Likes

This is what happens when I write and edit on the phone: that sentence I wrote is missing a "not" in there... I'll go back and edit.

My point being, taking as given that this is not achievable, then the achievable thing is to have writing import FoundationEssentials import a superset of those APIs, but nothing in that superset should be available on other platforms using some other import (i.e., the superset should contain only unavoidable platform-specific additions).

2 Likes