I think we might be coupling two issues unnecessarily.
Can I import FoundationEssentials on all platforms (and implicitly avoid the cost of a full Foundation on platforms that have meaningful separation, like Linux)? This is what many cross-platform package authors care about.
Is Foundation identical in all observable ways on all platforms? It's not today, and I don't think making it identical everywhere blocks us from solving (1).
So focusing only on (1), and putting (2) aside, as that's not a new concern, what would be the Foundation team's reservations here, @itingliu? Or could we get a fix for (1) here in the short term, and meaningfully improve the experience for authors of packages who don't write platform-specific code?
Since the newer Swift versions, on all platforms, the new Foundation implementation is used. (Differences in the behavior might still occur when using Swift.)
Can we be more specific about what "Can I import FoundationEssentials on all platforms" is asking? Do you mean
import FoundationEssentials gets expanded to this
if FoundationEssentials != Foundation {
// platforms where there is a meaningful separation
import FoundationEssentials
} else {
// Darwin
import Foundation
}
Or
2. import FoundationEssentials means import FoundationEssentials with no other types available even on Darwin?
I think a lot of discussion so far prefers (2). Suppose so, then we need to figure out what FoundationEssentials includes -- Do we really mean the source code of FoundationEssentials should be exactly the same on the two platforms, or are we ok to accept that they differ in non-meaningful way? (But again, what counts as non-meaningful)? In the end we inevitably have to address your second point right after the first point.
I'd disagree with this - it saves a few lines of code but doesn't solve the underlying problem of code compiling fine on macOS but failing as soon as you compile on Linux - this is a real pain point that would be masked but hacking in a patch like this and make the experience even worse
100% option 2. Option 1 makes for an awful cross platform experience. What types in FoundationEssentials are different on different platforms? The API is exactly the same correct?
I wholeheartedly agree with the sentiment expressed by Tim. If we had working documentation facilities that clearly indicate which APIs are present on which platforms, in which modules, and how their behavior differs between platforms, then maybe option 1 could be palatable within some limited timeframe (i.e. some transition period).
Without documentation explicitly reflecting any of these difference, it just leads to endless confusion not only for beginners, but also experienced developers.
No, as discussed above, the API is different and cannot be made the same—like, possibly ever:
Even now, we're actively considering adding APIs that will have platform-specific differences (though granted I haven't looked into whether it's part of the "essentials," I'm not sure why only one part of Foundation would be held to a stricter cross-platform standard here):
Right, they may have some implementation differences, but FileManager exists on both platforms, as does all of it's functions?
I mean there's an argument to be made that we could propose that if you import FoundationEssentials you get the same code on all platforms, including the same implementation of FileManager. It would be a great opportunity to 'break' the API to align them
I had not realised that had platform differences - I have strong thoughts on that but will add them to the review
Of course this is not to be taken in a strict sense, the operations have to be mapped to the actual OS implementations on the different platforms.
Because of this mapping, there might be bugs specfic to certain platforms or differences in behaviour. Of course you want to have a behaviour that is similar enough on all platforms so the common API works as expected.
The question then is, should there be API parts that only work on certain platforms and if there is a good mechanism to handle this. But I think this was not a question here. UPDATE: Oh yes, this in indeed the case for the new Subprocess stuff on Windows. As noted there, you have to you use e.g. #if os(Windows) … on Windows to use those parts. I think this could be OK here if those parts are only additional parts specifying the behaviour on Windows in a more precise way and your code could work without them.
I do not care about that much about the actual implementation parity. I enjoy being able to look into the source code but if that’s not possible for all types on all platforms I can live with it.
What Swift needs, to be considered truly cross-platform, is that code, that builds on one platform, builds on all other platforms as well. (Not looking at all the Darwin-only API's like SwiftUI, CoreData, …).
Given, there are bugs which are platform specific but every language has this issue, you cannot avoid it.
But I should be able to commit code to Git; developed, successfully built and tested on my Mac, and my colleague on Linux (or Windows for that matter) should be able to pull the code, build it successfully and run the tests successfully. And the CI server needs to be able to do the same.
I would actually prefer to get warnings like This code is only effective on Windows when it’s not wrapped in a #if os(windows) and compile anyways just with the methods implementation empty on all platforms. Like this, I can make use of the compiler while developing these parts as well on a "foreign" platform.
Just to clarify this point here, this is not strictly the case. FoundationEssentials on non-Darwin does not define NSObject (and we don't plan for it to as we feel NSObject is correctly layered above in Foundation from swift-corelibs-foundation). However on Darwin since FileManager is (and must continue to be for compatibility) imported from Objective-C, it must always be an NSObject subclass. Even if we lowered FileManager to a FoundationEssentials module on Darwin, we could not do so in a compatible manner without also lowering the NSObject parent class relationship. This means that the functionality that NSObject brings (ex. func isEqual(to:)) will be a discrepancy between FoundationEssentials on Darwin (where it's available) and FoundationEssentials on Linux (where it cannot be). The same goes for other types like ProcessInfo and I suspect there are other similar APIs that might fall into this bucket in practice, so this isn't strictly an isolated issue (although it is limited).
Now perhaps we find that this is an "acceptable" discrepancy and doesn't actually harm the full goal of ensuring that major APIs and usages are consistent from one platform to another. After all, the implementation of all of the main functions of FileManager do have shared sources and should behave the same (our unit tests run the same on both platforms). However my caution around this point is that I don't feel that we can be at a point where FoundationEssentials is "exactly" the same even just on the level of which functions are available, so we likely need to find what an acceptable level of difference is (I just don't quite know where that line might fall at the moment).
Stupid question - can we vend 2 versions of FileManager? So if you import Foundation you get the legacy Obj-C version and if you import FoundationEssentials you get the new one? If not, why not?
Second - can we break the API in a new version at some point (let's say Swift 7)? I'm aware that it would require significant work (such as on Apple platforms etc), but from a technical point of view there's nothing stopping it right?
Are you asking this from a technical perspective, or a practical perspective? Since FoundationEssentials doesn't currently exist on Darwin, it's technically possible to introduce new module called FoundationEssentials which is entirely unrelated to Foundation (so you get different types depending on what you import) — but at a practical level, this just shifts the platform problem elsewhere: now import FoundationEssentials would give you the same type on both Darwin and Linux, but then import Foundation would behave differently on Linux vs. Darwin.
(Besides the fact that it would be incredibly confusing that on the same platform, import Foundation and import FoundationEssentials would get you two types with the same name but slightly different and unrelated APIs, given that FoundationEssentials is supposed to be a subset of Foundation)
Again, from a technical perspective, there's nothing preventing API (or ABI breaks) — but the massive amounts of downstream code that are liable to break mean this is a non-starter. There's an extremely high bar for API breaks (and an almost insurmountable bar for ABI breaks) which require the API to be actively harmful; I don't believe there's been an intentional API break since API stability in the Swift 3 timeframe.
IMO the trade-off of having to namespace the imports if you import both FoundationEssentials.FileManager and Foundation.FileManager and the slight differences in APIs would be fine (and provide a migration path to move off the Obj-C code).
Unrelated side bar about dependency measurements
Do we have any actual measurements? Because sure the LOC would be massive, but if, say FileManager.isEqual(to:) instances were tiny it makes the thought of migrating it less horrifying.
However I think that's probably a distraction and the current question of "how much difference in APIs is acceptable" should be discussed as that's mainly the crux of the problem. I think having all FileManager APIs being available, but missing APIs from the NSObject inheritance etc is a fine trade off.
Unfortunately, I think the tradeoff goes well beyond just namespacing the imports:
The types need to be documented separately
API surfaces which include these types need to potentially be doubled to handle both of them
On Darwin platforms, all types participate in Obj-C interop, so it'll always be valid to pass in a FoundationEssentials type to Obj-C; it'd be a huge footgun if these types don't "just work" the way you might expect, or crash
I personally don't think this is the right tradeoff to make for the benefit of having FoundationEssentials work similarly across Darwin and Linux.
I would agree that it makes sense to try to keep the API surface in sync as much as possible (barring backwards-compatibility concerns), modulo platform differences. Linux never has, and will never have, Obj-C interop, so it'd make sense to me for FileManager to look identical in both places short of NSObject inheritance.
Like @jmschonfeld says, there is still API surface that differs between platforms beyond NSObject (both intentionally and unintentionally), but the goal (as far as I know) is still to get as close as possible.
Currently you can sort of do this already if your project includes FoundationEssentials as a package dependency, and do not do import Foundation anywhere.
Needless to say, it breaks soon when the downstream projects import Foundation in their code. They'd see two different FileManagers, one from toolchain Foundation, and the other one from FoundationEssentials. Like @itaiferber mentioned, these two FileManagers are entirely unrelated. There is also no way to convert from one to the other, and the API just collapses pretty soon. (Perhaps you could unsafeBitCast in a not-so-unsafe way?)
Improving documentation is reasonable and useful in general. We could start by having a list of available types in swift-foundation's readme file and noting platform specific behaviors.
Do you know of any projects that offer nice documentation that swift-foundation can learn from, specifically highlighting platform differences? Or do you have any suggestions for making helpful documentation?
The README.md file is only discoverable by contributors to swift-foundation. I suspect that vast majority of its users not looking for the GitHub repo wouldn't know this file even exists.
We already provide availability annotations for Darwin platforms and IIRC display those when rendered with DocC. I hope this could be extended to all platforms, not only Darwin.