No-one is proposing breaking currently functioning code on Apple platforms.
Some people are proposing that that code may not run unmodified on Linux. That’s a big question that does need to be hashed out, but I promise, “us Linux guys” are not planning to break “you Mac guys”. Especially as we are all also Mac users!
@lukasa @johannesweiss Is the main reason for moving it into its own module / repo so that you can push out more frequent releases of the code? If so, I would like to solve that for all of Foundation on Linux. Right now, if a PR is accepted into Foundation it will go into master having missed the 4.2 release window and might not be released until 5.0, if there is no 4.3 release in between.
If there were frequent releases of Foundation decoupled from the main Swift release, would you still require the separate module?
@spevans two main reasons:
libcurl and libxml2 and then transitively lots and lots of others is a big problem)What you're saying is that increasing the code iteration velocity would benefit all of Foundation. I agree. However, AFAIK (parts of) Foundation need 'private knowledge' (and API) about the implementation of the Swift runtime and the Swift stdlib which leads to the tight coupling between them. As I understand it would currently not be possible to use a newer/older version of Foundation with a given Swift compiler/stdlib and I believe this will remain true even after ABI stability. But that is just what I remember of the top off my head, @Tony_Parker and @millenomi are the people who actually know.
Thanks for starting this discussion - I think if we get this right it has the potential to improve things a lot.
I am in favour of exploring reimplementing URLSession on top of SwiftNIO, to remove the dependency on curl.
I also like the idea of building Foundation using SwiftPM and using that to decouple Foundation releases from Swift itself.
I am concerned about the suggestion of making import Foundation not include URLSession on Linux, though. Some bits of Darwin Foundation are not present on Linux and never will be (for good reasons), but having to do import FoundationNetworking on Linux does cause the Darwin and Linux APIs to diverge even further, in a source-breaking way, as others have commented.
Could the layering look like this instead?
swift-corelibs-foundation
|
SwiftNIO
|
swift-corelibs-foundation-core
Then, SwiftNIO could import FoundationCore so it can start using core types like Data without curl getting pulled in. Users who import Foundation would get everything, like today, except the implementation of URLSession could use SwiftNIO for its implementation (the SwiftNIO API would not be exposed out of Foundation).
As Tony mentioned that it might be possible to add FoundationNetworking as a dummy module on Darwin, perhaps it might be possible to add FoundationCore on Darwin, so Darwin users can choose between Foundation and FoundationCore as well.
While this is an interesting idea, I want to make sure everyone remembers that while the SwiftNIO API would not be exposed, the SwiftNIO ABI likely would be unless Foundation mangled its internal copy of SwiftNIO.
I mention this only to say that any proposal that involves trying to have swift-corelibs-foundation depend on SwiftNIO needs to be contingent on swift-corelibs-foundation becoming a SwiftPM package: otherwise, all we've done is change the offending library from libcurl to SwiftNIO. While that's an improvement (SwiftNIO definitely links fewer things than libcurl does!), it's not really addressing the core of the problem.
The subset of things that we would move to a new Networking module would be chosen such that it would not require specific Swift runtime SPI. Foundation proper remains revlocked with Swift.
An alternative I was amenable to exploring is the following:
We move the types to FoundationNetworking. Eg: FoundationNetworking.URLSession.
Foundation maintains all its current types (e.g. Foundation.URLSession).
Their implementation dynamically finds a linked FoundationNetworking and forwards to the corresponding type. So a Foundation.URLSession would own a FoundationNetworking.URLSession and forward to it. If it can’t, it’d trap with a message detailing what to do (add FN to your dependencies).
This preserves functionality and source compatibility at the cost of additional work by Linux authors (as they now have to link FN). I am looking to variants of this as well; a solution that works at compile time would be superior to this.
It seems a bit weird to make Foundation differ between Linux and darwin. How would it also affect other platforms (Windows namely)?
What about dropping libcurl dependencies and replace it with swiftNIO code related to this part?
Thanks Lily that makes sense! My understanding was that @spevans would like to have all of Foundation on Linux to be developed as a SwiftPM package so it could be released more frequently. I tried to explain why it won't be possible to have all of Foundation just be a a SwiftPM package. The URLSession part of Foundation however could be as you pointed out, especially if we leave the actual types in Foundation proper.
Is there a way that we can move the SPI and revlocked parts of Foundation into a -core library that is built as swift-corelibs-foundation is today with the current restrictions, and all other parts that don't have these restrictions (so not just URLSession) would stay in an SwiftPM buildable Foundation. ie as @IanPartridge was suggesting?
Especially with SwiftPM still not really useable in a sane way from Xcode.
While this is an interesting idea, we have to note there would be multiple side-effects. For example, simply using String(contentsOf:) or Data(contentsOf:) on online resources would trap the process.
What about making SwiftNIO a Swift core library? Then we could establish the following dependency tree:
swift-corelibs-foundation (Foundation)
|
swift-corelibs-foundation-networkio (Foundation.NetworkIO aka SwiftNIO)
|
swift-corelibs-foundation-base (Foundation.Base)
The main disadvantage would be that SwiftNIO would have to be distributed with Swift instead of a Swift Package.
I'm very happy for Foundation to modularise a bit, and integrating with SwiftNIO seems like it would benefit both projects and the Swift language. On the other hand, I think source compatibility across platforms is very important; I don't like one platform writing one thing and other platforms write another thing.
We do have a kind of compile-time solution: we could add a fix-it which writes an import FoundationNetworking statement and make it part of the Swift 5 migrator in Xcode. Darwin Swift projects are almost always built with Xcode (not always, but almost always), and it has a mechanism for stuff like this.
Another idea is that we could expose a compatibility typealias through higher-level Darwin-specific frameworks like AppKit/UIKit. A fix-it might be a sufficient fall-back solution for code which uses URLSession but does not import one of those frameworks.
I'm liking this idea of turning Foundation into a kind of umbrella module, with Foundation as the base set of functionality, plus FoundationNetworking (URLSession) and maybe FoundationXML too (to split libxml dependencies, which are also problematic for some). We could leave all of this in one repository, too.
@Tony_Parker sounds great! I'm just unsure how the dependencies would work out then. Because in that world, FoundationNetworking and FoundationXML would be SwiftPM projects but you could only import Foundation if you depend on both of them already, right?
I think we could use SwiftPM to build, but they would be distributed with Swift the same way they are today.
In other words, the problem of distributing more frequently is orthogonal to the problem of minimizing dependencies for clients.
Ok, so you're saying we could ship compiled versions of Foundation* with Swift as we do today but they could be updated through SwiftPM.
So let's imagine the following scenario: A hypothetical Swift version 42.0.0 ships with FoundationNetworking 23.0.0 which depends on and ships SwiftNIO 128.0.0. Now we're trying to compile the hypothetical web server Steam 256.0.0 which depends on SwiftNIO from: "128.5.6" using swift build. SwiftPM would now realise that SwiftNIO 128.0.0 as shipped with Swift isn't good enough so it would update SwiftNIO (that means download and compile) to say 128.7.0, then recompile FoundationNetworking and everything that depends on it with SwiftNIO version 128.7.0.
Is that accurate?
So, by the first and second Foundation you mean two different modules, the umbrella and the base one? 
Oh, sorry. I had something there but messed up the markdown.
FoundationTBD as the base module - a different name than the overall Foundation module, which would contain all the same functionality it does today to preserve source compatibility.
Great! Also, if we finally embed SwiftNIO with the Swift/Foundation distribution would it be available to import from other modules or would it be only for internal use (with a mangled module name to avoid naming collisions, I suppose)?