As we continue to push forward on completing swift-corelibs-foundation, I would like to propose a structural change to help us meet our goals: move URLSession and related types into a new library.
Some of the feedback we've received from the SwiftNIO team is that Foundation's dependency on libcurl brings in too many other libraries, some of which conflict with the ones that SwiftNIO needs to use. Additionally, we think it might be interesting to explore implementation of Foundation networking on top of SwiftNIO itself.
Therefore, I propose a moving the networking types (except URL and URLComponents) into another library, with a working name of swift-corelibs-foundation-networking. This library could still be part of the same git repository on GitHub or we could create a new one.
This change means that there would be a clear dependency graph for the 3 projects:
SwiftNIO would be able to use core Foundation types. We are separately exploring improvements to types like Data with feedback from the SwiftNIO team.
swift-corelibs-foundation-networking could use SwiftPM to build itself. This is currently not possible for Foundation itself for a variety of dependency-related reasons.
Other projects that wish to maintain a minimum set of dependencies and do not need networking support can reduce their footprint.
Challenges:
A new import statement would be needed for source code which uses URLSession. For example, import Foundation.Networking or import FoundationNetworking. We would need to decide if it is acceptable to deviate from the Darwin platform here, or if we need to introduce some kind of compatibility shim there as well. On Darwin Foundation must continue to include URLSession for source compatibility.
Due to the above, moving URLSession may introduce a source break for Linux clients.
Restructuring the projects will involve work for build scripts and infrastructure changes to CI.
I feel this is dangerous territory. This is going to set a precedent that open source Foundation doesn't have to evolve with Darwin Foundation. That being said, I think it's actually a good thing, I view the current connection as a bit of a burden on advancing Foundation and other "core" libraries. And if this is how we start having more "core" libraries that aren't tied to Darwin Foundation, I think it's a good idea.
I think the name for the new module is a bit weird though. Current convention in Swift is to import the entire module. Something like Foundation.Networking looks like we're entering a new convention where we have submodules (again, I'm not opposed, it's just a new precedent.) I also don't much like the name FoundationNetworking, mostly because of the Foundation bit. It's a bit of a shame we can't use Networking now.
I agree about the deviation not being ideal, but I did want to raise it for discussion regardless.
If we went this way, it may be possible for us to add FoundationNetworking as a module on Darwin as well, but like I said I think it would be empty since we can't break all the source code that currently just imports Foundation only.
We discussed this, but do not have currently the support from the language to create submodules in this way. (Submodule syntax is probably something that would need extensive Evolution discussion first ).
EDIT: To be clear: Foundation.Networking as a name is what I'm referring to here; but we are open to figuring out how to make it happen if we feel strongly that's the way to go.
I can see a major benefit of moving it into its own repo and being built by SwiftPM is that updates could be released more frequently in the same way swiftNIO is updated. However I think that splitting up Foundation on one platform and making it work differently to Darwin is not ideal since its main benefit is in being the same everywhere.
Would it be possible to change URLSession to use SwiftNIO instead of curl inside the current swift-corelibs-foundation and then once this is done see if it possible to make Foundation buildable by both its current method and also SwiftPM. This would hopefully make it possible to do frequent releases for Foundation on Linux in the same way SwiftNIO is released.
If its found that its not possible to build in this way then it could be split out into its own repo.
As to moving from curl to SwiftNIO I think this is a great idea as it also simplifies issues with static linking Foundation as libcurl requires a large number of extra libraries.
I think we should not deviate from the Darwin implementation. What we could do is to have Foundation.Base (better name required) and Foundation.Networking modules and a shim Foundation module that would reexport both of them. This would maintain cross-platform source compatibility while splitting Foundation in sub-modules.
I can't judge if it's worth papering over this specific problem with this kind of workaround ahead of a solution to the general problem here. I hope there is a solution to the general problem soon, though (e.g. submodules).
Thanks Tony for the proposal and all the input received by others.
As Tony points out, SwiftNIO and other server related projects struggle a lot with the fact that Foundation does quite a few dependencies through libcurl which depends on a libssl (usually OpenSSL).
Another issue with using the platform's libcurl is that URLSession in swift-corelibs-foundation does support HTTP/2 if and only if the distribution's libcurl was compiled with HTTP/2 support (which it AFAIK isn't on Ubuntu 14.04 and 16.04).
So from our point of view there's a number of concrete problems with the current approach:
impossible to link another (newer) libcurl/libssl because of 'symbol clash' on Linux
too many dependencies on the system in Foundation (so static binaries with anything that uses Foundation are not possible right now with the standard Swift distributions)
development velocity of URLSession rather slow because of the coupling with the Swift distribution
unclear path to supporting HTTP/2 on all platforms (as the system's libcurl is linked which might or might not be compiled with HTTP/2 support)
Having URLSession in its own module that is distributed through SwiftPM would address most of those issues so I think it is a clear win. Yes, there's a similar issue with the dependency on libxml2 for XMLParser but let's deal with one issue at the time.
Obviously there are drawbacks too: Others have pointed out already that separating URLSession from Foundation comes source compatibility issues but also Data's init(contentsOf url: URL, ...) might be a problem...
@Tony_Parker is there any way we could move Darwin towards (soft) requiring a separate import for URLSession too? Of course we need to stay compatible but maybe we could have a warning issued if URLSession is used without also importing the new URLSession module?
I'd like to ask people to worry less about spelling for now, but Foundation.Networking is probably not a viable option. As noted by Lily, there would be a huge evolution discussion on what this syntax means and what we can do with it; apart from submodules being a loaded term, the closest thing we have today (Clang submodules) wouldn't allow the "FoundationNetworking-depends-on-SwiftNIO-depends-on-Foundation" setup that Tony floated.
It may be infamous but imo the right solution would be to fix the underlying problem in Linux library handling instead of breaking foundation for all macOS/iOS users...
To be clear, if we put URLSession in another module, then we would provide some form of that module for Darwin too so that source code written for iOS/macOS would remain compatible. However:
it would have to be introduced in some future OS update
On Darwin it would likely import nothing, since we can't remove URLSession from import Foundation
Don't think of Swift as an Apple centric language. Swift has vast potential outside of Darwin platforms.
So is the idea that on Darwin import Foundation wouldn't import URLSession in Swift? Because if we have a disconnect between how we get URLSession on Darwin vs anything else, I'm firmly against that.
@Tony_Parker Swift’s Foundation lacks some classes implemented in Darwin’s Foundation (e.g. NSFileCoordination that I think can be implemented with fcntl() btw), thus moving URLSession is reasonable now that Swift’s Foundation did not become stable yet.
Second, Swift’s URLSession lacks some critical functionalities like background session or resuming download tasks. I think no one expects complete compatibility between Swift’s and Darwin’s implementation.
Third, libcurl dependency is an implementation detail. I think reasonable to think about switching to SwiftNIO in future. Having URLSession in a separate framework allows this switch without causing cyclic reference. The only missing part for that switch is FTP support I think, which you are free to reuse my ftp client implementation based on URLSessionStreamTask.
This is a great discussion. Thanks @Tony_Parker and @johannesweiss for clearly describing the rationale, merits and challenges of this proposal.
Just adding my two cents here. I think URLSession will benefit a lot from this separation. I've had a decent amount of experience with URLSession, libcurl and SwiftNIO and I'm inclined towards thinking that a SwiftNIO based implementation of URLSession would be easier to develop/maintain, look better and probably perform better too.
I'd like us to clearly define what classes from Foundation would be a part of the so called FoundationNetworking. Also regarding backward compatibility, if the shim approach does avoid breaking client code on Linux, I'd vote for that. That's the best case. Alternatively, we could start the shim approach and present a clear roadmap about the transition to something like import FoundationNetworking.
I don’t understand what you mean by that. The Darwin (macOS/iOS) implementation of Foundation would be exactly the same as it is now. I mean, its even a completely different codebase.
On Linux and other platforms, for me, the most important thing would be that import Foundation continues to import all of Foundation (base and networking) and if a given module wants only the base Foundation it could import specifically only that part (import FoundationBase / import Foundation.Base).
).
.