Pitch: Move URLSession to new FoundationNetworking module

It might be confusing to have Foundation.Networking and a similarly named Network framework.

2 Likes

Let me give some context first:

At our company we've been developing a pretty large cross-platform Swift code base over the past year which has to compile and run correctly on macOS, iOS, Linux. The goal is to additonally enable Android and Windows support in the future.

One problem that we've been constantly running into is that there are a lot of differences in how POSIX APIs are made available to the Swift side on macOS vs Linux which has forced us to sprinkle a lot of checks of this form into our code base:

#if(Linux)
...
#else
... 
#endif

which is bad because it is very easy for someone exclusively working on the Linux side to accidentally check in code that breaks compilation of the macOS side and vice versa.

So in that light:

A new import statement would be needed for source code which uses URLSession. For example, import Foundation.Networking or import FoundationNetworking.

I think that Foundation.Networking would be the more natural way to spell it and closer to what people generically expect when they are coming from languages like Java or C#.

We would need to decide if it is acceptable to deviate from the Darwin platform here, or if we need to introduce some kind of compatibility shim there as well.

No, deviation would not be acceptable. There are already way too many differences between e.g. Linux and macOS in terms of syntax and semantics (with some really big gotchas) which make writing and maintaining portable Swift code more time consuming, expensive and error prone than it is with a language like Java. The goal should be to reduce those differences - not to add to them.

5 Likes

I think we should not deviate from the Darwin implementation. What we could do is to have Foundation.Base (better name required) and Foundation.Networking modules and a shim Foundation module that would reexport both of them. This would maintain cross-platform source compatibility while splitting Foundation in sub-modules.

7 Likes

I can't judge if it's worth papering over this specific problem with this kind of workaround ahead of a solution to the general problem here. I hope there is a solution to the general problem soon, though (e.g. submodules).

I'm strongly opposed to breaking Darwin compatibility.

Thanks Tony for the proposal and all the input received by others.

As Tony points out, SwiftNIO and other server related projects struggle a lot with the fact that Foundation does quite a few dependencies through libcurl which depends on a libssl (usually OpenSSL).

Another issue with using the platform's libcurl is that URLSession in swift-corelibs-foundation does support HTTP/2 if and only if the distribution's libcurl was compiled with HTTP/2 support (which it AFAIK isn't on Ubuntu 14.04 and 16.04).

So from our point of view there's a number of concrete problems with the current approach:

  • impossible to link another (newer) libcurl/libssl because of 'symbol clash' on Linux
  • too many dependencies on the system in Foundation (so static binaries with anything that uses Foundation are not possible right now with the standard Swift distributions)
  • development velocity of URLSession rather slow because of the coupling with the Swift distribution
  • unclear path to supporting HTTP/2 on all platforms (as the system's libcurl is linked which might or might not be compiled with HTTP/2 support)

Having URLSession in its own module that is distributed through SwiftPM would address most of those issues so I think it is a clear win. Yes, there's a similar issue with the dependency on libxml2 for XMLParser but let's deal with one issue at the time.

Obviously there are drawbacks too: Others have pointed out already that separating URLSession from Foundation comes source compatibility issues but also Data's init(contentsOf url: URL, ...) might be a problem...

@Tony_Parker is there any way we could move Darwin towards (soft) requiring a separate import for URLSession too? Of course we need to stay compatible but maybe we could have a warning issued if URLSession is used without also importing the new URLSession module?

2 Likes

I'd like to ask people to worry less about spelling for now, but Foundation.Networking is probably not a viable option. As noted by Lily, there would be a huge evolution discussion on what this syntax means and what we can do with it; apart from submodules being a loaded term, the closest thing we have today (Clang submodules) wouldn't allow the "FoundationNetworking-depends-on-SwiftNIO-depends-on-Foundation" setup that Tony floated.

1 Like

It may be infamous but imo the right solution would be to fix the underlying problem in Linux library handling instead of breaking foundation for all macOS/iOS users...

To be clear, if we put URLSession in another module, then we would provide some form of that module for Darwin too so that source code written for iOS/macOS would remain compatible. However:

  • it would have to be introduced in some future OS update
  • On Darwin it would likely import nothing, since we can't remove URLSession from import Foundation
1 Like

I'm very strongly against that kind of thinking. I don't want to get Linux dominated in a Apple centric language...

Don't think of Swift as an Apple centric language. Swift has vast potential outside of Darwin platforms.

So is the idea that on Darwin import Foundation wouldn't import URLSession in Swift? Because if we have a disconnect between how we get URLSession on Darwin vs anything else, I'm firmly against that.

@Tony_Parker Swift’s Foundation lacks some classes implemented in Darwin’s Foundation (e.g. NSFileCoordination that I think can be implemented with fcntl() btw), thus moving URLSession is reasonable now that Swift’s Foundation did not become stable yet.

Second, Swift’s URLSession lacks some critical functionalities like background session or resuming download tasks. I think no one expects complete compatibility between Swift’s and Darwin’s implementation. :grinning:

Third, libcurl dependency is an implementation detail. I think reasonable to think about switching to SwiftNIO in future. Having URLSession in a separate framework allows this switch without causing cyclic reference. The only missing part for that switch is FTP support I think, which you are free to reuse my ftp client implementation based on URLSessionStreamTask :heart_eyes:.

Finally I vote for submodule approach. :stuck_out_tongue_winking_eye:

This is a great discussion. Thanks @Tony_Parker and @johannesweiss for clearly describing the rationale, merits and challenges of this proposal.

Just adding my two cents here. I think URLSession will benefit a lot from this separation. I've had a decent amount of experience with URLSession, libcurl and SwiftNIO and I'm inclined towards thinking that a SwiftNIO based implementation of URLSession would be easier to develop/maintain, look better and probably perform better too.

I'd like us to clearly define what classes from Foundation would be a part of the so called FoundationNetworking. Also regarding backward compatibility, if the shim approach does avoid breaking client code on Linux, I'd vote for that. That's the best case. Alternatively, we could start the shim approach and present a clear roadmap about the transition to something like import FoundationNetworking.

2 Likes

Hello? The leading design for Foundation is Apple. You linux guys can't just go and decide to break it for us mac users...

I'm sure the compatibility problem can be and should be worked out.

I don’t understand what you mean by that. The Darwin (macOS/iOS) implementation of Foundation would be exactly the same as it is now. I mean, its even a completely different codebase.

On Linux and other platforms, for me, the most important thing would be that import Foundation continues to import all of Foundation (base and networking) and if a given module wants only the base Foundation it could import specifically only that part (import FoundationBase / import Foundation.Base).

5 Likes

No-one is proposing breaking currently functioning code on Apple platforms.

Some people are proposing that that code may not run unmodified on Linux. That’s a big question that does need to be hashed out, but I promise, “us Linux guys” are not planning to break “you Mac guys”. Especially as we are all also Mac users!

5 Likes

@lukasa @johannesweiss Is the main reason for moving it into its own module / repo so that you can push out more frequent releases of the code? If so, I would like to solve that for all of Foundation on Linux. Right now, if a PR is accepted into Foundation it will go into master having missed the 4.2 release window and might not be released until 5.0, if there is no 4.3 release in between.

If there were frequent releases of Foundation decoupled from the main Swift release, would you still require the separate module?

@spevans two main reasons:

  1. reducing the number of dependencies Foundation has (linking libcurl and libxml2 and then transitively lots and lots of others is a big problem)
  2. increase code iteration velocity for URLSession by making it a SwiftPM package

What you're saying is that increasing the code iteration velocity would benefit all of Foundation. I agree. However, AFAIK (parts of) Foundation need 'private knowledge' (and API) about the implementation of the Swift runtime and the Swift stdlib which leads to the tight coupling between them. As I understand it would currently not be possible to use a newer/older version of Foundation with a given Swift compiler/stdlib and I believe this will remain true even after ABI stability. But that is just what I remember of the top off my head, @Tony_Parker and @millenomi are the people who actually know.

1 Like

Thanks for starting this discussion - I think if we get this right it has the potential to improve things a lot.

I am in favour of exploring reimplementing URLSession on top of SwiftNIO, to remove the dependency on curl.

I also like the idea of building Foundation using SwiftPM and using that to decouple Foundation releases from Swift itself.

I am concerned about the suggestion of making import Foundation not include URLSession on Linux, though. Some bits of Darwin Foundation are not present on Linux and never will be (for good reasons), but having to do import FoundationNetworking on Linux does cause the Darwin and Linux APIs to diverge even further, in a source-breaking way, as others have commented.

Could the layering look like this instead?

swift-corelibs-foundation
    |
SwiftNIO
    |
swift-corelibs-foundation-core

Then, SwiftNIO could import FoundationCore so it can start using core types like Data without curl getting pulled in. Users who import Foundation would get everything, like today, except the implementation of URLSession could use SwiftNIO for its implementation (the SwiftNIO API would not be exposed out of Foundation).

As Tony mentioned that it might be possible to add FoundationNetworking as a dummy module on Darwin, perhaps it might be possible to add FoundationCore on Darwin, so Darwin users can choose between Foundation and FoundationCore as well.

3 Likes
Terms of Service

Privacy Policy

Cookie Policy