Pitch: Move URLSession to new FoundationNetworking module

In other words, the problem of distributing more frequently is orthogonal to the problem of minimizing dependencies for clients.

1 Like

Ok, so you're saying we could ship compiled versions of Foundation* with Swift as we do today but they could be updated through SwiftPM.

So let's imagine the following scenario: A hypothetical Swift version 42.0.0 ships with FoundationNetworking 23.0.0 which depends on and ships SwiftNIO 128.0.0. Now we're trying to compile the hypothetical web server Steam 256.0.0 which depends on SwiftNIO from: "128.5.6" using swift build. SwiftPM would now realise that SwiftNIO 128.0.0 as shipped with Swift isn't good enough so it would update SwiftNIO (that means download and compile) to say 128.7.0, then recompile FoundationNetworking and everything that depends on it with SwiftNIO version 128.7.0.

Is that accurate?

So, by the first and second Foundation you mean two different modules, the umbrella and the base one? :thinking:

Oh, sorry. I had something there but messed up the markdown.

FoundationTBD as the base module - a different name than the overall Foundation module, which would contain all the same functionality it does today to preserve source compatibility.

3 Likes

Great! Also, if we finally embed SwiftNIO with the Swift/Foundation distribution would it be available to import from other modules or would it be only for internal use (with a mangled module name to avoid naming collisions, I suppose)?

I think I see what you're getting at here -- that you don't want to commit to ABI/API for SwiftNIO yet, so you want anything that depends on that to ship outside of Swift.

I'm not sure if we're there yet. Perhaps that is a blocker to adopting SwiftNIO inside URLSession.

Ok, let me be more precise on what my current position is (not 100% worked out as there are a few moving parts):

  • I'd argue that ABI does not matter on Linux because SwiftPM doesn't support binary dependencies anyway. And Swift isn't part of any Linux distribution so shipping binaries without the libraries is not a thing (at least right now).
  • We don't ever want to be fully API locked but we're happy to follow SemVer (we want to document what we consider public API more precisely soon anyway). For example if we said 'no new public types in SwiftNIO' without a new major version, we could probably live with that (my personal opinion, would need to talk to the rest of the team).

So I think we could get to a working system if we shipped (in the Swift distribution) a version of NIO and FoundationNetworking which would be used if you just run swiftc myfile.swift. If you use SwiftPM and your package depends on a version of SwiftNIO that isn't satisfied by the one that shipped with the Swift distribution, then SwiftPM would need to be able to update SwiftNIO and therefore also download & compile FoundationNetworking from github.com. It would be very important for us (and our users) that SwiftNIO can be updated if needed.

Incidentally, this model is exactly what Haskell does: GHC (the most important Haskell compiler) ships with a certain set of libraries in certain versions. All of those except for a package called base can be updated through the package manager. This feels quite similar actually: There's a small number (in Haskell's case 1 (base), in Swift's case maybe 3 (stdlib, FoundationCore/Base/TBD, Dispatch)) of packages that cannot be updated through the package manager but others can. This would require some work to SwiftPM though.

Does that sound workable for you or would you only consider a fully API & ABI stable NIO?

1 Like

There is some new precedent there if we ship something with the base distribution that's upgradable via SwiftPM. I'm not saying it's wrong, but it's a direction we probably should consider as part of a broader decision about how Swift is distributed on Linux.

I think we can still consider this smaller piece independently of that larger question, given that we don't depend on SwiftNIO (yet).

@Tony_Parker Indeed, it would set some new precedent but I'd argue that would be a great win for Foundation on Linux. If we had the mechanisms in place to update the parts of swift-corelibs-foundation through SwiftPM, the development could move much faster. It would also be a much more welcoming OSS project as you can release often und not when Apple decides to release a new Xcode version.

@spevans already expressed the feeling that this would be great for all parts of Foundation that do not need to be rev-locked with Swift and its runtime. I agree with that. URLSession could (because if its dependencies) just be the trailblazer into an arguably better model.

3 Likes

Thanks everyone for your feedback. I think we'll investigate the idea of Foundation as an umbrella module of some kind, and report back once we have made some progress there.

4 Likes

Sounds good! I really think we should investigate how we can have more parts of the Swift distribution (parts of Foundation, SwiftPM, ...) upgradeable through SwiftPM.
Should that not be something people think is useful or we can't decide it together with making Foundation an umbrella module we still have one option to implement URLSession with NIO: We could make a private _NIOForFoundation or something module which is a copy of a certain SwiftNIO release's NIO module that we update every once in a while (similarly to how CoreFoundation is updated). In other words: We would have Foundation vendor a private copy of NIO. That way everybody can still use the latest NIO release just Foundation would be stuck in the big-drop-twice-a-year model.

Can't share that enthusiasm as long as SPM is so badly integrated into Xcode.

SwiftNIO provides quite a nice Future / Promise implementation. With async / await out of reach for now, it could make sense to move them down from nio to foundation-core and expose them there.

1 Like

While the idea is good, SwiftNIO’s EventLoopFuture is quite specialised, as its name suggests. Specifically, each future is “bound” to an EventLoop, and the API contract is that each callback attached to the future will execute on that event loop thread. This contract makes writing NIO programs easier by allowing users to use the implicit mutex provided by the event loop to ensure thread safety.

This convenience does mean that EventLoopFuture is ill-suited to being a general purpose Future implementation, because it requires bringing in the notion of an EventLoop as well.

It may be better to consider either porting EventLoopFuture to a concrete type that does not make such an assertion, or using its interface to define a Future protocol, or both. That’s a separate discussion though.

3 Likes

Thanks for your answer. Where should this separate discussion be started? Is it a Swift, NIO or Apple discussion? Is a bigger audience even interested in generalizing the NIO Promises/Futures to a Standard Library or Foundation API? Or should we better wait for async / await and every lib will in the meantime add it's own promise classes?

I think it’s an appropriate discussion for the forums, but given the focus on ABI in the Swift 5 time frame I suspect that minimal progress will be made until that time.

In the meantime, third party packages should experiment with possible Future protocols to see what works well.

I think the direction of creating a common interface for future and promises can be relevant in helping us moving towards the direction of async/await. I was thinking about a forums post about that topic as well, but as cory pointed out the current focus with ABI will prevent major progress currently. Nevertheless, we could start writing a document which summarises current implementations of libraries in Swift that already implement them as well as have a look at how other languages use them. Maybe this can be useful in kickstarting the conversation as soon as Swift 5 is released

Any updates on this? Really looking forward to using a "more modular" Foundation. Losing libcurl dependency on Linux would be great, but also thinking of possible use of Foundation when WebAssembly target starts working. I doubt it would make sense to bundle SwiftNIO in WebAssembly builds of Foundation. URLSession would probably need to bridge to browser's implementation of fetch on JavaScript side.

Either way "more modular" Foundation would unblock a lot of things for many platforms.

4 Likes

Yup, we're still looking into this. @millenomi will be taking the lead on it, but first was getting the merge of iOS 12 / Mojave sources done.

3 Likes

Is there any news about this project?