Thanks Lily that makes sense! My understanding was that @spevans would like to have all of Foundation on Linux to be developed as a SwiftPM package so it could be released more frequently. I tried to explain why it won't be possible to have all of Foundation just be a a SwiftPM package. The URLSession part of Foundation however could be as you pointed out, especially if we leave the actual types in Foundation proper.
Pitch: Move URLSession to new FoundationNetworking module
Is there a way that we can move the SPI and revlocked parts of Foundation into a -core library that is built as swift-corelibs-foundation is today with the current restrictions, and all other parts that don't have these restrictions (so not just URLSession) would stay in an SwiftPM buildable Foundation. ie as @IanPartridge was suggesting?
Especially with SwiftPM still not really useable in a sane way from Xcode.
While this is an interesting idea, we have to note there would be multiple side-effects. For example, simply using String(contentsOf:) or Data(contentsOf:) on online resources would trap the process.
What about making SwiftNIO a Swift core library? Then we could establish the following dependency tree:
swift-corelibs-foundation (Foundation)
|
swift-corelibs-foundation-networkio (Foundation.NetworkIO aka SwiftNIO)
|
swift-corelibs-foundation-base (Foundation.Base)
The main disadvantage would be that SwiftNIO would have to be distributed with Swift instead of a Swift Package.
I'm very happy for Foundation to modularise a bit, and integrating with SwiftNIO seems like it would benefit both projects and the Swift language. On the other hand, I think source compatibility across platforms is very important; I don't like one platform writing one thing and other platforms write another thing.
We do have a kind of compile-time solution: we could add a fix-it which writes an import FoundationNetworking statement and make it part of the Swift 5 migrator in Xcode. Darwin Swift projects are almost always built with Xcode (not always, but almost always), and it has a mechanism for stuff like this.
Another idea is that we could expose a compatibility typealias through higher-level Darwin-specific frameworks like AppKit/UIKit. A fix-it might be a sufficient fall-back solution for code which uses URLSession but does not import one of those frameworks.
1 Like
I'm liking this idea of turning Foundation into a kind of umbrella module, with Foundation as the base set of functionality, plus FoundationNetworking (URLSession) and maybe FoundationXML too (to split libxml dependencies, which are also problematic for some). We could leave all of this in one repository, too.
6 Likes
@Tony_Parker sounds great! I'm just unsure how the dependencies would work out then. Because in that world, FoundationNetworking and FoundationXML would be SwiftPM projects but you could only import Foundation if you depend on both of them already, right?
I think we could use SwiftPM to build, but they would be distributed with Swift the same way they are today.
In other words, the problem of distributing more frequently is orthogonal to the problem of minimizing dependencies for clients.
1 Like
Ok, so you're saying we could ship compiled versions of Foundation* with Swift as we do today but they could be updated through SwiftPM.
So let's imagine the following scenario: A hypothetical Swift version 42.0.0 ships with FoundationNetworking 23.0.0 which depends on and ships SwiftNIO 128.0.0. Now we're trying to compile the hypothetical web server Steam 256.0.0 which depends on SwiftNIO from: "128.5.6" using swift build. SwiftPM would now realise that SwiftNIO 128.0.0 as shipped with Swift isn't good enough so it would update SwiftNIO (that means download and compile) to say 128.7.0, then recompile FoundationNetworking and everything that depends on it with SwiftNIO version 128.7.0.
Is that accurate?
So, by the first and second Foundation you mean two different modules, the umbrella and the base one? 
Oh, sorry. I had something there but messed up the markdown.
FoundationTBD as the base module - a different name than the overall Foundation module, which would contain all the same functionality it does today to preserve source compatibility.
3 Likes
Great! Also, if we finally embed SwiftNIO with the Swift/Foundation distribution would it be available to import from other modules or would it be only for internal use (with a mangled module name to avoid naming collisions, I suppose)?
I think I see what you're getting at here -- that you don't want to commit to ABI/API for SwiftNIO yet, so you want anything that depends on that to ship outside of Swift.
I'm not sure if we're there yet. Perhaps that is a blocker to adopting SwiftNIO inside URLSession.
Ok, let me be more precise on what my current position is (not 100% worked out as there are a few moving parts):
- I'd argue that ABI does not matter on Linux because SwiftPM doesn't support binary dependencies anyway. And Swift isn't part of any Linux distribution so shipping binaries without the libraries is not a thing (at least right now).
- We don't ever want to be fully API locked but we're happy to follow SemVer (we want to document what we consider public API more precisely soon anyway). For example if we said 'no new public types in SwiftNIO' without a new major version, we could probably live with that (my personal opinion, would need to talk to the rest of the team).
So I think we could get to a working system if we shipped (in the Swift distribution) a version of NIO and FoundationNetworking which would be used if you just run swiftc myfile.swift. If you use SwiftPM and your package depends on a version of SwiftNIO that isn't satisfied by the one that shipped with the Swift distribution, then SwiftPM would need to be able to update SwiftNIO and therefore also download & compile FoundationNetworking from github.com. It would be very important for us (and our users) that SwiftNIO can be updated if needed.
Incidentally, this model is exactly what Haskell does: GHC (the most important Haskell compiler) ships with a certain set of libraries in certain versions. All of those except for a package called base can be updated through the package manager. This feels quite similar actually: There's a small number (in Haskell's case 1 (base), in Swift's case maybe 3 (stdlib, FoundationCore/Base/TBD, Dispatch)) of packages that cannot be updated through the package manager but others can. This would require some work to SwiftPM though.
Does that sound workable for you or would you only consider a fully API & ABI stable NIO?
1 Like
There is some new precedent there if we ship something with the base distribution that's upgradable via SwiftPM. I'm not saying it's wrong, but it's a direction we probably should consider as part of a broader decision about how Swift is distributed on Linux.
I think we can still consider this smaller piece independently of that larger question, given that we don't depend on SwiftNIO (yet).
@Tony_Parker Indeed, it would set some new precedent but I'd argue that would be a great win for Foundation on Linux. If we had the mechanisms in place to update the parts of swift-corelibs-foundation through SwiftPM, the development could move much faster. It would also be a much more welcoming OSS project as you can release often und not when Apple decides to release a new Xcode version.
@spevans already expressed the feeling that this would be great for all parts of Foundation that do not need to be rev-locked with Swift and its runtime. I agree with that. URLSession could (because if its dependencies) just be the trailblazer into an arguably better model.
3 Likes
Thanks everyone for your feedback. I think we'll investigate the idea of Foundation as an umbrella module of some kind, and report back once we have made some progress there.
4 Likes
Sounds good! I really think we should investigate how we can have more parts of the Swift distribution (parts of Foundation, SwiftPM, ...) upgradeable through SwiftPM.
Should that not be something people think is useful or we can't decide it together with making Foundation an umbrella module we still have one option to implement URLSession with NIO: We could make a private _NIOForFoundation or something module which is a copy of a certain SwiftNIO release's NIO module that we update every once in a while (similarly to how CoreFoundation is updated). In other words: We would have Foundation vendor a private copy of NIO. That way everybody can still use the latest NIO release just Foundation would be stuck in the big-drop-twice-a-year model.
Can't share that enthusiasm as long as SPM is so badly integrated into Xcode.