I would love to see the community reevaluate parts of Foundation's design and functionality and build new Swift-native implementations of them. This would allow that functionality to move to community governance, fit better into the cross-platform swift ecosystem, and allow innovations in API areas that sometimes need to be dusted off and re-evaluated, both because Swift is different than Objective-C, but also because the industry has moved significantly forward in some areas.
25 Likes
How can the community bootstrap something like this to the point where replacement is even possible? In a general sense, the community is already doing this through a variety of libraries. It's just that Swift lacks the package ecosystem to expose these projects in any meaningful way. Barring Apple investment, even something as simple as what you suggest is unlikely. I would like to see something like @Ponyboy47's suggestion more formally, but again, it's not something the community can bootstrap itself.
I think if we just do it - perhaps offline with those who want to spearhead it to formalize a charter (or manifesto, which might be a better term). Then a thread starts with the charter to gather some initial focus points and a repo created for the process.
We're talking about the community driving something to show value to Apple - so it needs to be organized by the community until then.
But then again, that matches my personality of doing things and asking for forgiveness later 
IIRC, the way that boost started off is that it was just a small auxiliary library that gained popularity due to it having a bunch of functionality that was well implemented and often used. It wasn't the only library that did this, e.g. loki was another popular alternative. It just happened to be that enough people preferred one style over the other and it had enough useful functionality that it became the de facto standard. I think that if someone wrote a high quality library that replaced the functionality, it could be grown into that. These days, GitHub does provide an easy means of centralising the development effort. Having this be driven by Apple only encourages the same thing to occur again - I think that this really should be done external to the actual project to experiment with what is possible, with feedback back into the core for enhancements for the language, standard library, compiler, etc to support this effort.
3 Likes
I think this is key. Start small. Either as improvements on top of Foundation or as alternative implementation to a small subset of Foundation.
3 Likes
There are already several examples of people making improvements to Foundation or replacements for parts of it in the wild.
Finding these libraries is a huge problem though. There's no central package list (and there may never be one) so everything spreads either through word of mouth or by getting lucky with a google search. There may be a "better" library with a cleaner API, better cross-platform support, and a dedicated set of developers that has very small adoption because it's very difficult to find new 3rd party swift libraries.
Just look at all the Path libraries:
- FileKit
- 1,954 stars
- Extensive additions built around FileManager
- Doesn't even support Linux
- PathKit
- 999 stars
- Simple convenience wrappers around FileManager
- FileSmith
- 15 stars
- Simple type-safe path interactions through FileManager
- TrailBlazer
- 1 star
- Total path library redesign built on top of C libs
And those are just the ones I know about.
It's nearly impossible to compete with Foundation because everyone knows about Foundation, but finding a 3rd party library is so difficult. They have weird names sometimes so if you don't search "just right" then you won't even find all the possibilities. It can be hard to bring yourself to use a newer library that doesn't have a large number of stars. Some people may not even think to look outside of Foundation or may not be able to. SPM doesn't work on iOS/tvOS/watchOS yet and so those devs are stuck with Foundation or they have to use Carthage/CocoaPods.
There are so many reasons why I think the idea of "just let the community build it and the best one will win" will never actually work, heck, the #1 Path library outside of Foundation doesn't even support non-Darwin platforms!
This is why I suggested creating another working group along the lines of the SSWG. The motivation is nearly the same. There were 3+ different implementations of very similar network frameworks (Vapor, Kitura, Perfect, etc). All these devs got together to build a single library and now there are tons of NIO-based libraries out there. Today we have Foundation and all the hundreds of 3rd party libraries that either replace or add useful functionality on top of it. Why not use the same motivation and concepts to rebuild Foundation?
A 'Boost' working group wouldn't replace all of Foundation right off the bat. They too would start small to replace subsets of Foundation. Over time we would have a full-featured boost-like library that rivals Foundation, but feels more swifty. One that used the good and learned from the bad. One that was built with cross-platform stability from ground zero. One that has gathered input from the entire swift community instead of a 3rd party library which only gathers input from the few which use it.
9 Likes
I would like to add that without "blessed" support it is highly unlikely that a single 3rd party library will ever gain the widespread support of the community.
Again pulling from the example of the SSWG, there were 3 libraries that all had thousands of GitHub stars and significant adoption rates (they all still do). However because they were competing with each other they all had their own implementations of network protocols which each had their own sets of pros/cons and it is highly unlikely there ever would have been "one true winner."
Now that there is a single NIO framework from which all of these libraries can build, there is one less area in which they are competing. There is a single framework which has been built with cross-platform support, is highly performant, and meant to be extensible. The entire community has won and is still winning because of the creation of the SSWG and the resulting NIO frameworks. Logging is now becoming unified and tons of useful libraries are being created on top of NIO that wouldn't have even been possible in the pre-SSWG swift world.
Instead of waiting for a winner in a never-ending competition, we should work together to create the building blocks of a great language library, just like what's been going on with the SSWG in networking.
What really bugs me about the redesign for Data is that it's completely thrown away NSData's support for toll-free bridging of discontiguous dispatch_data_t's (NSData forces contiguity when you access bytes but otherwise you can work with it discontiguously and even has a method to enumerate the discontiguous regions). This isn't documented, and the only hint is the fact that Data.regions is a CollectionOfOne<Data> (and the fact that enumerateBytes is deprecated, but the deprecation notice says to use regions). I had to dig through the implementation to discover that the moment you cast an NSData to a Data it access the bytes property, which linearizes the data.
I'm sure the implementation is a lot easier to deal with when it just has to manage a single byte buffer, but Data already has 4 different internal storage types already, surely it could have had a storage type that supported discontiguous regions.
I'm also annoyed that regions on NSData and DispatchData are arrays instead of lazy sequences. I get that the existing dispatch_data_t API only supports internal iteration of regions and not external iteration, but this is the Foundation team, they could have added the requisite dispatch_data_t APIs to the framework for external iteration. Ultimately, the point is I shouldn't have to allocate just to read discontiguous data.
And on a related note, it's also now impossible to cast a DispatchData to an NSData, even though that's supposed to work. I can write something like unsafeDowncast(data as AnyObject, to: NSData.self) but this is gross and unsafe.
All of this really should have gone through some kind of review process visible to the community. Doesn't even necessarily have to be the full swift-evolution process, it could just be "Here's a document in proposal format explaining the changes we're making, there's no review process, unless someone has a really serious objection we haven't considered".
12 Likes
One can always pick the best parts of existing libraries too (including open source Foundation). And have a website and a place in these forums similar to SSWG. All it needs is few active people to drive it forward and engage with the rest of the community to get something that appeals to many if not most swift developers.
While the vision might be to be a full alternative for Foundation, for the first years, the plan has to be something much smaller.
2 Likes
Agreed, I would suggest something along the lines of the boost model. Boost has been great for the c++ community because it has allowed development of libraries with an active community of users, which has been a great way to aide standardization. Upon standardization, the libraries do get changed (they just don't get adopted as is) in many cases, but this has led to things (e.g. filesystem APIs, hey we need those in Swift!) that wouldn't have happened with the full burden of the C++ committee process.
4 Likes
If this is actually true, though, that suggests that there is a problem with the Swift community.
For example, while rubygems does have a central index, I don't think that most popular Ruby gems are found through this way. Rather people find libraries and tools through:
- Reading blog posts
- Reading reddit discussions
- Listening to/watching pod-/screencasts
- Learning from other open source applications or libraries/frameworks (for example, Rails)
- Going to meetups and conferences
- etc.
It's probably the same for many other language communities, e.g. JavaScript (not sure about the Java world, they might be a bit more conservative about trying new stuff).
I don't know why this doesn't work for Swift. Maybe the language is too young and/or the community not diverse enough (most Swift users write iOS/macOS apps).
2 Likes
I would disagree here. I've programmed in ruby for 10 years both professionally and also personally. The overwhelming majority of packages I have found have been directly through the rubygems central package index.
That's not to say that there haven't been a handful of libraries that I came across while reading blog posts, watching tutorials, or looking into another open source applications' dependencies, but most packages I found because I had a need and searched the central index to see if there was anything available that would suit my needs.
Of course I'm basing my thoughts entirely on my own experiences, which may or may not reflect the true norm, in which case you would be correct in your statement.
Separate rant about the usefulness of a Swift Package Index
It's extremely handy to have a need like, "I need an LZ4 compression library." Go to rubygems.org, search "lz4", pick and use the one that's right for you (I actually use extlz4 in production code at the company I work for).
Even if you do a google search for "rubygem NAME" you primarily get results from rubygems.org.
Today, the primary way to find an SPM package is to search GitHub, or to search google. Neither of which is going to yield all the results you may be looking for.
I don't think that anyone is questioning the efficacy of a centralised collection of packages. That is certainly a very powerful tool (and one of the strongest benefits of Linux distributions IMO), to the point where it has been replicated everywhere. I think that the difference is that I (and seemingly others) think that it is not an absolute requirement to create something viable.
As has been pointed out, by you nonetheless, there are people writing libraries to help create a better API set for file system operations. This is wonderful, but, what prevents the top libraries from actually combining efforts and unifying into a single paradigm? That type of work is what is crucial to efforts like this. Once the need is identified, driving towards a convergent design with the flexibility to make radical alterations.
Although Foundation is not perfect, it is a well exercised piece of software which is portable (it already works on macOS, Linux, FreeBSD, android, and Windows). Many of the examples which you pointed out miss that functionality, which means that adopting them would be a major regression. Driving the projects towards a unified model and porting it to the same set of platforms as Foundation should make something like that interesting to look at because then we can truly see what it would look like. As an example, TrailBlazer uses the C APIs - which is going to require a new library to abstract over because the APIs on Windows are different.
Repeating my initial sentiments - this is something which takes time and effort and is easy to underestimate the amount of work involved. There are improvements which can be made, but, it needs someone from the community to drive it to actually show the possibilities (as you can see from the thread, everyone agrees that we can do better, but we just don't know how yet).
2 Likes
My primary argument is not to create a package index. I've already acknowledged upthread that we may never get one from Apple. It was merely meant to be an example of what we're lacking that makes our current situation for finding the supposed "best" 3rd party library so difficult.
There is nothing that stops anyone from trying, but it's unlikely to ever happen naturally. There are significant differences in the approaches taken by each of those libraries. No one wants to abandon all their work to join forces with someone on something that will take years to reach the adoption levels they may already have.
Look again at the SSWG. Nothing stopped Kitura, Vapor, or Perfect from joining forces to build a common network stack. Why didn't it happen naturally then? Why was the SSWG needed/created at all? I bet that they never would have joined together of their own accord to build NIO. They needed a central authority with guaranteed oversight and support and adoption.
What I'm primarily pushing for is an organized effort with some oversight to build this new boost-like library.
You (and many others) agree there needs to be a community driven and public effort. This is all I've been pushing for. The SSWG is still largely driven by the community. The devs are not all associated with Apple. There is a proposal process where items must gain the support of the community. It's not quite as rigid or involved as the swift-evolution proposal process, but the process is still there and I think it works well as a model for what we should do in a Boost-like library.
There would be minimal oversight, but guaranteed support from Apple. With this "blessed" support, adoption and contributions are far more likely to happen. A minimal proposal process would ensure we're meeting the needs of the community and not doing anything stupid.
A Boost Working Group would be community driven. The name/organization just guarantees support. Apple's blessing means a lot. If NIO came out as it's own separate community driven effort then we'd just be in the XKCD standards situation where there would be one more competing framework. Giving it the Apple stamp of approval meant that "this is the future." It guaranteed the massive adoption its already had despite being a new framework.
3 Likes
I'd love to see the point of view of people from the Linux Foundation team. Is there any indication that they are aware of this discussion?
2 Likes
Well, as a Linux and Windows Foundation developer/user, I am in favour of this as long as there is a concerted effort to ensure that the APIs are consistent and correct across Linux and Windows.
7 Likes
Ping: @tkremenek @Tony_Parker @Philippe_Hausler @itaiferber
While it is all very nice and interesting to consider which future processes we could adopt, we still don't have official confirmation that Foundation has any process right now. A fundamental data-type was entirely redesigned without any formal review and, so far, nobody knows whether that was an error or not.
What's more is that we have even discovered that some functionality has been lost in the transition. Was this intentional?
2 Likes
Hi all,
Thanks for starting this discussion. I think it gives us a great opportunity to bring more clarity to our process. Let's start with a clear definition of the projects we're talking about here.
Foundation
Foundation is a framework and Swift "overlay" that ships with each version of macOS, iOS, watchOS, and tvOS (sometimes collectively referred to as "Darwin"). It is closed source, written in a variety of languages (C, Objective-C, and Swift), and uses an Apple-internal process for evolution of its API and functionality. The goal of Foundation is to provide first-class API for Darwin platforms, with a focus on these goals (outlined in our README):
- Providing basic utility types
- Establishing consistent conventions
- Support internationalization and localization
- Provide a level of OS independence
As a part of the operating system, Foundation is often tasked with adding API that forms a part of a larger feature across a release. Foundation also adds other new API requested by developers (both internal and external) after the team has agreed upon its general utility, implemented and tested the functionality, and reviewed it.
Sometimes, features that Apple wants to add to Foundation will require changes to the Swift compiler or standard library. In those cases, the Foundation team at Apple will run a Swift Evolution proposal. Examples include SE-0161 Smart KeyPaths, SE-0166 Swift Archival & Serialization, SE-0167 Swift Encoders, and SE-0170 NSNumber bridging.
swift-corelibs-foundation
swift-corelibs-foundation is an open-source library created concurrently with the release of the Swift project. Its goal is to bring API parity with Foundation to non-Darwin platforms. These "core" APIs (along with XCTest and libdispatch) exist to support a broad level of common functionality that developers can rely upon for all Swift development across platforms.
A large subset of Foundation's implementation has actually been open source for a long time. This was imported into swift-corelibs-foundation to form a common base of implementation. This library is called CoreFoundation ("CF"). Over the past few years, Apple has open sourced some key parts of Foundation by moving them into CF. For example: CFURLComponents and CFCalendar Enumeration.
Thanks to an amazing level of effort from the open source community (highlighted above in this thread), an incredible amount of progress towards realizing this goal has been made.
The Intersection
These two projects have different goals, but of course they are also inexorably linked with each other.
- Some implementation can be shared, as detailed above. The Foundation team at Apple is hoping to do more of this over time, and Swift reaching ABI stability will improve our ability to do so.
- Most of the team that works on Foundation is also involved with swift-corelibs-foundation. This includes code review, rewriting Objective-C in Swift or C so it is more portable, and writing completely new implementations.
- API added to Foundation should also be added to swift-corelibs-foundation, to maintain parity.
It is not a goal of swift-corelibs-foundation to add API to Foundation. However, the Foundation team does recognize the opportunity that Swift provides to take advantage of new language features to improve our interfaces. As these changes are made to Foundation, swift-corelibs-foundation also benefits. Examples include improved Scanner API and FileHandle API.
swift-corelibs-foundation is a hybrid open source project. It is clearly related to Foundation, but it does not bring all of Foundation to open source. The Foundation team welcomes suggestions for improvements to our APIs for Swift, including those in swift-corelibs-foundation. In addition, the Foundation team will continue to drive improvements to both projects on its own. As a key part of the operating systems and the Swift project, both Foundation and swift-corelibs-foundation are expected to be well supported and maintained alongside Swift itself.
Thanks,
Tony
18 Likes
I think Tony covered the primary aspects of the thread here so I won't add to that — I did, however, want to address some technical concerns here related to Data specifically:
While designing this API, we worked closely with the standard library team to discuss placement for this protocol (along with DataProtocol itself). I personally think it would equally appropriately live in the stdlib, but at the time this was not desired. In the future, if we decide that the better home for this is indeed in the standard library, we will explore mechanisms for sinking it down in an ABI-compatible way. (e.g. symbol aliasing and replacement with a typealias on the platforms which are relevant)
This was a decision that was closely considered and it was decided that Foundation is currently the most appropriate home for this protocol.
This aspect was long discussed and debated internally, and was one big driving force for the change in implementation here. Specifically, the fact that NSData and dispatch_data_t are toll-free bridgeable (unidirectionally) is purposefully not documented, and the fact that Data could wrap such a discontiguous data instance was often lost given that many method accesses (touching bytes anywhere) would implicitly and silently flatten the data anywhere.
After much debate, it was decided that it would be better to explicitly promise contiguity (which has allowed us to both simplify the implementation in several areas, and achieve much improved performance) for Data, with the added benefit of introducing API (DataProtocol) which would abstract over various types by allowing discontiguous access to the data. The intention here was to allow generic abstraction over both Data and DispatchData in a way that publicly enables discontiguous access rather than making this an implicit, undocumented fact, only if you don't look at the Data instance too funny.
The hope is that by leveraging DataProtocol where possible, we can help promote usage of DispatchData too to make discontiguous data instances more widely and easily accessible.
Along with this explanation, I'd like to personally take responsibility for doing such a poor job at communicating these changes to the community. The APIs here went through months (if not collectively years) of review and discussion internally, and I would have liked to express that better at the time of introduction, in spite of some internal constraints at the time.
I promise to you that we did not "forget" about our dedication to the community, though I completely understand the frustration; I'm saddened that the process here has caused you to feel that it is "amateurish", but am keenly devoted to improving this in the future.
18 Likes
Thank you for this writeup @Tony_Parker.
So to be clear: let's say I primarily use Swift on Linux, for my server backend or whatever. Now I want to propose some new API or make some changes to what comes bundled with the toolchain. If the API is to live in the standard library, I may pitch/implement it and participate in its review. However, changes to corelibs-foundation's API may only happen if they reflect a corresponding change to Foundation, therefore I should file a radar and let it go through Apple's opaque, internal process?
If that is the case - well, it is what it is, but I'm not sure that's obvious to everybody.
Well, it is actually documented. I actually agree that Data can't realistically provide a contiguous and non-contiguous API at the same time, but it would have been good to have some rationale and design discussion all the same. It's mostly just a big shock to learn that Apple continues to reserve the right to make such big changes unilaterally. Again, it is what it is, but I feel it undermines the idea that Foundation is some kind of extension of the standard library.
1 Like