Changes to Foundation must go through swift-evolution

I feel that it doesn't hurt it as much as one might expect at the surface. Software paradigms have largely converged to a very similar, albeit spelt very differently, set of behaviours. Modern Windows (particularly with the Windows Store) and Linux (particularly with flatpak) and macOS have a lot of similarities in the fact that everything is bundled together and distributed as a single entity.

UI considerations have changed and also largely developed harmoniously (much to the ire of many - I want my UI specialised for my environment! :stuck_out_tongue:) with converged UIs for traditional desktop and touch enabled environments (see GNOME and Windows at the very least).

Overall, many of the problems with modern day application writing is very similar and the APIs here seem to work extremely well for those considerations. There are still improvements which can be made, but, I don't see Foundation hindering cross-platform adoption but rather encouraging it. Having to re-implement readlink for every target is quite boring and uninteresting, and having to do that poorly for each library sounds very revolting.

1 Like

Well said.

When swift-corelibs-foundation was opensourced, it looked like this:

$ tokei
-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Assembly                3           60           51            0            9
 C                      78        90273        73140         7341         9792
 C Header              260       122831        30950        79695        12186
 Markdown                6          470          470            0            0
 Objective C             1          198          141           46           11
 Python                 10         1905         1541           96          268
 Swift                 108        23094        14274         4916         3904
-------------------------------------------------------------------------------
 Total                 466       238831       120567        92094        26170
-------------------------------------------------------------------------------

Today it looks like this:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Assembly                3           89           72            0           17
 BASH                    1          141          104           18           19
 C                      97       103832        84009         8341        11482
 C Header              283       125734        33056        80012        12666
 CMake                   7         1973         1758           97          118
 Markdown               11         1317         1317            0            0
 Objective C             1          180          141           28           11
 Python                 12         2838         2394          111          333
 Shell                   1            5            4            1            0
 Swift                 253       114753        83276        14300        17177
 Plain Text (!)          5            5            5            0            0
 XML                     2           27           27            0            0
-------------------------------------------------------------------------------
 Total (!)             676       350894       206163       102908        41823
-------------------------------------------------------------------------------

Huge progress has been made, but of course everyone wishes we could go faster.

3 Likes

Given the overall sentiment in this thread, I wonder if there is something to be said for the C++ style of development.

Effectively, boost ends up being a playground to try out new APIs, which are then adopted into the std::experimental (previously std::tr1) namespace, bringing it into the standard library for broader adoption. The APIs continue developing until they are formally brought into the std namespace.

This could give us an area to experiment with new APIs with people being fully aware that the APIs will continue to evolve until such a point where they are merged into the library. I think that will reduce some of the burden of the evolution process and allow for quicker iteration and by the time that API is ready to be promoted into Foundation or the standard library, it will have seen enough usage to make it fairly easy to push for through the evolution process.

8 Likes

I would definitely support that. Many of the recent additions to the standard library could have been done as separate libraries.

One example: I think the recent CollectionDifference functionality should have been prototyped in a library before becoming part of the standard library. I mentioned it during the review. Now we have threads with people complaining about the usability of parts of it, and the proposal author has to explain that it evolved like that over time. Again, not to point fingers, but it's worth getting actual experience using the thing from people who didn't write the implementation.

This seems to be my word-of-the-week, but that kind of process is just amateurish. Even if you've been working on something privately for years, if you approached the C++ standards committee without lots of real-world usage data, they'd laugh you out of the room.

Are we building this language for the long-term? If yes, then what harm does a 1/2 year incubation period really do?

2 Likes

I'd like to point out that this is somewhat how the server group is currently working.

3 Likes

This is something I could get behind as well. I've seen things like this suggested before, but it never goes anywhere.

Clearly Foundation is here to stay, it has decades of work behind it (beginning at NextStep before being acquired by Apple, thus the NS prefix). However, too much of its cross-platform implementation is NSUnimplemented() to really call it cross-platform and many of its APIs are unintuitive and not swifty (just try getting file attributes if you disagree).

Rather than simply porting Foundation directly into swift with all it's flaws, we should redesign it. There is a lot that can be learned from Foundation, but we're not in the 1980s world where it began any more. I think everyone would benefit from a new set of APIs that feel more intuitive and fit better into swift than what we have today with Foundation.


You bring up a good point which leads to my proposed idea:

  1. Create a new working group, just like the Swift Server Work Group, that is aimed at cross-platform Foundation-like functionality
  2. Create a swift version of Boost (name debatable) which will eventually replace Foundation
  3. Determine the set of functionality that deserves "blessed" support and should be a part of Boost, but doesn't necessarily belong in the standard library
    • This will probably just come from what Foundation does today, but may include some extra things and leave out other things
  4. Begin designing/building this functionality one piece at a time, just like the SSWG is doing
    • All functionality should go through an evolution process
    • Each piece of functionality will begin as its own separate library before inclusion in Boost
  5. Once a library is determined stable on all of Swift's supported platforms merge it into Boost
    • Perhaps require ABI stability as well so that anything could potentially be merged into the standard library

We don't have namespacing and so I'm not sure how the process of moving from Boost into the standard library should work. It might just be as simple as remove it from Boost at the same time we add it to the standard library.

6 Likes

People who want to talk about Foundation in general are welcome to create new threads. Karl has asked that this thread be specific to his topic; I will delete further posts that seem to be off-topic.

14 Likes

I’ve often felt that Foundation is missing a process. The process contemplated by Karl’s post and Jacob’s amplification is on the right track.

Giving Foundation a process will help to attract contributors.

4 Likes

I would love to see the community reevaluate parts of Foundation's design and functionality and build new Swift-native implementations of them. This would allow that functionality to move to community governance, fit better into the cross-platform swift ecosystem, and allow innovations in API areas that sometimes need to be dusted off and re-evaluated, both because Swift is different than Objective-C, but also because the industry has moved significantly forward in some areas.

25 Likes

How can the community bootstrap something like this to the point where replacement is even possible? In a general sense, the community is already doing this through a variety of libraries. It's just that Swift lacks the package ecosystem to expose these projects in any meaningful way. Barring Apple investment, even something as simple as what you suggest is unlikely. I would like to see something like @Ponyboy47's suggestion more formally, but again, it's not something the community can bootstrap itself.

I think if we just do it - perhaps offline with those who want to spearhead it to formalize a charter (or manifesto, which might be a better term). Then a thread starts with the charter to gather some initial focus points and a repo created for the process.

We're talking about the community driving something to show value to Apple - so it needs to be organized by the community until then.

But then again, that matches my personality of doing things and asking for forgiveness later :slight_smile:

IIRC, the way that boost started off is that it was just a small auxiliary library that gained popularity due to it having a bunch of functionality that was well implemented and often used. It wasn't the only library that did this, e.g. loki was another popular alternative. It just happened to be that enough people preferred one style over the other and it had enough useful functionality that it became the de facto standard. I think that if someone wrote a high quality library that replaced the functionality, it could be grown into that. These days, GitHub does provide an easy means of centralising the development effort. Having this be driven by Apple only encourages the same thing to occur again - I think that this really should be done external to the actual project to experiment with what is possible, with feedback back into the core for enhancements for the language, standard library, compiler, etc to support this effort.

3 Likes

I think this is key. Start small. Either as improvements on top of Foundation or as alternative implementation to a small subset of Foundation.

3 Likes

There are already several examples of people making improvements to Foundation or replacements for parts of it in the wild.

Finding these libraries is a huge problem though. There's no central package list (and there may never be one) so everything spreads either through word of mouth or by getting lucky with a google search. There may be a "better" library with a cleaner API, better cross-platform support, and a dedicated set of developers that has very small adoption because it's very difficult to find new 3rd party swift libraries.

Just look at all the Path libraries:

  • FileKit
    • 1,954 stars
    • Extensive additions built around FileManager
    • Doesn't even support Linux
  • PathKit
    • 999 stars
    • Simple convenience wrappers around FileManager
  • FileSmith
    • 15 stars
    • Simple type-safe path interactions through FileManager
  • TrailBlazer
    • 1 star
    • Total path library redesign built on top of C libs

And those are just the ones I know about.

It's nearly impossible to compete with Foundation because everyone knows about Foundation, but finding a 3rd party library is so difficult. They have weird names sometimes so if you don't search "just right" then you won't even find all the possibilities. It can be hard to bring yourself to use a newer library that doesn't have a large number of stars. Some people may not even think to look outside of Foundation or may not be able to. SPM doesn't work on iOS/tvOS/watchOS yet and so those devs are stuck with Foundation or they have to use Carthage/CocoaPods.

There are so many reasons why I think the idea of "just let the community build it and the best one will win" will never actually work, heck, the #1 Path library outside of Foundation doesn't even support non-Darwin platforms!

This is why I suggested creating another working group along the lines of the SSWG. The motivation is nearly the same. There were 3+ different implementations of very similar network frameworks (Vapor, Kitura, Perfect, etc). All these devs got together to build a single library and now there are tons of NIO-based libraries out there. Today we have Foundation and all the hundreds of 3rd party libraries that either replace or add useful functionality on top of it. Why not use the same motivation and concepts to rebuild Foundation?

A 'Boost' working group wouldn't replace all of Foundation right off the bat. They too would start small to replace subsets of Foundation. Over time we would have a full-featured boost-like library that rivals Foundation, but feels more swifty. One that used the good and learned from the bad. One that was built with cross-platform stability from ground zero. One that has gathered input from the entire swift community instead of a 3rd party library which only gathers input from the few which use it.

9 Likes

I would like to add that without "blessed" support it is highly unlikely that a single 3rd party library will ever gain the widespread support of the community.

Again pulling from the example of the SSWG, there were 3 libraries that all had thousands of GitHub stars and significant adoption rates (they all still do). However because they were competing with each other they all had their own implementations of network protocols which each had their own sets of pros/cons and it is highly unlikely there ever would have been "one true winner."

Now that there is a single NIO framework from which all of these libraries can build, there is one less area in which they are competing. There is a single framework which has been built with cross-platform support, is highly performant, and meant to be extensible. The entire community has won and is still winning because of the creation of the SSWG and the resulting NIO frameworks. Logging is now becoming unified and tons of useful libraries are being created on top of NIO that wouldn't have even been possible in the pre-SSWG swift world.

Instead of waiting for a winner in a never-ending competition, we should work together to create the building blocks of a great language library, just like what's been going on with the SSWG in networking.

What really bugs me about the redesign for Data is that it's completely thrown away NSData's support for toll-free bridging of discontiguous dispatch_data_t's (NSData forces contiguity when you access bytes but otherwise you can work with it discontiguously and even has a method to enumerate the discontiguous regions). This isn't documented, and the only hint is the fact that Data.regions is a CollectionOfOne<Data> (and the fact that enumerateBytes is deprecated, but the deprecation notice says to use regions). I had to dig through the implementation to discover that the moment you cast an NSData to a Data it access the bytes property, which linearizes the data.

I'm sure the implementation is a lot easier to deal with when it just has to manage a single byte buffer, but Data already has 4 different internal storage types already, surely it could have had a storage type that supported discontiguous regions.

I'm also annoyed that regions on NSData and DispatchData are arrays instead of lazy sequences. I get that the existing dispatch_data_t API only supports internal iteration of regions and not external iteration, but this is the Foundation team, they could have added the requisite dispatch_data_t APIs to the framework for external iteration. Ultimately, the point is I shouldn't have to allocate just to read discontiguous data.

And on a related note, it's also now impossible to cast a DispatchData to an NSData, even though that's supposed to work. I can write something like unsafeDowncast(data as AnyObject, to: NSData.self) but this is gross and unsafe.

All of this really should have gone through some kind of review process visible to the community. Doesn't even necessarily have to be the full swift-evolution process, it could just be "Here's a document in proposal format explaining the changes we're making, there's no review process, unless someone has a really serious objection we haven't considered".

12 Likes

One can always pick the best parts of existing libraries too (including open source Foundation). And have a website and a place in these forums similar to SSWG. All it needs is few active people to drive it forward and engage with the rest of the community to get something that appeals to many if not most swift developers.

While the vision might be to be a full alternative for Foundation, for the first years, the plan has to be something much smaller.

2 Likes

Agreed, I would suggest something along the lines of the boost model. Boost has been great for the c++ community because it has allowed development of libraries with an active community of users, which has been a great way to aide standardization. Upon standardization, the libraries do get changed (they just don't get adopted as is) in many cases, but this has led to things (e.g. filesystem APIs, hey we need those in Swift!) that wouldn't have happened with the full burden of the C++ committee process.

4 Likes

If this is actually true, though, that suggests that there is a problem with the Swift community.
For example, while rubygems does have a central index, I don't think that most popular Ruby gems are found through this way. Rather people find libraries and tools through:

  • Reading blog posts
  • Reading reddit discussions
  • Listening to/watching pod-/screencasts
  • Learning from other open source applications or libraries/frameworks (for example, Rails)
  • Going to meetups and conferences
  • etc.

It's probably the same for many other language communities, e.g. JavaScript (not sure about the Java world, they might be a bit more conservative about trying new stuff).

I don't know why this doesn't work for Swift. Maybe the language is too young and/or the community not diverse enough (most Swift users write iOS/macOS apps).

2 Likes

I would disagree here. I've programmed in ruby for 10 years both professionally and also personally. The overwhelming majority of packages I have found have been directly through the rubygems central package index.

That's not to say that there haven't been a handful of libraries that I came across while reading blog posts, watching tutorials, or looking into another open source applications' dependencies, but most packages I found because I had a need and searched the central index to see if there was anything available that would suit my needs.

Of course I'm basing my thoughts entirely on my own experiences, which may or may not reflect the true norm, in which case you would be correct in your statement.

Separate rant about the usefulness of a Swift Package Index

It's extremely handy to have a need like, "I need an LZ4 compression library." Go to rubygems.org, search "lz4", pick and use the one that's right for you (I actually use extlz4 in production code at the company I work for).

Even if you do a google search for "rubygem NAME" you primarily get results from rubygems.org.

Today, the primary way to find an SPM package is to search GitHub, or to search google. Neither of which is going to yield all the results you may be looking for.