Changes to Foundation must go through swift-evolution

Evolution Discussion
1

I'd like to talk about something which has been bugging me for a while. Apologies for not doing it sooner, but recent events have brought this to my mind again and I think we should discuss it.

For Swift 5.0, the Foundation team totally redesigned the Data type and did not put it up for review, in complete contradiction to their own guidelines for contributing:

The interface of Foundation is intended to be both stable and cross-platform. This means that when API is added to Foundation, it is effectively permanent.

It is therefore critical that any code change that affects the public-facing API go through a full swift-evolution review process.

They didn't do that. Besides the possible improvements we as a community may have been able to suggest, they introduced things like the ContiguousBytes protocol which I feel is very likely to be duplicated by the standard library at some point (in fact, it already is - there's an internal _HasContiguousBytes protocol which does exactly the same thing).

I mentioned this when I happened to spot the PR for the new Data design, and was told by @itaiferber that only standard library stuff needs to go through swift-evolution. This implies that Apple can make whatever changes they like to Foundation's public API. I mentioned it again during the recent review for ContiguousCollection, with the hope that we would hoist the protocol to the stdlib before ABI stability kicked in, and was just ignored. I accept that perhaps I should have pressed the issue a bit more.

In any case, experiences like this severely diminish my enthusiasm for contributing to this project. I think we need to do better in future.

As a first step, we need to reassert swift-evolution's authority over the Foundation project. When Apple made it a part of the open-source project, my understanding was that they were accepting that the community would have a say in future non-Darwin-specific public APIs. Is that not the case?

Ping: @tkremenek @Chris_Lattner3 @Tony_Parker @Philippe_Hausler

17 Likes
2

I know this isn’t a comfy answer… but Foundation is Apple’s. It’s up to the community to provide alternatives to the parts found in Foundation that can thus be produced with an evolution like process.

Having said this, Data should probably be in the stdlib, it’s rather fundamental. If it isn’t to move maybe evolution should be extended to encompass it.

Pragmatically I don’t think the whole of Foundation could ever be under Evolution, it is very large, has a wide scope and is integral to Apple business decisions.

8 Likes
3

I think this is only true of Darwin platforms. The linux foundation code is open source.

4

Does it, though? This is exactly the thing that I would like some clarification about.

Just you thinking that corelibs-foundation belongs to Apple and they have some special right to change whatever they like illustrates the problem. (EDIT: Not just "Foundation". That's an overloaded term which can also refer to the Obj-C wrapper around CF, which of course is still an Apple-private project. We're talking about the Swift wrapper around CF).

In fact, historically, we have plenty of proposals which are / only / about / Foundation. Features like Codable started out as pure Foundation-only proposals until review feedback made a case for making it part of the standard library (I remember very well - I was one of the ones who argued that case!). I'm pretty sure I'm not just dreaming this.

Really, I chalk this up to a simple mistake/misunderstanding. The Foundation team were probably working on a lot of prototypes in parallel, and after working on this project in private for so long, they likely just 'forgot' that they can't just make changes to the public Swift API like they can for Obj-C. It happens; we're all only human and I don't think it's fair to point the finger at any individual. Lots of people saw my earlier attempts to raise the issue and said nothing, and I myself should have pressed the issue more.

As I said, I think we should clarify what the situation actually is, and if mistakes were made, we should endeavour to do better in future.

If the DataProtocol design had gone through a swift-evolution review, perhaps somebody would have noticed the similarity to _HasContiguousBytes, and like Codable, perhaps parts of it would have been lifted to the standard library. Maybe even DataProtocol itself (in lieu of an actual stdlib Data type -- I agree with you about how fundamental it is). The project is too big for any single person to know everything, and we benefit from having as many eyes as possible scrutinise the design.

8 Likes
5

I will restate my opinion on this subject even if it is going to void :slight_smile:

I don't understand why we are mimicking Objc/Apple Foundation framework on Swift side. It brings many issues:

  • Foundation is driven by Apple, not by community nor Swift
  • We have weird types for Swift (NSArray, NSNumber, ...)
  • There are missing methods returning fatalError at runtime, making the written code not safe
  • We have dependencies issues because Foundation is a big monolithic requiring us to do what I think is "awkward" stuff (Pitch: Move URLSession to new FoundationNetworking module - #54 by Max_Desiatov)

Why are we not making a real Swift API, one starting from scratch using Swift idioms and not related to any ObjC/Apple stuff? That API would be available on all platforms and would be the one to use to make portable code.
On the other hand, Foundation would only be available and usable by macOS/iOS users.

7 Likes
6

When Foundation for Linux was announced and open sourced, it was specifically stated that the core team would not consider any proposals for replacement or duplication of functionality in Foundation. That decision has harmed Swift to this day. It seems like Apple has used it as an excuse not to development the native functionality Swift needs, either by pointing to Foundation and saying "There it is!" or by saying "If Foundation's so bad, we'll see what the community comes up with!", while Swift has no package ecosystem. Both responses lend credence to the outside opinion that Swift is an Apple language and isn't useful anywhere else, despite the heroic efforts of server-side Swift community.

The thing is, Apple knows about these issues, it just refuses to address them, while at the same time refusing to acknowledge they won't address them. As long as Apple is the controlling influence in Swift's development, there will be certain things, like a package repo and search engine, that just won't exist without their blessing. There are a lot of solutions to these problems, it just seems like Apple has chosen the "none of the above" option.

9 Likes
7

I would like to point out something which bothers me about this thread - @millenomi has done an amazing job with Foundation, and I think that the work that has gone into it is being overlooked or marginalised. There have been a bunch of improvements, even if they are not at the rate that many would like (I know, I wouldn't mind the support on Linux and Windows improving faster).

Designing a new library can be an interesting (and challenging) endeavour. It is not a task to be taken lightly, particularly when the library is not intended to fulfill a single well defined role. Furthermore, there is the larger ecosystem of software to consider as well.

Would the intention be to also remove swift-package-manager, SourceKit-LSP, etc from the things which are deemed part of Swift? Without a LSP alternative, particularly one which can match the language changes and is being improved, I'm certain this would prove to be detrimental to the project overall - that is constantly one of the pieces that most new developers to the language request.

Foundation bridges over a lot of details of the underlying system. Have you looked at the details of how bundle resource loading actually works? Or perhaps how the file system accesses are abstracted? Can you honestly say that you would be able to provide a sufficiently flexible interface that can work across HFS, NTFS, EXT4, and APFS? (yes, there are other file systems, and they all have their own idiosyncratic bits, but this set covers some wide variety of systems). Another area that one may not think much of - TimeZones. This is again not something that is easily made uniform over all the targets.

I really have to wonder what this is about. The fact that there remain a few classes which have their NS prefixes remaining? The implementation is incomplete in certain areas? The fact that the API is not what you want?

If the naming of the public interfaces is an actual concern, perhaps that can be addressed - perhaps you could start working on Foundation to help migrate the interfaces while ensuring the bridging interfaces remain intact. If it is the missing pieces in the library, perhaps you could help improve that coverage on other targets. It really does feel at times that @spevans is the only one who is pushing on Foundation for Linux. As others have stated, there is a broad surface here, and part of the problem is the number of people working on it.

Having worked extensively within the implementation of Foundation (and CoreFoundation) quite recently, I feel that although the code base may be large and complicated, it really does a pretty good job at abstracting the details of the underlying system. That is not to say that the library is perfect. I have my own reservations with parts of it, but, broadly speaking, I feel like it does the platform abstracting bit fairly well.

Note that I am not suggesting to not try, but, I think that Foundation is needed and will remain necessary for a very long time. In fact, one approach may be to back the new APIs by Foundation! Designing a separate API could be interesting and perhaps if it is sufficiently well designed, it would be adopted by the greater Swift ecosystem and in a decade people will be arguing that it needs to be replaced :laughing:

19 Likes
8

Well, the point of this thread is not for everybody to air all of their grievances about any part of Foundation. It's about clarifying the process by which changes to Foundation's public API are made.

There are lots of things I like about Foundation, and the new DataProtocol design. At the same time, not going through a review has led to a worse result because we now have duplicate protocols for ContiguousBytes. That means we will need some hacks to bring it up to the stdlib, or types will need to conform to both Foundation's and the stdlib's version in order to hit all the fast-paths the protocol was designed to support.

9

My point, if not others, is that Foundation made these abstractions years, if not decades ago, and not all of them have aged well. Forcing them to be the de facto standard for a new language seems like a generally poor decision, one driven by an attempt to save development effort rather than create the best language. Nearly every Foundation type has some sort of historical wart which not only confuses users new to Swift but which Linux Foundation is forced to match, wasting development effort and propagating the weird behaviors.

That's exactly the point: development of Swift-native functionality was immediately cut off for all platforms when Apple open sourced the language, and yet they continue to invest minimal resources into making the language cross-platform. The broad surface of Foundation actually works against the language here, as it forced any and all developers interested in the cross platform story into development of a framework which is both massive and yet must match the behavior of a closed-source counterpart. Not only is this not a recipe for prolific community contribution (I, personally, have no desire to build out counterparts to the Foundation APIs I have to use in my day and night jobs since I don't find them good Swift.), but it means that there are still, years after it was introduced, silent bombs of unimplemented functionality hidden with Foundation on Linux.

No one is arguing that Foundation doesn't work, or that those who have spent their time developing it on Linux don't deserve credit for such a thankless task. It's just that the library just fundamentally doesn't fit into Swift and is necessarily and deeply limited by Apple's control and that it has limited Swift's growth as an open source language.

I think it has only become necessary because there's no alternative. I think more could've been accomplished in the same time by developing native Swift abstractions of many of the same features that live in Foundation, but which would've been available equally across all platforms Swift supports, without having to match the eccentricities and depth of Foundation.

4 Likes
10

I agree with some of the issues here but I think to be fair this has been a trade off also to help out compatibility with a lot of the Obj-C frameworks we use. Imagine the confusion if we came up with an incompatible type of AttributedString and then tried to hoist it on UITextView? Ouch.

I don’t necessarily think there are any completely good options here because of how essential Foundation is to Apple Platforms, which, to be realistic, are the core for Swift development at the moment.

This is a platform compromise that hurts cross-platform adoption. I think the first step is we need to admit the limitations of this solution. Then we need to assess whether corelibs foundation is part of a reasonable compromise, and whether Apple is comfortable with us contributing to the Swift interface to Foundation, for the reason that the compromise leaves the swift foundation interface have a much larger target market than just Apple Proprietary Frameworks.

1 Like
11

Sure, there will be APIs which age well and some which won't. If you look at the Foundation APIs, they have been evolving over the past couple of years, trying to replace the APIs as it makes sense. Again, I think that this is a problem of velocity - and that is only made worse if people abstain from working on the problems rather than trying to find a way to solve them.

Sure, but the same can be said for new APIs as well. Heck, JavaScript has its own set of warts that everyone knows ... its just the fundamental problem of software engineering - there are tradeoffs, they are made with the current information at hand, and will introduce warts that will live longer than we like. But, we do have the ability to smooth them over with them.

Oh, if the desire is to create a library with a very narrow use case specifically for yourself, I don't think that anyone is going to stop you :slight_smile: But, generally speaking, the cross platform nature of the existing libraries is, IMO, a major benefit. Writing software is difficult, and porting software is even more difficult because you now have to also understand what the authors were trying to do in what way and for what reason(s).

I think that we can agree to disagree here.

I think that this is the crux of the issue. You do not care about portability and want the software to work in a very specific environment. That is fair, and I think that is reasonable reason for you to develop your own set of abstractions to work in that environment. If the environment is not very eccentric, then perhaps you could open source the library even! But, for many others, portability is a fundamental concern, and when you start working with sufficiently many environments, you find that the abstractions created in Foundation are necessary to deal with the differences.

1 Like
12

Oh my gosh, we're going down the road of big essays and rants about anything people don't like.

Can we please try to keep it on-topic? This is about process.

7 Likes
14

I feel that it doesn't hurt it as much as one might expect at the surface. Software paradigms have largely converged to a very similar, albeit spelt very differently, set of behaviours. Modern Windows (particularly with the Windows Store) and Linux (particularly with flatpak) and macOS have a lot of similarities in the fact that everything is bundled together and distributed as a single entity.

UI considerations have changed and also largely developed harmoniously (much to the ire of many - I want my UI specialised for my environment! :stuck_out_tongue:) with converged UIs for traditional desktop and touch enabled environments (see GNOME and Windows at the very least).

Overall, many of the problems with modern day application writing is very similar and the APIs here seem to work extremely well for those considerations. There are still improvements which can be made, but, I don't see Foundation hindering cross-platform adoption but rather encouraging it. Having to re-implement readlink for every target is quite boring and uninteresting, and having to do that poorly for each library sounds very revolting.

1 Like
15

Well said.

When swift-corelibs-foundation was opensourced, it looked like this:

$ tokei
-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Assembly                3           60           51            0            9
 C                      78        90273        73140         7341         9792
 C Header              260       122831        30950        79695        12186
 Markdown                6          470          470            0            0
 Objective C             1          198          141           46           11
 Python                 10         1905         1541           96          268
 Swift                 108        23094        14274         4916         3904
-------------------------------------------------------------------------------
 Total                 466       238831       120567        92094        26170
-------------------------------------------------------------------------------

Today it looks like this:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Assembly                3           89           72            0           17
 BASH                    1          141          104           18           19
 C                      97       103832        84009         8341        11482
 C Header              283       125734        33056        80012        12666
 CMake                   7         1973         1758           97          118
 Markdown               11         1317         1317            0            0
 Objective C             1          180          141           28           11
 Python                 12         2838         2394          111          333
 Shell                   1            5            4            1            0
 Swift                 253       114753        83276        14300        17177
 Plain Text (!)          5            5            5            0            0
 XML                     2           27           27            0            0
-------------------------------------------------------------------------------
 Total (!)             676       350894       206163       102908        41823
-------------------------------------------------------------------------------

Huge progress has been made, but of course everyone wishes we could go faster.

3 Likes
16

Given the overall sentiment in this thread, I wonder if there is something to be said for the C++ style of development.

Effectively, boost ends up being a playground to try out new APIs, which are then adopted into the std::experimental (previously std::tr1) namespace, bringing it into the standard library for broader adoption. The APIs continue developing until they are formally brought into the std namespace.

This could give us an area to experiment with new APIs with people being fully aware that the APIs will continue to evolve until such a point where they are merged into the library. I think that will reduce some of the burden of the evolution process and allow for quicker iteration and by the time that API is ready to be promoted into Foundation or the standard library, it will have seen enough usage to make it fairly easy to push for through the evolution process.

8 Likes
17

I would definitely support that. Many of the recent additions to the standard library could have been done as separate libraries.

One example: I think the recent CollectionDifference functionality should have been prototyped in a library before becoming part of the standard library. I mentioned it during the review. Now we have threads with people complaining about the usability of parts of it, and the proposal author has to explain that it evolved like that over time. Again, not to point fingers, but it's worth getting actual experience using the thing from people who didn't write the implementation.

This seems to be my word-of-the-week, but that kind of process is just amateurish. Even if you've been working on something privately for years, if you approached the C++ standards committee without lots of real-world usage data, they'd laugh you out of the room.

Are we building this language for the long-term? If yes, then what harm does a 1/2 year incubation period really do?

3 Likes
18

I'd like to point out that this is somewhat how the server group is currently working.

3 Likes
19

This is something I could get behind as well. I've seen things like this suggested before, but it never goes anywhere.

Clearly Foundation is here to stay, it has decades of work behind it (beginning at NextStep before being acquired by Apple, thus the NS prefix). However, too much of its cross-platform implementation is NSUnimplemented() to really call it cross-platform and many of its APIs are unintuitive and not swifty (just try getting file attributes if you disagree).

Rather than simply porting Foundation directly into swift with all it's flaws, we should redesign it. There is a lot that can be learned from Foundation, but we're not in the 1980s world where it began any more. I think everyone would benefit from a new set of APIs that feel more intuitive and fit better into swift than what we have today with Foundation.

You bring up a good point which leads to my proposed idea:

  1. Create a new working group, just like the Swift Server Work Group, that is aimed at cross-platform Foundation-like functionality
  2. Create a swift version of Boost (name debatable) which will eventually replace Foundation
  3. Determine the set of functionality that deserves "blessed" support and should be a part of Boost, but doesn't necessarily belong in the standard library
    • This will probably just come from what Foundation does today, but may include some extra things and leave out other things
  4. Begin designing/building this functionality one piece at a time, just like the SSWG is doing
    • All functionality should go through an evolution process
    • Each piece of functionality will begin as its own separate library before inclusion in Boost
  5. Once a library is determined stable on all of Swift's supported platforms merge it into Boost
    • Perhaps require ABI stability as well so that anything could potentially be merged into the standard library

We don't have namespacing and so I'm not sure how the process of moving from Boost into the standard library should work. It might just be as simple as remove it from Boost at the same time we add it to the standard library.

6 Likes
21

People who want to talk about Foundation in general are welcome to create new threads. Karl has asked that this thread be specific to his topic; I will delete further posts that seem to be off-topic.

14 Likes
22

I’ve often felt that Foundation is missing a process. The process contemplated by Karl’s post and Jacob’s amplification is on the right track.

Giving Foundation a process will help to attract contributors.

4 Likes