Changes to Foundation must go through swift-evolution

Agreed, I would suggest something along the lines of the boost model. Boost has been great for the c++ community because it has allowed development of libraries with an active community of users, which has been a great way to aide standardization. Upon standardization, the libraries do get changed (they just don't get adopted as is) in many cases, but this has led to things (e.g. filesystem APIs, hey we need those in Swift!) that wouldn't have happened with the full burden of the C++ committee process.

4 Likes

If this is actually true, though, that suggests that there is a problem with the Swift community.
For example, while rubygems does have a central index, I don't think that most popular Ruby gems are found through this way. Rather people find libraries and tools through:

  • Reading blog posts
  • Reading reddit discussions
  • Listening to/watching pod-/screencasts
  • Learning from other open source applications or libraries/frameworks (for example, Rails)
  • Going to meetups and conferences
  • etc.

It's probably the same for many other language communities, e.g. JavaScript (not sure about the Java world, they might be a bit more conservative about trying new stuff).

I don't know why this doesn't work for Swift. Maybe the language is too young and/or the community not diverse enough (most Swift users write iOS/macOS apps).

2 Likes

I would disagree here. I've programmed in ruby for 10 years both professionally and also personally. The overwhelming majority of packages I have found have been directly through the rubygems central package index.

That's not to say that there haven't been a handful of libraries that I came across while reading blog posts, watching tutorials, or looking into another open source applications' dependencies, but most packages I found because I had a need and searched the central index to see if there was anything available that would suit my needs.

Of course I'm basing my thoughts entirely on my own experiences, which may or may not reflect the true norm, in which case you would be correct in your statement.

Separate rant about the usefulness of a Swift Package Index

It's extremely handy to have a need like, "I need an LZ4 compression library." Go to rubygems.org, search "lz4", pick and use the one that's right for you (I actually use extlz4 in production code at the company I work for).

Even if you do a google search for "rubygem NAME" you primarily get results from rubygems.org.

Today, the primary way to find an SPM package is to search GitHub, or to search google. Neither of which is going to yield all the results you may be looking for.

I don't think that anyone is questioning the efficacy of a centralised collection of packages. That is certainly a very powerful tool (and one of the strongest benefits of Linux distributions IMO), to the point where it has been replicated everywhere. I think that the difference is that I (and seemingly others) think that it is not an absolute requirement to create something viable.

As has been pointed out, by you nonetheless, there are people writing libraries to help create a better API set for file system operations. This is wonderful, but, what prevents the top libraries from actually combining efforts and unifying into a single paradigm? That type of work is what is crucial to efforts like this. Once the need is identified, driving towards a convergent design with the flexibility to make radical alterations.

Although Foundation is not perfect, it is a well exercised piece of software which is portable (it already works on macOS, Linux, FreeBSD, android, and Windows). Many of the examples which you pointed out miss that functionality, which means that adopting them would be a major regression. Driving the projects towards a unified model and porting it to the same set of platforms as Foundation should make something like that interesting to look at because then we can truly see what it would look like. As an example, TrailBlazer uses the C APIs - which is going to require a new library to abstract over because the APIs on Windows are different.

Repeating my initial sentiments - this is something which takes time and effort and is easy to underestimate the amount of work involved. There are improvements which can be made, but, it needs someone from the community to drive it to actually show the possibilities (as you can see from the thread, everyone agrees that we can do better, but we just don't know how yet).

2 Likes

My primary argument is not to create a package index. I've already acknowledged upthread that we may never get one from Apple. It was merely meant to be an example of what we're lacking that makes our current situation for finding the supposed "best" 3rd party library so difficult.

There is nothing that stops anyone from trying, but it's unlikely to ever happen naturally. There are significant differences in the approaches taken by each of those libraries. No one wants to abandon all their work to join forces with someone on something that will take years to reach the adoption levels they may already have.

Look again at the SSWG. Nothing stopped Kitura, Vapor, or Perfect from joining forces to build a common network stack. Why didn't it happen naturally then? Why was the SSWG needed/created at all? I bet that they never would have joined together of their own accord to build NIO. They needed a central authority with guaranteed oversight and support and adoption.

What I'm primarily pushing for is an organized effort with some oversight to build this new boost-like library.

You (and many others) agree there needs to be a community driven and public effort. This is all I've been pushing for. The SSWG is still largely driven by the community. The devs are not all associated with Apple. There is a proposal process where items must gain the support of the community. It's not quite as rigid or involved as the swift-evolution proposal process, but the process is still there and I think it works well as a model for what we should do in a Boost-like library.

There would be minimal oversight, but guaranteed support from Apple. With this "blessed" support, adoption and contributions are far more likely to happen. A minimal proposal process would ensure we're meeting the needs of the community and not doing anything stupid.

A Boost Working Group would be community driven. The name/organization just guarantees support. Apple's blessing means a lot. If NIO came out as it's own separate community driven effort then we'd just be in the XKCD standards situation where there would be one more competing framework. Giving it the Apple stamp of approval meant that "this is the future." It guaranteed the massive adoption its already had despite being a new framework.

3 Likes

I'd love to see the point of view of people from the Linux Foundation team. Is there any indication that they are aware of this discussion?

2 Likes

Well, as a Linux and Windows Foundation developer/user, I am in favour of this as long as there is a concerted effort to ensure that the APIs are consistent and correct across Linux and Windows.

7 Likes

Ping: @tkremenek @Tony_Parker @Philippe_Hausler @itaiferber

While it is all very nice and interesting to consider which future processes we could adopt, we still don't have official confirmation that Foundation has any process right now. A fundamental data-type was entirely redesigned without any formal review and, so far, nobody knows whether that was an error or not.

What's more is that we have even discovered that some functionality has been lost in the transition. Was this intentional?

2 Likes

Hi all,

Thanks for starting this discussion. I think it gives us a great opportunity to bring more clarity to our process. Let's start with a clear definition of the projects we're talking about here.

Foundation

Foundation is a framework and Swift "overlay" that ships with each version of macOS, iOS, watchOS, and tvOS (sometimes collectively referred to as "Darwin"). It is closed source, written in a variety of languages (C, Objective-C, and Swift), and uses an Apple-internal process for evolution of its API and functionality. The goal of Foundation is to provide first-class API for Darwin platforms, with a focus on these goals (outlined in our README):

  • Providing basic utility types
  • Establishing consistent conventions
  • Support internationalization and localization
  • Provide a level of OS independence

As a part of the operating system, Foundation is often tasked with adding API that forms a part of a larger feature across a release. Foundation also adds other new API requested by developers (both internal and external) after the team has agreed upon its general utility, implemented and tested the functionality, and reviewed it.

Sometimes, features that Apple wants to add to Foundation will require changes to the Swift compiler or standard library. In those cases, the Foundation team at Apple will run a Swift Evolution proposal. Examples include SE-0161 Smart KeyPaths, SE-0166 Swift Archival & Serialization, SE-0167 Swift Encoders, and SE-0170 NSNumber bridging.

swift-corelibs-foundation

swift-corelibs-foundation is an open-source library created concurrently with the release of the Swift project. Its goal is to bring API parity with Foundation to non-Darwin platforms. These "core" APIs (along with XCTest and libdispatch) exist to support a broad level of common functionality that developers can rely upon for all Swift development across platforms.

A large subset of Foundation's implementation has actually been open source for a long time. This was imported into swift-corelibs-foundation to form a common base of implementation. This library is called CoreFoundation ("CF"). Over the past few years, Apple has open sourced some key parts of Foundation by moving them into CF. For example: CFURLComponents and CFCalendar Enumeration.

Thanks to an amazing level of effort from the open source community (highlighted above in this thread), an incredible amount of progress towards realizing this goal has been made.

The Intersection

These two projects have different goals, but of course they are also inexorably linked with each other.

  • Some implementation can be shared, as detailed above. The Foundation team at Apple is hoping to do more of this over time, and Swift reaching ABI stability will improve our ability to do so.
  • Most of the team that works on Foundation is also involved with swift-corelibs-foundation. This includes code review, rewriting Objective-C in Swift or C so it is more portable, and writing completely new implementations.
  • API added to Foundation should also be added to swift-corelibs-foundation, to maintain parity.

It is not a goal of swift-corelibs-foundation to add API to Foundation. However, the Foundation team does recognize the opportunity that Swift provides to take advantage of new language features to improve our interfaces. As these changes are made to Foundation, swift-corelibs-foundation also benefits. Examples include improved Scanner API and FileHandle API.

swift-corelibs-foundation is a hybrid open source project. It is clearly related to Foundation, but it does not bring all of Foundation to open source. The Foundation team welcomes suggestions for improvements to our APIs for Swift, including those in swift-corelibs-foundation. In addition, the Foundation team will continue to drive improvements to both projects on its own. As a key part of the operating systems and the Swift project, both Foundation and swift-corelibs-foundation are expected to be well supported and maintained alongside Swift itself.

Thanks,

Tony

18 Likes

I think Tony covered the primary aspects of the thread here so I won't add to that — I did, however, want to address some technical concerns here related to Data specifically:

While designing this API, we worked closely with the standard library team to discuss placement for this protocol (along with DataProtocol itself). I personally think it would equally appropriately live in the stdlib, but at the time this was not desired. In the future, if we decide that the better home for this is indeed in the standard library, we will explore mechanisms for sinking it down in an ABI-compatible way. (e.g. symbol aliasing and replacement with a typealias on the platforms which are relevant)

This was a decision that was closely considered and it was decided that Foundation is currently the most appropriate home for this protocol.

This aspect was long discussed and debated internally, and was one big driving force for the change in implementation here. Specifically, the fact that NSData and dispatch_data_t are toll-free bridgeable (unidirectionally) is purposefully not documented, and the fact that Data could wrap such a discontiguous data instance was often lost given that many method accesses (touching bytes anywhere) would implicitly and silently flatten the data anywhere.

After much debate, it was decided that it would be better to explicitly promise contiguity (which has allowed us to both simplify the implementation in several areas, and achieve much improved performance) for Data, with the added benefit of introducing API (DataProtocol) which would abstract over various types by allowing discontiguous access to the data. The intention here was to allow generic abstraction over both Data and DispatchData in a way that publicly enables discontiguous access rather than making this an implicit, undocumented fact, only if you don't look at the Data instance too funny.

The hope is that by leveraging DataProtocol where possible, we can help promote usage of DispatchData too to make discontiguous data instances more widely and easily accessible.


Along with this explanation, I'd like to personally take responsibility for doing such a poor job at communicating these changes to the community. The APIs here went through months (if not collectively years) of review and discussion internally, and I would have liked to express that better at the time of introduction, in spite of some internal constraints at the time.

I promise to you that we did not "forget" about our dedication to the community, though I completely understand the frustration; I'm saddened that the process here has caused you to feel that it is "amateurish", but am keenly devoted to improving this in the future.

18 Likes

Thank you for this writeup @Tony_Parker.

So to be clear: let's say I primarily use Swift on Linux, for my server backend or whatever. Now I want to propose some new API or make some changes to what comes bundled with the toolchain. If the API is to live in the standard library, I may pitch/implement it and participate in its review. However, changes to corelibs-foundation's API may only happen if they reflect a corresponding change to Foundation, therefore I should file a radar and let it go through Apple's opaque, internal process?

If that is the case - well, it is what it is, but I'm not sure that's obvious to everybody.


Well, it is actually documented. I actually agree that Data can't realistically provide a contiguous and non-contiguous API at the same time, but it would have been good to have some rationale and design discussion all the same. It's mostly just a big shock to learn that Apple continues to reserve the right to make such big changes unilaterally. Again, it is what it is, but I feel it undermines the idea that Foundation is some kind of extension of the standard library.

1 Like

Thank you for your explanation, @itaiferber. To be clear, I'm ok with the idea that Data represents contiguous bytes if there is a performance benefit to doing so. My real issue here is threefold:

  1. AFAIK there was no RFC to discuss the behavioral change here. As this is a framework type, you're certainly within your rights to not have an RFC, but in the future for significant changes like this you may consider at least writing up an already-accepted RFC and publishing it so the community can read about the change and understand the motivation.
  2. The new behavior of Data is undocumented. The overview doesn't even mention that it's contiguous storage, the enumerateBytes deprecation notice just says to use regions, and the regions property is undocumented.
  3. DataProtocol is also completely undocumented, without even an overview (based on Apple's docs; I see there are some minimal doc comments, but still no overview, and these comments didn't make it to the published documentation). With zero documentation, and zero mention on the Data type, nobody's going to change their code to use DataProtocol instead of Data. In its current state it feels like an implementation detail rather than something I'm expected to use.
4 Likes

As been mentioned many times in the forums an additional experimental library like c++ boost, would probably be the best approach here, so that the best parts of it would eventually be sherlocked into Foundation.

As long as we don’t have the swift community working towards single boost-like library, then it’s up to individuals doing their own github repos and promoting them on these forums. I don’t think it’s primarily Apple’s role to drive this.

Hi @Karl,

We are actually in the fortunate position that the same people that work on the Objective-C framework at Apple are also highly involved with the open source project. This includes new code contributions, code review approvals and comments on GitHub, forum posts and more. We are all very interested in moving both projects forward and are very receptive to ideas from the community. We can discuss those ideas here. When there is overlap with open source work and new API (as there has been for things like Codable support in Foundation), there is an opportunity to participate in both the review and the implementation.

The file handle and scanner improvements I mentioned are a direct result of the work on swift-corelibs-foundation and community feedback. We have also experimented with asking for targeted API enhancements in cooperation with the community (e.g. RFP: Ordered Set), although I admit that hasn't moved forward at the pace I would like.

In part I think the concern originating this post is that it is not clear enough how Darwin's Foundation and swift-corelibs-foundation are related and managed.

Phrases like the work on swift-corelibs-foundation and community feedback and API enhancements in cooperation with the community, could transmit that Foundation is in some degree an open-design project, as Swift is, but it is not. Design is closed and done in-house (taking external feedback, of course, but as all Apple frameworks) and the community is limited to re-implementing parts of Foundation supporting other platforms and sending feedback (it is not even clear where that feedback should go — Apple Radar / Swift Bug Reporter / Forums ?).

I think it would be a lot clearer and better for the community and Apple if Foundation would be structured code-wise in two parts:

  • The core which would be the Objective-C Foundation on Darwin and swift-corelibs-foundation on other platforms. That part would remain be closed-design.
  • A public open-design common Swift Foundation overlay where the community could really propose features and extend Foundation (Swift Evolution-like). Maybe other module? Foundation+? :sweat_smile:

There are a lot of Foundation features that are needed in the overall Swift ecosystem but are a very low priority for Apple, this could solve that tension.

Example

Take Process, it is a powerful tool in Foundation but it lacks really basic things like finding an executable by name in the PATH, checking the output and status of the execution or easily extracting the process output. For comparison:

On Python:

import subprocess

key_data = subprocess.check_output(["openssl", "genrsa", "2048"])

Current Swift (no joke; minimum equivalent cross-platform implementation):

import Foundation

#if os(Windows)
let pathSeparator: Character = ";"
let defaultExecutableExtensions = ["COM", "EXE", "BAT", "CMD"]
#else
let pathSeparator: Character = ":"
let defaultExecutableExtensions = [""]
#endif

var uppercasedEnvironment: [String : String] = [:]
for (key, value) in ProcessInfo.processInfo.environment {
    uppercasedEnvironment[key.uppercased()] = value
}

var executableDirectories: [URL] = []
if let path = uppercasedEnvironment["PATH"] {
    executableDirectories = path.split(separator: pathSeparator).map { URL(fileURLWithPath: String($0)) }
}

var executableExtensions = defaultExecutableExtensions
#if os(Windows)
if let pathExtensions = uppercasedEnvironment["PATHEXT"] {
    executableExtensions = pathExtensions.split(separator: pathSeparator).map(String.init)
}
#endif

func getExecutableURL(name: String) -> URL? {
    for executableDirectory in executableDirectories {
        for executableExtension in executableExtensions {
            let executableURL = executableDirectory
                .appendingPathComponent(name)
                .appendingPathExtension(executableExtension)
            if FileManager.default.fileExists(at: executableURL) {
                return executableURL
            }
        }
    }
    return nil
}

var executableURL = getExecutableURL(name: "openssl")!

let opensslProcess = Process()
opensslProcess.executableURL = executableURL
opensslProcess.arguments = ["genrsa", "2048"]

let outputPipe = Pipe()
opensslProcess.standardOutput = outputPipe
try opensslProcess.run()
try opensslProcess.waitUntilExit()

if opensslProcess.terminationStatus != 0 {
    throw NSError(
        domain: NSCocoaErrorDomain,code: Int(self.terminationStatus)),
        userInfo: [NSLocalizedDescriptionKey: "Process terminated with failure (termination status code: \(self.terminationStatus))"])
}

let keyData = outputPipe.fileHandleForReading.readDataToEndOfFile()

Future Swift :slight_smile::

import Foundation

let opensslProcess = Process(executableName: "openssl", arguments: ["genrsa", "2048"])
let keyData = try opensslProcess.readOutput()

This hypothetical high-level Process features would be very useful in server-side Swift and when building CLI tools but understandably they are a very low-level priority for Apple.

Having this open-design part of Foundation would be an awesome scape-hatch for this types of features. And it would be the first part of Foundation to have a shared implementation between Darwin and other platforms.

5 Likes

It seems like we'd be better off putting our efforts into building a separate library that isn't encumbered by Apple internal process at all. Anything tied to Foundation will likely have mysterious delays that the Apple engineers can't talk about (sometimes for months at a time). If we want something that can be 100% community driven, we'll need to look elsewhere.

I'm not trying to disparage Foundation or the people who work hard to make it great! But if we want something Boost-like for Swift, I think that would be more successful as a separate project.

8 Likes

As a teacher, I'd like to stress the importance of moving these projects forward, preferably at a faster pace and with more community input.

A language is only as good as the tools and libraries that surround it, and at the moment, Swift is a hard sell as a cross-platform language. New learners should not have to worry about the differences between Foundation and swift-corelibs-foundation and should be offered a single, official, well-documented and cross-platform "standard" (or core) library that makes the most of Swift's features.

I'm hoping that whatever process results from these discussions will encourage the community to get more involved in the implementation and evolution of Foundation/swift-corelibs-foundation. We need better tools, libraries and platform support a lot more than we need language features!

6 Likes

IMHO we should just start that boost-like project and see where it goes.
If it attracts people and gives birth to nice things then we could define it as a requirement to any foundation swift-evolution.

It would also has the advantage that API would have been battle-grounded before being accepted.

However I would stick to the point that API created through this process should be Objective-C/CF free (and cross-platform obviously).

The reason I ask is because there are parts of Foundation's API that I would like to help improve. @pvieito has a good example with Process, but there are others. I had to use Calendar extensively in a project recently, and I had to make my own iterators and sequences so that it would compose nicely with other common Swift idioms. I considered upstreaming it, but it wasn't clear how that should happen.

I think the consensus from these discussions is that we need a new library. corelibs-foundation was billed as plugging the gaps that the purposefully-minimal standard library would not tackle, but an evolution process which happens mostly/entirely internally within a team at Apple is obviously not going to be acceptable to many.

2 Likes

So in line with Apple’s recent branding, why don’t you @Karl start a Foundation+ Framework that extends corelibs-Foundation. Do your own addons, mix it with the Process extensions. Then publish a thread here for the community to discuss the additions. Maybe with time it will get even somewhat official status in swift.org