Changes to Foundation must go through swift-evolution

Thank you for your explanation, @itaiferber. To be clear, I'm ok with the idea that Data represents contiguous bytes if there is a performance benefit to doing so. My real issue here is threefold:

  1. AFAIK there was no RFC to discuss the behavioral change here. As this is a framework type, you're certainly within your rights to not have an RFC, but in the future for significant changes like this you may consider at least writing up an already-accepted RFC and publishing it so the community can read about the change and understand the motivation.
  2. The new behavior of Data is undocumented. The overview doesn't even mention that it's contiguous storage, the enumerateBytes deprecation notice just says to use regions, and the regions property is undocumented.
  3. DataProtocol is also completely undocumented, without even an overview (based on Apple's docs; I see there are some minimal doc comments, but still no overview, and these comments didn't make it to the published documentation). With zero documentation, and zero mention on the Data type, nobody's going to change their code to use DataProtocol instead of Data. In its current state it feels like an implementation detail rather than something I'm expected to use.
4 Likes

As been mentioned many times in the forums an additional experimental library like c++ boost, would probably be the best approach here, so that the best parts of it would eventually be sherlocked into Foundation.

As long as we don’t have the swift community working towards single boost-like library, then it’s up to individuals doing their own github repos and promoting them on these forums. I don’t think it’s primarily Apple’s role to drive this.

Hi @Karl,

We are actually in the fortunate position that the same people that work on the Objective-C framework at Apple are also highly involved with the open source project. This includes new code contributions, code review approvals and comments on GitHub, forum posts and more. We are all very interested in moving both projects forward and are very receptive to ideas from the community. We can discuss those ideas here. When there is overlap with open source work and new API (as there has been for things like Codable support in Foundation), there is an opportunity to participate in both the review and the implementation.

The file handle and scanner improvements I mentioned are a direct result of the work on swift-corelibs-foundation and community feedback. We have also experimented with asking for targeted API enhancements in cooperation with the community (e.g. RFP: Ordered Set), although I admit that hasn't moved forward at the pace I would like.

In part I think the concern originating this post is that it is not clear enough how Darwin's Foundation and swift-corelibs-foundation are related and managed.

Phrases like the work on swift-corelibs-foundation and community feedback and API enhancements in cooperation with the community, could transmit that Foundation is in some degree an open-design project, as Swift is, but it is not. Design is closed and done in-house (taking external feedback, of course, but as all Apple frameworks) and the community is limited to re-implementing parts of Foundation supporting other platforms and sending feedback (it is not even clear where that feedback should go — Apple Radar / Swift Bug Reporter / Forums ?).

I think it would be a lot clearer and better for the community and Apple if Foundation would be structured code-wise in two parts:

  • The core which would be the Objective-C Foundation on Darwin and swift-corelibs-foundation on other platforms. That part would remain be closed-design.
  • A public open-design common Swift Foundation overlay where the community could really propose features and extend Foundation (Swift Evolution-like). Maybe other module? Foundation+? :sweat_smile:

There are a lot of Foundation features that are needed in the overall Swift ecosystem but are a very low priority for Apple, this could solve that tension.

Example

Take Process, it is a powerful tool in Foundation but it lacks really basic things like finding an executable by name in the PATH, checking the output and status of the execution or easily extracting the process output. For comparison:

On Python:

import subprocess

key_data = subprocess.check_output(["openssl", "genrsa", "2048"])

Current Swift (no joke; minimum equivalent cross-platform implementation):

import Foundation

#if os(Windows)
let pathSeparator: Character = ";"
let defaultExecutableExtensions = ["COM", "EXE", "BAT", "CMD"]
#else
let pathSeparator: Character = ":"
let defaultExecutableExtensions = [""]
#endif

var uppercasedEnvironment: [String : String] = [:]
for (key, value) in ProcessInfo.processInfo.environment {
    uppercasedEnvironment[key.uppercased()] = value
}

var executableDirectories: [URL] = []
if let path = uppercasedEnvironment["PATH"] {
    executableDirectories = path.split(separator: pathSeparator).map { URL(fileURLWithPath: String($0)) }
}

var executableExtensions = defaultExecutableExtensions
#if os(Windows)
if let pathExtensions = uppercasedEnvironment["PATHEXT"] {
    executableExtensions = pathExtensions.split(separator: pathSeparator).map(String.init)
}
#endif

func getExecutableURL(name: String) -> URL? {
    for executableDirectory in executableDirectories {
        for executableExtension in executableExtensions {
            let executableURL = executableDirectory
                .appendingPathComponent(name)
                .appendingPathExtension(executableExtension)
            if FileManager.default.fileExists(at: executableURL) {
                return executableURL
            }
        }
    }
    return nil
}

var executableURL = getExecutableURL(name: "openssl")!

let opensslProcess = Process()
opensslProcess.executableURL = executableURL
opensslProcess.arguments = ["genrsa", "2048"]

let outputPipe = Pipe()
opensslProcess.standardOutput = outputPipe
try opensslProcess.run()
try opensslProcess.waitUntilExit()

if opensslProcess.terminationStatus != 0 {
    throw NSError(
        domain: NSCocoaErrorDomain,code: Int(self.terminationStatus)),
        userInfo: [NSLocalizedDescriptionKey: "Process terminated with failure (termination status code: \(self.terminationStatus))"])
}

let keyData = outputPipe.fileHandleForReading.readDataToEndOfFile()

Future Swift :slight_smile::

import Foundation

let opensslProcess = Process(executableName: "openssl", arguments: ["genrsa", "2048"])
let keyData = try opensslProcess.readOutput()

This hypothetical high-level Process features would be very useful in server-side Swift and when building CLI tools but understandably they are a very low-level priority for Apple.

Having this open-design part of Foundation would be an awesome scape-hatch for this types of features. And it would be the first part of Foundation to have a shared implementation between Darwin and other platforms.

5 Likes

It seems like we'd be better off putting our efforts into building a separate library that isn't encumbered by Apple internal process at all. Anything tied to Foundation will likely have mysterious delays that the Apple engineers can't talk about (sometimes for months at a time). If we want something that can be 100% community driven, we'll need to look elsewhere.

I'm not trying to disparage Foundation or the people who work hard to make it great! But if we want something Boost-like for Swift, I think that would be more successful as a separate project.

8 Likes

As a teacher, I'd like to stress the importance of moving these projects forward, preferably at a faster pace and with more community input.

A language is only as good as the tools and libraries that surround it, and at the moment, Swift is a hard sell as a cross-platform language. New learners should not have to worry about the differences between Foundation and swift-corelibs-foundation and should be offered a single, official, well-documented and cross-platform "standard" (or core) library that makes the most of Swift's features.

I'm hoping that whatever process results from these discussions will encourage the community to get more involved in the implementation and evolution of Foundation/swift-corelibs-foundation. We need better tools, libraries and platform support a lot more than we need language features!

6 Likes

IMHO we should just start that boost-like project and see where it goes.
If it attracts people and gives birth to nice things then we could define it as a requirement to any foundation swift-evolution.

It would also has the advantage that API would have been battle-grounded before being accepted.

However I would stick to the point that API created through this process should be Objective-C/CF free (and cross-platform obviously).

The reason I ask is because there are parts of Foundation's API that I would like to help improve. @pvieito has a good example with Process, but there are others. I had to use Calendar extensively in a project recently, and I had to make my own iterators and sequences so that it would compose nicely with other common Swift idioms. I considered upstreaming it, but it wasn't clear how that should happen.

I think the consensus from these discussions is that we need a new library. corelibs-foundation was billed as plugging the gaps that the purposefully-minimal standard library would not tackle, but an evolution process which happens mostly/entirely internally within a team at Apple is obviously not going to be acceptable to many.

2 Likes

So in line with Apple’s recent branding, why don’t you @Karl start a Foundation+ Framework that extends corelibs-Foundation. Do your own addons, mix it with the Process extensions. Then publish a thread here for the community to discuss the additions. Maybe with time it will get even somewhat official status in swift.org