What’s next for Foundation

[austintatious (Austin)]

@Tony_Parker

Stream, Port, Timer

Stream, Port, and Timer are inherently tied to RunLoop, which is itself inherently thread-based. Given the move away from concrete threads with the introduction of tasks in structured concurrency, it doesn't seem like these API are the right model to encourage going forward. I'm still open to discussion on that point.

It may be the case that we could use a new API for structured concurrency that adjusts the fire date like Timer and RunLoop do. I'm not sure.

Structured concurrency is still in its infancy and suffers a lot in terms of performance. Regardless of that removing well established classes for the purpose of forcing the still-in-infancy “new” way feels more ideologically motivated.

[tera]

@tera

One of the features of timer is that it it is drift free: say if I fire a timer with 1 second interval it would fire at, say:

0.02s, 1.01s, 1.999s, 3.0s, 4.001s, 4.98s, ...

Replicating this behaviour with "sleep" is doable but not quite convenient.

Forgot to mention the "leeway" functionality that optimises system power consumption by coalescing timers' firing.

Besides, using timer doesn't has a viral effect on the caller (the way async await does).

This is quite serious. Not all code can be (or afford being) async/await-able. If RunLoop concept in its whole is a no-no in the new foundation consider some simplified version of it, e.g. an explicit "make this call every now and then when you can" model that would be useful for clients outside of structured concurrency.

I'd also miss "Lock".

Locks (various flavours) and the mentioned above atomics are very important for lower level code. "Is it useful for foundation itself?" could be a good litmus test on what to include. IMHO Swift should be generic enough to implement Swift compiler itself, a new implementation of Foundation, a device driver, a realtime function, etc.

[esummers (Eric)]

The proposal is to use a flat module structure. Submodules are not needed. I assume the Foundation module would just re-export the modules like is done in other places:

@_exported import FoundationEssentials
@_exported import FoundationInternationalization
@_exported import FoundationNetworking
...etc...

[esummers (Eric)]

Hopefully there will be some modernization of FileManager. I'm a fan of moving FileManager APIs to use a FilePath struct similar to swift-system. It would be great to see FileManger drop the delegate pattern. Callbacks might be fine for the features that currently require a delegate.

[mickeyl (Dr. Mickey Lauer)]

Disclaimer: Don't let me spoil your excitement, I find this exciting as well, but I keep thinking about the miserable status quo and what that means for people who are already writing cross-platform programs.

I think we can all agree that swift-corelibs-foundation is kind of stuck. Despite dozens of valuable bug reports (among those ones opened by the Swift Server Team themselves) and even more valuable pull requests (like this one here), it doesn't look like anyone is scheduled to move forward with this repository.

This may or may not be a reason why there is this sudden movement about the future of foundation.

Although swift-corelibs-foundation does not fulfill its mission (source compatibility), many people are actively using it, because they must ­– there are just no feasible alternatives available for many of the critical parts, such as RunLoops, Streams, and more (and for the fun of it, exactly those critical parts are now scheduled to be removed…)

Let me tell you why Streams and RunLoops are critical to me: I write software that talks to OBD2 adapters for automotive diagnostics. Those adapters come in various forms with various interfaces,
e.g. WiFi, BLE, MFI, Bluetooth Classic, TTY, and what not. At the end of the day there is always a byte stream abstraction, hence Stream is the best common abstraction layer (in fact, it's a must, since on iOS, it's inevitable when you want to use L2CAP channels or an ExternalAccessory protocol).

Most of the automotive business logic is cross-platform, hence being able to run the lower part of this stack on Linux is a must for me. Hence I put a lot of work into a cross-platform streams library that makes communicating with such (and other) devices a breeze.

Now if you say Stream, you also need to say RunLoop, I don't know any other way to make use of streams. Swift Concurrency is not an option here and will not be for quite a long time ­– at least not until e.g., Apple changes their L2CAP and ExternalAccessory frameworks.

And rather than shrinking the functionality and semantics gap of swift-corelibs-foundation, it looks like you now want to "officially" abandon this. And even more, you don't even want to consider critical parts of the infrastructure in your rewrite.

If this is true, could you at least help us in providing the means to fork swift-corelibs-foundation, so that we can unblock it for !APPLE platforms?

In any case, this is going to be tough for me and it really dampens my enthusiasm over all the new development in Foundation.

[davdroman (David Roman)]

@Tony_Parker

Date itself is just a Double, really, so it can be used independently of any dependency on ICU

A bit of a side note, but I personally think this might be the perfect (and only) opportunity to rename Date to something more accurate to what it actually is — e.g. Instant.

[dhoepfl (Daniel Höpfl)]

Great to hear … but … scary.

Is there a time frame for when to expect the new implementation? I’m asking because I fear that this makes it official that there will be no bug fixes in the existing foundation from now on.

I agree 100% with @mickeyl that “it doesn't look like anyone is scheduled to move forward with this repository.“ Actually this caused a lot of frustration in me:

I am tracking a more than 6 years old bug report where I provided a fix more than two years ago that is not merged. Communication just stopped.

When I noticed this bug, that was first report one year earlier, I provided a simple, 3 lines fix. Even this simple pull request has not been merged for nearly a month now. Someone got assigned, CI was triggered, that’s it. No comment, nothing. Just ignored.

For a bug that is crashing the app.

[tera]

Hold on, are Threads also out? I hope not. Threads are essential for low level apps.

[tclementdev]

As to where things should go, I think threads should go in the standard library, not Foundation. Same goes for locks and atomics. These are the building blocks to write everything else. The fact that they are lacking from the standard library is a problem (in my opinion) and maybe is also one of the reasons why almost all of swift concurrency is actually written in C++, not Swift.

[tera]

@tclementdev

As to where things should go, I think threads should go in the standard library, not Foundation. Same goes for locks and atomics. These are the building blocks to write everything else.

OTOH putting threads / locks / atomics into a separate framework makes total sense IRT platforms that don't need those things (embedded systems or other handicapped / minimal environments). For the same reason unicode handling should also be separable, IMHO.

[tclementdev]

My reasoning was that this all needs to be available to swift concurrency which is itself already in the standard library. C++ also seems to have it all as part of its standard library. But maybe there's some clever module dependencies that can be made available right down to the stdlib, I don't know enough to give a good answer.

[tclementdev]

oh also it seems that parts of swift concurrency are already being modified to be compatible with more constrained environments (see 1, 2, 3). The same could apply to locks, atomics and threads.

[Tony_Parker]

I understand how frustrating this is. In fact, that is a big reason we've decided to take this new direction. swift-corelibs-foundation, as it exists today, is fundamentally a fork. We can't use the Objective-C code for NSXMLParser (as in your example) for Linux, and we couldn't use the Swift code on Darwin. The need to do everything twice is just not supportable.

It will take us some time to get up to unifying the XML implementations, but once we get there then this kind of bug fix should be much easier to deal with. I don't have an exact time frame yet. I think we will know more about the velocity of the project once we get through some of the bring-up tasks to get Essentials and Internationalization ready.

[zaneenders]

Hi,

I was wondering how I could help with Date, FileManager, FileHandle, Process, Pipe, and ProcessInfo if possible. I am new to open source and a CS student and was hoping I could maybe learn more about how Files, and Processes actually work well learning/ practicing more Swift. I am taking Computer Systems and a Compilers class in Spring23 (C and C++).

This week I started playing around/ using SwiftNIO as well as SwiftSystem a bit. I am wondering how these repos may or may not be involved in the new Foundation. NIO's NIOPipeBootstrap and NonBlockingFileIO look to be a smart abstractions, but I don't have enough experience to understand what downsides these might have.

Let me know how I can help/ get involved, maybe even with just the Swift NIO project? I know I have a lot to learn about Computer Science but Concurrency and Distributed Systems interest me a lot and is one reason why I'm Interested in learning more about Swift.

Thanks,
Zane

[amebahead]

I Love it :)

[rtharston (Reed Harston)]

Hey mickeyl, did you see Tony's reply regarding Foundation on Darwin platforms?

@Tony_Parker

Darwin Foundation (what we will call 'Foundation framework' to distinguish it) will be a superset of the package, because it will include all of the implementations we cannot remove or refactor as part of this work (stuff written in Objective-C or which we decide to omit from the package for some reason). Clearly we can't just yank the rug out from all existing apps on iOS or macOS. Therefore, the main task here is to refactor the code that we choose to have a common implementation into Swift and make that the core instead of C or ObjC.

I don't fully understand your use case, but I think what Tony says here means you'll be fine. Darwin platforms will have all the same old Foundation API available to them, but the bits that are common with other platforms will have a new core. The discussion about what to include in the new Foundation packages is about what goes in that cross platform core.

Also, thanks for sharing your streams library. I haven't had to deal with communicating directly with devices in Swift yet, but I have had to on other platforms before.

[bastie (Sebastian Ritter)]

Love to hear the first step of Swifts digital sovereignty (SDS) and hope this SDS go ahead to the full SDS ecosystem. For example we have not all (public) methods to work with SPM in result that git use a compression method which not directly in Swift compression framework supported.

Do not stop at Foundation!

If you are right and Swift implementation is fast as C/ObjC go ahead. And also build not an only FoundationObjC-Bridge. Let FoundationFFI come to us and ObjC, C++, Python, Java and important COBOL Bridges are the following step.

Question:
When Swift is so fast, is some reimplementation of Darwin OS the next step? Hope dies last.

[austintatious (Austin)]

It really is sad that Apple’s ObjC was not fully ported to other systems or that working current implementation of ObjC on Linux wasn’t done.

[YOCKOW]

By the way, to what extent does "written in Swift" mean?

For example, as to XML stuff;

• Current FoundationXML:
  • XML* wraps CFXML* that depends on libxml2.
• Future FoundationXML:
  • XML* will directly depend on libxml2 (or other C library).
  • Or XML stuff will be all written in Swift from scratch.

I know it's a kind of "future direction", but I'm curious about how we decide goals.

[esummers (Eric)]

Yeah, I hope language bridges in general move to use the in the works C++ interop stuff. Objective-C is a pretty ancient language, so I think bridges should start to move away from that as soon as they can.