What’s next for Foundation

[Tony_Parker]

We have some exciting news to share about the future of Foundation. As announced at ServerSide.swift and on the Swift blog we are embarking on a project to create a unified implementation of Foundation, written in Swift, for all platforms.

An important part of our vision is to provide the option of smaller, more granular modules for server side apps.

I'd like to kick off some discussion about the project by walking through a breakdown of which types should appear in which modules.

Module Breakdown

Below is a preliminary proposal of where we think the best separation lines are. This is not a final list -- we are seeking community input to the process.

FoundationEssentials

These types are useful in most applications and have no additional system dependencies. This package may choose to rely on key Swift packages like Collections or Algorithms, but each new dependency will be added only after careful consideration of the tradeoff to the overall size of Essentials. This module contains types which advance the overall mission of Foundation as a library for providing basic utility classes, setting precedent for design patterns, and providing a level of OS independence to enhance portability.

• URL
• Data
• UUID
• Date, DateInterval
• PropertyListSerialization, JSONSerialization
• PropertyListDecoder, JSONDecoder and encoders
• NotificationCenter
• AttributedString
• SortDescriptor
• Measurement, Dimension, Unit
• ProcessInfo
• UserDefaults (scope may be limited, e.g. to certain kinds of domains)
• FileManager, FileHandle, Process, Pipe
• Bundle

Some of these API deserve a new look due to advances in the language since they were introduced. For example, we think API like Process should adopt async/await. By including them in the Essentials package, we hope we can work with the community to advance the API together. In the short term, the existing API is still useful for many projects that depend on it.

FoundationInternationalization

These types are useful for working with dates and times or formatting data for user presentation.

• FormatStyle protocol and all of concrete format style types
• Locale, Calendar, TimeZone, DateComponents
• Locale-specific String extensions
• CharacterSet (This API may be redesigned or augmented in the future to better fit with Swift String)
• URLResource
• LocalizedError
• Morphology

FoundationNetworking

The FoundationNetworking module has already been split off from Foundation. It will continue to provide the same networking API. Unifying this implementation in Swift is a highly desired future direction after we stabilize the essential types, especially URL.

• URLSession, URLRequest, URLResponse, other associated types
• HTTPCookie, HTTPURLResponse, other associated types

FoundationXML

The FoundationXML module has already been split off from Foundation. It will continue to provide the same XML parsing API. Unifying this implementation in Swift is a future direction after we stabilize the essential types and FoundationNetworking.

• XMLDocument
• XMLDTD
• XMLDTDNode
• XMLElement
• XMLNode
• XMLParser

FoundationObjCCompatibility

These types are useful for cross-compilation with Darwin Foundation or legacy code.

• NSObject
• NSValue, NSNumber
• NSError
• NSNull
• Geometry types, NS/CGRect, NS/CGPoint, NSEdgeInsets, etc.

Micromodules vs Monoliths

A frequently asked question is: why don't we split each type in Foundation into a separate package, so they can be imported independently? We believe the right answer is a balance between individual types in independent modules and one large module with everything.

In a system with each component as a separate module, the number of relationships between the modules can grow rapidly. Each interface between the modules must be stable, and public. This can inadvertently tie our hands for future improvements to the API surface as a whole, when we find that an interface we thought was just for a "friend" module is actually used elsewhere.

Instead, we choose to divide the modules by external dependencies. External dependencies are typically where significant binary size growth comes from, and lack of control over the transitive dependencies can cause conflicts for downstream clients.

Removed types

On Darwin platforms, we are required to maintain compatibility for all existing API surfaces. However, we choose to focus our new unified implementation on the most useful Swift APIs. This is a shift from swift-corelibs-foundation's goal of 100% source compatibility. Many of Foundation's features have been subsumed by direct support in the language. These types are currently not planned to be brought forward into the new package:

• RunLoop, Lock, OperationQueue, Stream, Port, Timer etc. - replaced by structured concurrency
• NS-prefix collection types - originally provided for compatibility, but utility has been very small
• NSCoding, NSKeyedArchiver - replaced by Codable.
• Progress - No external dependencies, but intersection with structured concurrency has not yet been fully designed.

On Darwin, the Foundation framework will continue to maintain implementations for these types in a combination of C, Objective-C and Swift.

Future discussions

We hope this is just the start of discussions about this exciting new project. In future posts, we want to discuss plans for the existing swift-corelibs-foundation project and figure out how we transition the ecosystem to the new unified implementation.

[davdroman (David Roman)]

I'm beyond excited for this.

The future of Swift looks more exciting and bright than ever.

All I can say is "bring it on", and looking forward to contributing.

[Jon_Shier]

There's no real structured concurrency replacement for Lock, Stream, or Port, so are there plans to introduce new capabilities in Foundation, or shift things down to the standard library?

Codable isn't equivalent to NSCoding, is there a plan for enhancements here? Will we be getting Codable improvements to remove its various bottlenecks?

What about URLProtocol? This is a rather archaic, if powerful, type I'd be happy to replace with something better but sad to lose its capabilities.

What about background URLSession support? Will that remain an Apple-internal capability?

Does this give us the opportunity to rebuild URLSession using common types that can be shared among the community (HTTPHeader, etc.)?

Can this version of Foundation have external dependencies on non Apple frameworks, even while it uses swift-algorithms or other Apple-authored dependencies?

[bjhomer]

This is very exciting! I'm thrilled to hear about it!

I'm a little unclear on how this new project relates to the Darwin implementation of Foundation. For example:

@Tony_Parker

FoundationInternationalization

• CharacterSet (This API may be redesigned or augmented in the future to better fit with Swift String)

Will the cross-platform implementation only have the newer API, but the Darwin implementation of CharacterSet have both the old and new APIs?

@Tony_Parker

On Darwin, the Foundation framework will continue to maintain implementations for these types in a combination of C, Objective-C and Swift.

Does this mean that the on Darwin, most of Foundation will be the cross-platform implementation, but it will be augmented by additional code that implements these removed types? And thus the existing implementations of the "useful parts" of Foundation will be migrated to Swift?

[Morten_Bek_Ditlevsen]

This is so exciting! As I read it, changes to Foundation will then be a part of these forums with similar possibilities to create pitches and proposals as there is with the evolution of Swift itself, right?
Is there then a (public) Foundation core team?

I look so much forward to getting another shot at proposing key coding strategy changes to JSONEncoder and JSONDecoder!

[xwu (Xiaodi Wu)]

Any announcement on where Decimal will live?

[hassila (Joakim Hassila)]

Just wanted to post a short positive note as someone who explicitly have avoided Foundation (for all the known reasons) - this looks like an excellent pragmatic approach and structuring modules based on possible external dependencies makes good sense.

For our use cases Essentials and Internationalization will cover all we want/need and having a native Swift implementation that is feature-for-feature (or bug-for-bug ;-)) compatible cross platform is very good news - much looking forward to see this unfold and to try it out soon and try to contribute as we can - and to reevaluate our (non-) usage of FoundationXXX.

[iMostfa (Mostfa Essam)]

i'm very excited for this !
thanks team for sharing, will wait for announcement on Github.

[Jon_Shier]

It does seem like FoundationEssentials could use another separation between common currency
types like URL and Data, and high level types like NotificationCenter, UserDefaults, Bundle, and FileManager. FoundationCore vs. FoundationSystems?

[Jon_Shier]

Also, since this is a rewrite, is there room for wholesale replacement of things like FileManager and FileHandle with modern Swift types, or do such things have to match the old naming even if the API doesn't match?

[Alejandro_Martinez]

Huge news! This is great!

A couple of questions that come to mind,

• Does this mean that eventually Darwin Foundation will just become this new Swift Foundation? (except on the types that are no longer part of this new world)
  In other words, will iOS/macOS apps use this new Swift Foundation? and if so, will still be provided by the OS or would we be able to import it (and update it!) like any other spm?

• How is the evolution process for these packages look like? Are they part of SE?

@Tony_Parker

FoundationEssentials [...] This package may choose to rely on key Swift packages like Collections or Algorithms, but each new dependency will be added only after careful consideration of the tradeoff to the overall size of Essentials.

When reading this I'm reminded of some useful functioanlity that Apple frameworks have where they extend each other only when both are imported (MapKit + SwiftUI). It would seem useful to have that for this where Essentials would get collection extensions when importing that other library. Is this something the language (or SPM) would be able to support eventually? Is very useful for library design and modularization.

[xwu (Xiaodi Wu)]

@Alejandro_Martinez

Is this something the language (or SPM) would be able to support eventually? Is very useful for library design and modularization.

This is the already pitched cross-import overlays:

Cross-import overlays

This is a follow-up to the previous pitch by @typesanitizer. Cross-import overlays allow Swift to automatically import additional “overlay” modules based on the combination of imports in a particular source file. They allow one library or framework to seamlessly offer tailored APIs for interoperating with another, without imposing additional dependencies or code on clients who don’t need it. This feature has now been implemented and is in master and 5.3 nightly compilers, hidden behind the -…

[Sajjon (Alexander Cyon)]

Fantastic news! Will each type get more or less identical APIs as Foundation at first? Or will some types get new APIs?

Especially will Data get upgraded with hex conversion? I’ve been using Swift Crypto’s hex utils for this in like 25 repos…

[David_Smith]

@Sajjon

Or will some types get new APIs?

From the OP:

Some of these API deserve a new look due to advances in the language since they were introduced. For example, we think API like Process should adopt async/await. By including them in the Essentials package, we hope we can work with the community to advance the API together. In the short term, the existing API is still useful for many projects that depend on it.

[OneSadCookie (Keith Bauer)]

Love this. Platform inconsistencies make Foundation much less useful than it could be, and structured concurrency has made it abundantly clear how unfit for purpose most of this stuff is.

My concern is around fixing some of the archaic poor API design decisions. For example, I think we can all agree that URL should be a simple value type, but today it contains numerous features for fetching and caching the data available at the URL. Or ProcessInfo doesn’t actually offer “info about any process” but rather a hodgepodge of “info about this process”, “static info about this machine”, “dynamic info about the machine”, and perhaps most strangely, “APIs to control the execution and scheduling of this process”. The whole organization of it could use a rethink!

Will this change include a careful assessment of the current API designs, or otherwise provide more opportunity to rethink them?

I’m particularly concerned about cases where the existing API designs preclude sendability (eg due to open classes), but there’s a strong need for the type to be Sendable.

[filip-sakel]

What I found really difficult when contributing to Foundation was how entrenched it was into the compiler-build pipeline. Are there any plans to separate Foundation into a more modular, Swift package, while still bundling it with the toolchain? This would also pave the way for the standard library to be modified independently of the actual compiler.

[millenomi (Aura Lily Vulcano)]

@filip-sakel

Are there any plans to separate Foundation into a more modular, Swift package, while still bundling it with the toolchain?

I want to highlight this from the blog post:

A Swift implementation, developed as a package, also makes it easier for Swift developers to inspect, understand, and contribute code.

The intent is indeed to develop as a Swift package.

[sspringer (Stefan Springer)]

Great.

I would hope that this brings more involvement from the community and therefore makes some things easier to use / fill some “gaps”. I keep my own “Utilities” around as do many others I think, and I hope that this will become mostly unnecessary. (Oh, one of the parts of my utilities is the “AutoreleasepoolShim” that I borrowed from an Apple project, I guess the use of autoreleasepool will not be necessary any more when using the new libraries.)

I would then like to do testing on Windows when work begins, as Windows can be a special beast on its own.

[dlbuckley (Dale Buckley)]

Firstly this is great to hear. I’m glad Foundation is getting some modern Swift love out in the open. Although it does raise some questions:

As we move to a more ‘smaller but targeted’ package ecosystem - will SPM be getting module caching support? Some of us already have significant time penalties on our CI infrastructure due to dependencies having to be compiled each and every time. In our case we have a heavily modularised app and have invested heavily in work arounds which include cache invalidation etc so we only compile and test the bits we need to. There are open projects like https://tuist.io which are also trying to solve this issue. As we move to having more dependencies this is going to become an issue for more and more people.

It’s great that some APIs are going to have a rethink, I think that is the right thing to do but it does raise the question of adoption for large codebase which can’t move across all at once. I guess the first question is the new package going to be called ‘Foundation’ or ‘FoundationEssentials’ ? If it’s the former then is there going to be a support layer which mimics the older APIs to bridge the adoption gap? The core of my question is ‘what does adoption look like?’ for this new package. Also what is the minimum version of Swift this will target, or will it target the latest and greatest version from the get-go?

Finally; are there any rough timelines for this? It’s obviously not a small undertaking as a lot of APIs will go through a public evolution stage, but an indication of what the community should expect helps us understand how much we need to contribute (whether it’s evolution discussions or code) and when.

[Joannis_Orlandos]

With this news in mind. I think we need to seriously talk about Data.