Pitch: System in the toolchain

Evolution Pitches
1

Hello, I'd like to start a discussion of having the System module available in the Swift toolchain, rather than the current situation of it being available only on Darwin with SystemPackage available on other platforms. I wrote this up as a "pitch" or "proposal", but that's just the vehicle for the discussion as there's no formal pitch or proposal mechanism for toolchain inclusion (and I'm not convinced there should be one).

You can see the full gist here

Integrating Swift System into the Swift Toolchain

Introduction

This proposal requests the inclusion of Swift System in official Swift toolchain releases for all supported platforms. Swift System provides idiomatic Swift interfaces to system calls and low-level currency types such as FilePath, serving as a foundational building block for the Swift ecosystem.

Motivation

Swift System provides idiomatic Swift interfaces to system calls and low-level currency types. It is the replacement for importing C system interfaces directly.

Swift is designed to have both the standard library and Foundation in the toolchain. System fits naturally alongside them, and toolchain libraries like Foundation can use System types in their API once System is in the toolchain.

Toolchain Versioning

Components in the Swift toolchain are versioned together with each Swift release. The standard library, Foundation, and imported platform interfaces all share this property: they come with the toolchain and their APIs are tied to Swift versions. This allows toolchain components to use the exact same types in their API as the application. The toolchain's Swift.Int is the same as the application's Swift.Int, even under separate compilation.

This is not true for types defined in packages. A type from one version of a package is a distinct type from a type with the same name from another version. If a toolchain library exposes a type in its API, then user code must have the exact same version of that type. If the toolchain were to use a type from a package, then the toolchain's version has to transitively constrain all other versions, leading to whole-ecosystem rev-locking.

Swift System belongs in the toolchain, versioned with the imported C interfaces and Foundation.

Platform Parity

Currently, Swift developers face an inconsistent experience across platforms:

  • Darwin platforms: import System works out of the box via the Apple SDK
  • Linux/Windows: Developers must add Swift System as a package dependency and use import SystemPackage

This inconsistency creates confusion and friction for developers writing cross-platform code. A library that works seamlessly on macOS requires additional configuration and a different import statement on Linux. Including Swift System in the toolchain would enable import System to work uniformly across all platforms.

Consider swift-build, which must work both when built into Xcode (where the SDK provides System) and when built as a standalone SwiftPM package on Linux (where it must depend on swift-system's SystemPackage):

// Package.swift
.target(
    name: "SWBBuildService",
    dependencies: [
        "SWBBuildSystem",
        "SWBServiceCore",
        "SWBTaskExecution",
        .product(name: "SystemPackage", package: "swift-system", 
                 condition: .when(platforms: [.linux, .android, .windows])),
    ],
),

// BuildServiceEntryPoint.swift
#if canImport(System)
import System
#else
import SystemPackage
#endif

With System in the toolchain, this becomes:

import System

Deploying Updates

The current fragmentation has led some libraries to only use SystemPackage, meaning that the package's implementation is compiled into the application. Run-time updates and improvements, such as security fixes, cannot be applied without recompiling. This defeats the purpose of shared libraries and OS updates on platforms that vend the Swift toolchain as part of the OS.

Note on Server-Side Deployments

Toolchain inclusion means getting new System API requires a toolchain update. From a server-side perspective, toolchain inclusion might appear to be a regression if the server deployment uses older toolchains while adopting newer package versions. However, this is already the case for Foundation, which has a far larger API surface. Furthermore, server-side deployments already depend on toolchain updates for security patches and new functionality. System in the toolchain does not meaningfully change this situation.

Production Readiness

Swift System already ships as a source-stable and ABI-stable module on Darwin platforms, demonstrating its maturity and suitability for inclusion in official releases.

Proposed Solution

We propose that Swift System be added to official Swift toolchain releases, governed as part of the Swift project under the following structure:

Aspect Proposal
Repository Move to the swiftlang GitHub organization
License Apache 2.0 with Runtime Library Exception (current)
Governance Foundation Workgroup
Evolution Lightweight process for direct API additions, formal evolution for others

Detailed Design

What Would Be Added

The toolchain would include all current Swift System API, including platform-specific interfaces such as IORing on Linux and Mach.Port on Darwin.

Repository

Swift System is currently developed at GitHub - apple/swift-system: Low-level system calls and types for Swift.

The repository is already structured consistently with other Swift project repositories (build system, CI infrastructure, documentation). We see no obstacles to moving it to the swiftlang GitHub organization and believe this transition is straightforward.

Licensing

Swift System is already licensed under the Apache 2.0 license with Runtime Library Exception. This is the same license used by the Swift project, so no licensing changes are required.

Governance

As a library providing fundamental types used by Foundation and other core libraries, Swift System would be managed by the Foundation Workgroup.

Swift Evolution Process

We propose a tiered review process appropriate to the nature of Swift System's API:

Pull request review: Small additions that follow established System conventions can be reviewed and approved directly in the pull request by the Foundation Workgroup. This includes adding a missing syscall wrapper, a new overload to an existing API family, or rounding out coverage gaps. These are straightforward translations of existing, documented platform APIs where the Swift interface follows patterns already established in System.

Forum review: Larger additions go through public discussion on the Swift forums before approval. This includes new topic areas (e.g. sockets), new abstractions or non-trivial types (e.g. TerminalAttributes), or anything that involves novel API design rather than direct translation. These benefit from more formal review and visibility, especially since they may establish patterns that future API will follow.

Versioning

Once included in the toolchain, Swift System would follow toolchain versioning rather than independent package versioning. New API would be tied to Swift releases.

Source Compatibility

This change is additive. Existing code is unaffected.

On Linux and Windows, code currently using import SystemPackage can migrate to import System for consistency with Darwin.

Migration from SystemPackage

While code can be written identically against System and SystemPackage (they are source-compatible), they define distinct types. A SystemPackage.FileDescriptor is not the same type as System.FileDescriptor. Code that passes a FileDescriptor across a module boundary must agree on which module it comes from.

Toolchain inclusion improves this situation, as all code in the same process sees the same System.FileDescriptor.

For gradual migration, SystemPackage will provide conversions between its types and the toolchain's System types. For example:

import System
import SystemPackage

let packageFD: SystemPackage.FileDescriptor = ...
let systemFD: System.FileDescriptor = System.FileDescriptor(converting: packageFD)

See the adaptors sketch.

ABI Compatibility

Swift System is already ABI-stable on Darwin. The toolchain-included version would maintain ABI stability across all platforms.

Alternatives Considered

Continue as a Package Only

We could continue shipping Swift System only as a package, requiring non-Darwin platforms to add it as a dependency. However, this perpetuates the platform inconsistency and prevents Foundation from depending on System types in a cross-platform manner. The fragmentation between System (Darwin SDK) and SystemPackage (Swift Package) creates ongoing friction for the ecosystem.

Include Only a Subset (e.g., FilePath without io_uring)

We considered including only the cross-platform subset of Swift System, omitting platform-specific APIs like IORing (Linux) and Mach.Port (Darwin). However, this would artificially limit the library's utility for systems programming. Platform-specific APIs are a natural part of a systems library and are already handled through conditional compilation. Developers expect to access platform-native capabilities when writing systems code.

Different Governance Structure

We considered placing Swift System under the Language Workgroup or creating a dedicated Systems Workgroup. However, Swift System's primary role is providing foundational types that Foundation and other core libraries depend on, making the Foundation Workgroup the natural home for governance decisions.

Sink Cross-Platform Types into Stdlib, Keep Syscall Wrappers in Package

We could add cross-platform types like FilePath to the standard library while keeping lower level types (e.g. FilePermissions) and syscall wrappers in a package. This would be reviewed on a case by case basis and more types might get moved over time. Each movement would cause some migration for users of SystemPackage to move off of a package defined type to the toolchain defined type.

However, a type without relevant operations is awkward to use and leaves developers unsure on where to look for them. Looking for this functionality inside the toolchain would likely lead them to using the imported C interfaces and all the problems therein: extracting rawValues and losing type safety, accessing raw pointers and losing memory safety, forgetting to check for errors or interrupts, etc.

This approach also risks fragmenting the ecosystem. We are adding a type to the stdlib, but it is a package that is ultimately defining the semantics and use of the type. That is, the package "owns" the API while the stdlib owns the type definition (and ABI representation, e.g. memory layout). We may need some mechanism to communicate to developers to look for the package to actually use the type, while also communicating to package authors that a certain package should be treated as the "base" definition for a type in the stdlib and their extensions should be layered on top of that other package.

While it might be the case that Swift is improved by moving types from one place to another (such as FilePath), that's a separate discussion and wouldn't fix the fundamental issues this proposal addresses.

Acknowledgments

Swift System was originally developed at Apple with contributions from the broader Swift community.

22 Likes
2

Swift System is currently listed in the "Apple Swift Packages" collection (available in Xcode via File > Add Package Dependencies…).

https://developer.apple.com/swift/packages/collections/apple.json

I assumed that was somehow the reason why those 10 repositories hadn't already moved to the swiftlang GitHub organization.

3

I'm not familiar with all the policies around what repos migrated and when they migrated. My understanding is there's no significant reason System didn't make the move and there's no reason it couldn't move now.

4

My team writes a lot of multi-platform tools and for a while we adopted a policy of "just import SystemPackage everywhere" to simplify our code. That worked well until swift-subprocess came along and we wanted to use it as well, because it does the "use System on Apple, otherwise SystemPackage" dance and then we needed to make sure all of our FilePath usage came from the matching module.

So having this fundamental module be included with the toolchain is the right choice to me. It also leads me to wonder if other packages like swift-collections will see a similar outcome eventually (I know that Foundation uses it internally, so I guess it would only matter if Foundation wanted to vend APIs that used those collections). I feel like this is a good direction to go in general while avoiding growing a massive standard library since a lot of folks pitch something and say "it should be in the standard library" when they really just want "it should be easy to access out of the box".

The one drawback, of course, is having new features being locked behind new OS releases on Apple platforms. I know that vendor-specific releases and back-deployment are outside the scope of Swift Evolution, but if swift-system is added to the toolchain I hope efforts would be made to back-deploy as much as is feasible for those of us who still need to support OSes a few releases back.

12 Likes
5

Thanks for bringing up this pitch. I think the current problem of having two similar modules in the Darwin public SDK and in a package is confusing and has led to problems in the ecosystem. Having said that, I do think we should explore some of the alternatives more before deciding to move the module into the toolchain as-is. We should think about the resulting developer experience and the future evolution of the Swift ecosystem.

Developer experience as a toolchain module

As positioned, the proposed toolchain module provides both C syscall wrappers and currency types at the same time. This makes using the module very difficult for developers since they do not know what symbol is actually available on what platform. This has been one reason why developers are frustrated with the corelibs. The System module from the Darwin toolchain makes it very clear what is available since it is tied to a single platform and provides public documentation (System | Apple Developer Documentation). The SystemPackage module has the same discoverability problem but makes it a little bit better since developers can jump into the source from their development environments and see the compiler conditions. We could argue that this is solved by enhancing our documentation tools to expose information about what API is available on what platform; however, I would argue that even if we have that, an on-the-surface harmless import System will still lead to a potentially frustrating developer experience when writing cross-platform Swift code. I do think it is worth taking a look at how other ecosystems such as Rust have solved this problem. Rust provides cross-platform currency types in separate std lib modules such as path or net. Any OS-specific APIs are offered through the os module. This makes it very clear at the time of import that this results in non-cross-platform code. Additionally, Rust does provide raw bindings to libc outside of the standard library which is in turn used by the standard library as an implementation detail.

Mixing currency types and syscall wrappers

As an extension of the previous point from just looking at the public API of the System module, you don't know what is a currency type and what is merely a syscall wrapper. I personally would like us to consider splitting the responsibilities between one or more modules that provide currency types and a module that provides syscall wrappers, as outlined in the "Sink Cross-Platform Types into Stdlib, Keep Syscall Wrappers in Package" alternative. From what I can see, only the existing FilePath type is a clear cross-platform currency type, which we have a large need for in the ecosystem. I would argue this type should not be in a module that offers platform-specific syscall wrappers but in a different module, such as a separate Path module or potentially the Swift module itself. This is similar to what other languages do:

However, this perpetuates the platform inconsistency and prevents Foundation from depending on System types in a cross-platform manner. The fragmentation between System (Darwin SDK) and SystemPackage (Swift Package) creates ongoing friction for the ecosystem.

This alternative considered alludes to Foundation wanting to use System types in its public API. At this point in time, this is only FilePath or are there other types that Foundation would like to use?

Evolution speed of syscall wrappers

Toolchain inclusion means getting new System API requires a toolchain update. From a server-side perspective, toolchain inclusion might appear to be a regression if the server deployment uses older toolchains while adopting newer package versions. However, this is already the case for Foundation, which has a far larger API surface. Furthermore, server-side deployments already depend on toolchain updates for security patches and new functionality. System in the toolchain does not meaningfully change this situation.

One of the primary reasons for why swift-system adoption throughout the server ecosystem has been slow is that it does offer only a fraction of the libc interface, which means almost any package that needs to seriously interact with the system's libc had to write their own syscall wrappers e.g. NIO still hasn't migrated to swift-system since there is a large surface area of missing wrappers. I am fearing that moving these syscall wrappers into the toolchain and tying them to the toolchain release cadence is going to make the situation worse. It will take multiple releases to add the missing wrappers. This, together with the fact that most packages in the server ecosystem support the last 3 minor Swift releases, means that realistically this can only be adopted broadly in 2+ years at the earliest. Whereas if we would ship the syscall wrappers in a package that is released frequently, we could adopt that as soon as the missing APIs were added.

This is worse for syscall interfaces such as io_uring that are still evolving rapidly and would result in requiring server developers to update their Swift versions, which is frequently a multi-week effort to access new io_uring features instead of a simple package update.

13 Likes
6

Isn't the libc binding issue -- which is valid, mind, and something I've been thinking about recently -- orthogonal to System's? I haven't studied System very deeply, but I get the impression that the platform wrapping aims to be very rich for Swift, but not for fidelity with the interface libc provides.

7

It’d be great if we could get availability attributes that specify which platforms a specific API supports.

+1

8

Thanks for writing the pitch. I'll try to keep it short but I'm very happy to expand on all points:

  1. Making the correct FilePath accessible without #ifs: Yay, 100% necessary
  2. Mixing currency types like FilePath that everybody needs and syscall wrappers that hopefully only very few people need however is very suboptimal. People will include System (for FilePath) and they'll get all sorts of low-level stuff that they should mostly avoid.

On top of that, the major issue we had with swift-system is that it doesn't release enough and doesn't have a clear policy that allows every syscall to be brought in and released in very short order.

This has lead to a really annoying issue: I know pretty much no software that targets syscalls directly that found everything necessary in swift-system or import System. There's always something missing -- and we lack a clear policy that makes adding another syscall wrapper (e.g. renameatx_np or socket or recvmmsg) very quick, with a quick release.

Putting the syscall wrappers into the toolchain (non-Darwin) and SDK (Darwin) makes this problem a lot worse. This means that adding a syscall wrapper is significantly slowed down, especially on Darwin where you would have to wait for an OS release to bring in a little wrapper for another syscall.

My recommendation:

  • Add FilePath to the stdlib
  • Fix FilePath's really annoying and slow Codable implementation (String(decoding: JSONEncoder().encode(FilePath("/this/is/a/straightforward/path")), as: UTF8.self) becomes "{\"_storage\":{\"nullTerminatedStorage\":[47,116,104,105,115,47,105,115,47,97,47,115,116,114,97,105,103,104,116,102,111,114,119,97,114,100,47,112,97,116,104,0]}}")
  • Keep the syscall wrappers in swift-system (or a new package)
  • Make a policy for swift-system to accept & release contributions for any straightforward syscall add
7 Likes
9

One of the things I like about the Rust stdlib is that it attempts to abstract the underlying OS a little. There's a kind of implicit promise there that you're allowed to imagine a future without libc, or even without POSIX.

swift-system is like, the opposite of this. Yes, it's a pragmatic way to get at the raw functionality of current OSes, but is that something that should underpin Swift forever?

I'm completely on board with "FilePath should be in the stdlib".

I'm completely on board with "the stdlib should include robust APIs for files, directories, processes, sockets, etc."

(I'd absolutely like to discuss a future where Foundation loses its exalted position in the ecosystem and the interesting bits of it are made more swiftier & included in the stdlib)

I'm far less on board with "Let's provide as a de-facto standard library the least-abstracted most platform-specific ways of doing everything". Swift's first-class C interoperability already does pretty well at this, and swift-system was always just a package dependency away.

3 Likes
10

First of all, I think the goal of making FilePath easier to import is excellent. This has been a long-standing inconvenience for me, so I really appreciate the effort to improve this area.

Regarding the other considerations, the proposal seems to reject the approach of adding only FilePath to the standard library, but I personally think this direction makes sense. A System package that broadly wraps external system APIs does not seem well suited to the toolchain’s evolution and release cycle. If such a package were to be included in the toolchain, it would likely need to be as comprehensive as replacing SwiftNIO ’s custom system layer, and only after the need for frequent updates has largely settled.

Additionally, for many Swift users, the need to interact directly with low-level system APIs is relatively uncommon. In those cases, depending on an external package when needed seems reasonable. Compared to the problems being addressed here, the overall demand appears much smaller.

On the other hand, file paths are something many people need today, and there is currently no good way to handle them. Foundation.URL is designed to cover many schemes such as HTTP, which makes it a higher-level abstraction. As a result, it brings concerns like percent-encoding that are often undesirable when working specifically with file paths, making it awkward for that purpose.

One of the concerns mentioned in the proposal is that separating a type from its associated operations is problematic. However, I do not think this issue is resolved simply by adding the System package to the toolchain.

I had high expectations for FilePath and once tried to use it as a replacement for Foundation.URL. However, even for a very simple task like enumerating the contents of a directory, I could not achieve this directly and had to write my own adapter that went through Foundation.FileManager.

In other words, even if FilePath is included in the toolchain as part of the System package, the APIs required to actually use it in practice cannot be sufficiently provided by System alone. In the end, one still needs to depend on Foundation. From this perspective, the concern about separating the type and its operations does not seem specific to the “FilePath-only in the standard library” approach.

This does not feel like a small issue where directory enumeration was simply overlooked. Rather, it seems inherent to the fact that the System package focuses on low-level concerns, making it difficult for it to provide functionality that meets typical application-level needs.

1 Like
11

@Michael_Ilseman would you be able to expand on the reasons for wanting to include it in the toolchain? I share the concerns with many of the others in the thread about the speed of change if the package were to move to the toolchain.

As far as I read it, one of the desires was to allow other packages in the toolchain to use System but there’s nothing stopping that from happening now, in the same way that collections is used by Foundation right? Or are you wanting to expose system types in Foundation’s APIs?

Regarding platform parity, I share the pain with this. But system was never designed to have cross-platform parity and explicitly calls this out in the README. So even if the imports are fixed, that would still end up with complicated code at the call sites. (And don’t get me started on FoundationEssentials).

One other issue will be what to do when someone is currently depending on System as a package then tries to build with a toolchain that includes System? There is precedent for this, with Testing but it has complicated things like adoption in other libraries, like Swift Syntax. I think these edge cases need to be explored before a decision is made.

Given the Swift version support for Swiftlang is current + previous 2 versions, I would prefer to see the API expanded and settled before locking it to toolchain releases and the pain that will introduce. I however see no reason as to why the package can’t be sponsored for the swiftlang org, or be depended upon by Foundation.

I also agree the documentation tooling needs to be improved to expose platform support, but this is a wider issue for the ecosystem as well

2 Likes
Packaging divisions