I would push back on this and say rather that it takes them out of the swift-pm versioning domain and puts them into the toolchain versioning domain.
The fundamental tension to resolve here is that these two versioning domains are more-or-less incompatible, so we must ask which domain these APIs are more closely aligned with.
I would push back on this and say rather that it takes them out of the swift-pm versioning domain and puts them into the toolchain versioning domain.
It takes them out of the SwiftPM versioning domain and brings them into ...
... the OS version domain for Apple platforms but
... the toolchain version domain for non-Apple platforms.
This adds a lot of extra complexity, particularly because the OS version an end user runs the software on is usually out of control of the programmer. Even more vexing, the available version is defined by different things: OS version on Apple platforms vs. Swift version elsewhere.
Furthermore, it's often advantageous for a programmer to keep things in the SwiftPM versioning domain because that allows by far the fastest updates.
I’m still yet to see any concrete arguments as to why System should be in the toolchain. What exact benefits does it give to end users instead of pulling it in via SwiftPM other than needing to declare it as a dependency?
Because the downsides of slower releases, an even bigger toolchain, new features that can’t be adopted for 18 months should be a very high bar to including something in the toolchain.
My conspiracy theory as to what happened is that Apple put a dependency on System into Foundation, and now they've got layering problems...
But maybe they just want to be able to use System to implement the open-source Foundation, rather than have it reinvent all the wrappers.
Regardless, they should have been up-front with the motivation!
Module organization is mostly orthogonal to toolchain inclusion. I say mostly because including System in the toolchain actually enables us to do module reorganization that we otherwise could not do because it solves the versioning problem. Note that Swift's concept of "module" is a little different than in other languages and a big part of this is due to Swift's unique focus on supporting separate compilation.
I will note that those examples you listed, as well as nearly every other programming language that I know of, whether or not they support separate compilation the way Swift does, distributes that functionality in their equivalent of the stdlib/toolchain instead of doing so via packages.
I don't know if this point got explicitly clarified. System-in-the-toolchain or not is irrelevant to using System in implementation code. The versioning problem emerges for types that need to cross the toolchain/package versioning domains: they have to all use the same version.
The toolchain could pin a specific version of SystemPackage so that System is rev-locked with the toolchain. That would have all of the downsides and more of putting System in the toolchain without the platform parity upsides.
There is no System module available as a package currently, just SystemPackage. That is, the System module only exists on Darwin.
Another way of viewing this is that there is a void/erroneous System module on those other platforms containing nothing in it and requiring the import SystemPackage to access their system's interfaces. We are talking about, in effect, creating that module and filling it in to solve the versioning problem and improve platform parity.
See the Motivation section, notably the parts regarding toolchain versioning and platform parity. Also see @allevato's response earlier in the thread:
I'm not sure what technical problem you're describing. Cross import overlays are not layering and Apple's SDK is versioned together. That is, this is a non-issue on Apple platforms. Similarly, the versioning problem doesn't come up in pure implementation code, so that's also a non-issue for the implementation of open source Foundation.
What is an issue is that the versioning problem on non-Apple platforms leads to disparity. As for providing motivation up-front, please refer to the "Motivation" section in the initial post in this thread. Note the parts regarding toolchain versioning and platform parity.
To be fair, that doesn't have to be interpreted as a reason for System to be in the toolchain. In our case, if swift-subprocess had chosen to always depend on SystemPackage instead, we would have been fine. The problem there is that swift-subprocess exports public APIs that reference a different module on different platforms.
This situation is, I believe, an unfortunate vendor-induced problem. Apple decided to distribute System with its operating systems, making it an attractive nuisance for people who want to quickly use APIs like FilePath. And people want to use FilePath because the standard/core Swift libraries don't have better abstractions for this (every time I have to translate between file paths and URLs for Foundation, I shed a tear).
I understand the motivation to include things with the toolchain, but I wish we had a better general solution for large-scale back-deployment or a way for the toolchain/system library and the package to coexist. It would be great if we could solve the problem of "I have a system framework that returns a System.FilePath, using the symbol from the built-in version of that module, and I'm passing it through to something in my own code, which uses the package version of that API."
As someone who doesn't use System but would like to, it sounds to me like the problems described above are that parts of System on non-Darwin are stable and parts are still being added and iterated on. So maybe the solution here is to split System into two modules, a more stable module and another for APIs that are still being baked, then only include the stable one in the toolchain. That way, parts of the System package can still be added and updated as an independent package much more quickly.
Given SwiftPM's compilation model, I don't know if this would require splitting System into separate stable and unstable packages or if this could be done by separate SwiftPM "targets."
I guess this sufficiently motivates the idea FilePath should be in the set of toolchain libraries. Seems like most everyone here is on board with that though, with a general preference for putting it in the stdlib.
It doesn't make much of an argument for any of the other SystemPackage types. (Foundation traffics in some of these, but not in very interesting or limiting ways; SystemPackage could provide these APIs atop Foundation's existing APIs using a package trait, or a new SystemFoundation package could provide them).
It doesn't make any argument for any of the SystemPackage functions/methods. I think there is an argument that having access to these would benefit the open-source Foundation, but that isn't mentioned as a motivation.
Apple & Foundation have spent a lot of years trying to convince everyone to access local files via URLs, rather than via file paths. I wonder what the inclusion of FilePath into the stdlib or a blessed layer between stdlib and Foundation, does to this narrative.
This is true of FilePath but not the case for other types like ioring or machport.
I still don’t see this as a motivation for adding it to the package. We could use this argument to include CryptoKit in the toolchain because there’s Crypto that covers all platforms apart from Apple OSes. Or if we’re arguing for platform parity I’d say effort on FoundationEssentials would be a much better spend of time as that’s a real pain point.
IMO the motivation section doesn’t provide any concrete examples. The point is:
Swift is designed to have both the standard library and Foundation in the toolchain. System fits naturally alongside them, and toolchain libraries like Foundation can use System types in their API once System is in the toolchain.
This is a key point - wanting to use System types in Foundation APIs. And totally fair, but there are no actual examples of APIs that would be desired. The pitch would be much stronger if it was accompanied by a Foundation Evolution pitch with the new APIs and types that want to be added.
The crux of it (and correct me if I’m wrong) seems to be to want to use FilePath (and possibly friends) in Foundation and expose that in APIs. There is some discussion in “Sink Cross-Platform Types into Stdlib, Keep Syscall Wrappers in Package” but it doesn’t appear to address the concerns. Part of it seems to be a documentation issue/education issue in terms of letting people know about the types and different packages, which is not an exclusive problem to types from SystemPackage, it happens with Crypto, HTTP types, Foundation Essentials/Foundation mismatch.
This approach also risks fragmenting the ecosystem
It risks fracturing it even more if system is added because:
Note on Server-Side Deployments
Toolchain inclusion means getting new System API requires a toolchain update. From a server-side perspective, toolchain inclusion might appear to be a regression if the server deployment uses older toolchains while adopting newer package versions. However, this is already the case for Foundation, which has a far larger API surface. Furthermore, server-side deployments already depend on toolchain updates for security patches and new functionality. System in the toolchain does not meaningfully change this situation.
This is only true for developers of Foundation/stdlib and application users. Library developers (who are more likely to use these types than application developers) will have to wait ~18 months to be able to use any types, which will definitely fragment the ecosystem as shims will need to be added to address this.
We’ve yet to see a security issue hit the standard library, and it’s been several years since we’ve seen anything on Foundation. Is the expectation that we’ll only patch the current version or will security patches be released for older versions? I can’t remember if there have been any similar issues on the Apple platform side
The final paragraph of the pitch seems to discuss the main concerns raised in this thread:
While it might be the case that Swift is improved by moving types from one place to another (such as
FilePath), that's a separate discussion and wouldn't fix the fundamental issues this proposal addresses.
Isn’t this the exact discussion? There’s a desire to use types like FilePath in the toolchain libraries, adding those types to the standard library or Foundation would address all of the problems outlined in the pitch, allow new APIs to be provided, and allow SystemPackage to continue to evolve at the pace needed and just adopt the new libraries, under a SemVer major if necessary.
There's a lot of concepts, concerns, and topics to grapple with here. I think it might make sense to divide up some of this discussion into threads that can progress somewhat independently. Here's my attempt at teasing them apart.
There seems to be broad support for migrating the repo over to swiftlang. This will need some guidance by a steering group and is a prerequisite that can be tackled independently.
For ongoing governance, I think the Foundation Workgroup is the right fit. Any objections or other suggestions?
When types are versioned together in the toolchain, they can appear in each other's API.
For example, Data has the following:
extension Data {
init(contentsOfFile path: String)
func write(toFile path: String, ...)
init(contentsOf url: URL, ...)
func write(to url: URL, ...)
}
It's not unreasonable to also want a FilePath overload in a context where FilePath is visible:
extension Data {
init(contentsOf path: FilePath) throws
func write(to path: FilePath, ...) throws
}
and a FileDescriptor overload in a context where FileDescriptor is available:
extension Data {
init(reading fd: FileDescriptor) throws
func write(to fd: FileDescriptor, ...) throws
}
A very important and related, but potentially separate, topic is the specifics of module organization and visibility. I.e., what imports are necessary for what types to be visible where.
FilePath migrates to toolchainThere is broad support for FilePath migrating from being vended by a package to being vended by the toolchain. This would mean that FilePath is versioned with everything else in the toolchain, and thus can appear in API. There's then the question of what the import statement should be. Note that when a type is in the toolchain, it can move between modules, meaning the specific import statement can be refined over time.
FilePath is your target operating system's concept of a path, including the encoding (e.g. UCS2/WTF-16 vs bag-of-bytes). The name System.FilePath is not an unreasonable name for this type. It's "cross platform" in the sense that there might be a common API between some platforms, that is it might be more or less "portable" in the POSIX sense. It's not a "cross platform" abstract notion of location because it actually encodes a path as understood by your specific OS. It could also be Swift.FilePath, that is it is brought in from the implicit import that every Swift program has. Or it could be some other module, but we'd have to pick a module for it to land in.
FilePath is different than some other systems-level types in that it as a concept reaches higher up the tech stack than say, FileDescriptor.
Do we have general agreement that it is a non-goal to migrate only the type but none of the syntactic operations on it? That is, we're talking about migrating the both type and the useful operations on it together.
FileDescriptor and other low-level types and concernsThis thread didn't talk much about FileDescriptor specifically, but it is an example of a low-level currency type that makes sense to unify across the ecosystem. It is portable in the POSIX sense and there's real correctness value to unifying on it (e.g. closeAfter has to handle tricky edge cases).
While we can migrate FilePath as a one-off special case (because it is so cross-cutting), that doesn't help other libraries that care about file descriptors, such as Foundation and Swift Subprocess (which uses both FilePath and FileDescriptor in API).
The release cadence and versioning concerns are real pain points. I think there are tooling improvements and release process changes that can greatly address them. (I'm sketching possibilities here, not claiming technical feasibility).
When a type moves from a package to the toolchain, we could provide type-converting initializers and migration methods. If we had the means to automatically apply these for specific circumstances, that would greatly ease migration and allow for a more gradual move of types over time.
On the release management side, we could release delta packages containing changes since a given toolchain version, allowing the package to ship new functionality without waiting for the next toolchain release.
This should probably be discussed more broadly, as it's a matter that affects more than just System.
I want to see whether we have general agreement that it is an explicit non-goal for a developer to have to write multiple imports for basic functionality on a type. That is, we're not advocating for a situation where developers have to workaround our not being able to solve the versioning and release problem. Developers shouldn't have to write:
import SystemTypes // Give me FileDescriptor as a currency type
import SystemFunctions // Now give me methods on FileDescriptor
The question is more about whether and how to keep certain functionality on the vended-by-package side of the fence. io_uring is a very clear candidate for this kind of treatment, but any new syscalls added could be handled similarly. It gets complicated when those involve new types and this clearly could benefit from better tooling.
Another important concern is how to address the problem of documentation and discoverability of what is platform-specific and what is not. This is a broader problem and there's a need for solutions other than leaning on module names to organize this. Modules carry many concerns beyond communicating platform availability, and I don't think they're the right tool for this particular problem.
"Cross-platform" can also quickly get nuanced when System is involved, as it's more about groups of platforms over which API may be supported or "portable" (in the POSIX sense). That is, an API might be cross platform if your platforms are Darwin, Linux, and FreeBSD (and others) but not for Windows. Note that even FilePath being available requires that your target has a file system in the first place, which might not be the case.
This seems like a larger and broader, and potentially separate, discussion.
I can understand Foundation and aren't averse to it, but I think Platforms would also have input if this is becoming more of a foundational module for systems access.
Yep definitely agree with this, Foundation makes sense as a Workgroup, though the PSG or ESG could have equal argument over it.
Just to clarify - this would be a breaking change correct?
Yes I would assume if the types were to move, the relevant APIs would move as well
I think both FilePath and FileDescriptor both make sense to migrate
I broadly agree with this for basic functionality. There is some precedent in the toolchain for requiring multiple imports (FoundationEssentials vs Foundation) but generally I agree that basic functionality should live in the same import.
Yeah there is a lot to be done here for the toolchain, but definitely out of scope for this pitch
My understanding of Swift's compilation model (which is a couple years out of date) is that there is no technical reason why this would be a breaking change, so long as it occurs between modules that are versioned together. There is some internal tooling for this purpose (@_originallyDefinedIn), but I'm not familiar with their current limitations.
I take this to mean that in addition to FileDescriptor, we'd migrate FileDescriptor.open(), close(), read(), write(), etc. Would it make sense (assuming the timing aligns) to migrate the span-family methods that are being pitched as well?
Let's pull on this yarn a little bit and think through a hypothetical. Let's say we didn't migrate FileDescriptor.SeekOrigin and corresponding seek methods because we determined that SeekOrigin didn't carry enough weight as a currency type for inclusion in the toolchain.
In this scenario, I'm writing a low-level file compression library. It's meant to sit directly on top of the underlying OS, compressing bytes on write and decompressing on read. I absolutely want to use FilePath and FileDescriptor in my API. I want to similarly provide read/write and parallel read/write interfaces along the lines of System's.
Everything's going fine and I have a successful 1.0. Then, I realize that I need to expose my own wrapper around seek that is phrased in terms of positions or offsets into uncompressed data. That is, logical data offsets (block-aligned, let's say) rather than offsets into the compressed physical storage.
extension CompressedFileDescriptor {
/// Repositions the read/write offset in terms of uncompressed content bytes.
/// The library translates this to the corresponding position in the compressed file.
///
/// The corresponding operation on the underlying file descriptor uses `lseek`.
func seek(
toUncompressed offset: Int64,
from origin: FileDescriptor.SeekOrigin
) throws -> Int64
But FileDescriptor.SeekOrigin didn't make the cut for the toolchain and so my library has to do the SystemPackage dance and users of my library are similarly dancing around me.
The boundary between "currency type" and "the extra junk that comes with it" isn't as clear and clean in System. And that makes sense, as most operating system interfaces have had decades to accrue complexity and special cases. System's role is to provide Swift interfaces such that using these complex interfaces requires wrestling with the complexity inherent to those interfaces and concepts, without simultaneously wrangling with C's weak typing, unsafe pointers everywhere, bit mask twiddling, lack of namespaces, etc., on top of those concerns.
Declarations can move between modules in the toolchain in both a source stable and ABI stable manner using a combination of re-exports and use of the @_originallyDefinedIn attribute. Source stability is preserved by introducing an @_exported import of the new module to the old module, ensuring that clients of the old module import the new module implicitly. @_originallyDefinedIn preserves the mangling of the symbols and also leaves metadata on the symbols which ld/dyld use to ensure that the symbol is expected in the right dylib at launch depending on which OS version the binary is running on. And finally the old dylib needs to be configured to re-export the new dylib so that clients do not need to be explicitly configured to link the new dylib.
Practically speaking the only direction declarations can move, though, is down. The new home of a moved declaration should either be an existing, lower level module or a new module which is layered below the previous owner. The new module approach is usually the wisest, since the @_exported import that is necessary does have source compatibility concerns if it means that an existing module might be implicitly imported in more clients where conflicts might arise.
Having read through the pitch thread and thought a lot about it, I have a few comments:
I think sinking FilePath into the standard library makes a lot of sense, as it's really just a special kind of string type that handles pathnames.
I'm less sold on FileDescriptor, if only because POSIX-style file descriptors aren't really a thing on Windows*. The equivalent on Windows is a HANDLE, and the set of operations on those is similar but different.
I think I would prefer to see an io package that provided a proper platform-independent I/O layer, ideally for both synchronous and async I/O.
I agree with @FranzBusch and @johannesweiss that a key concern here is the ability to quickly iterate on things.
As I've expressed privately to @Michael_Ilseman, I think the System package is a bit of an oddity and I think ideally it shouldn't exist. What do I mean by that? In my mind, the ideal would be that we had:
print and input) doesn't do things like I/O.async low-level I/O with a Swift interface and good performance.Darwin or WinSDK that expose platform-specific functionality. Most programs ideally should not need to include these; right now we're pulling these in all the time.Foundation, which provides a rich, high-level system-agnostic API.If we had that, I don't think we'd need System at all?
* On Windows, the C library you're using typically provides an emulation of some of the POSIX file descriptor API, albeit these days with underscore prefixes on the majority of the function names. It is quite an incomplete emulation, mind, and additionally because it's provided by the C library, and because there can be multiple C libraries involved in any given process, you can't safely pass file descriptors across a module boundary.
** Today, it can be hard to use C library APIs from Swift in a cross-platform application because of differences in the underlying declarations, in particular nullability and the use of forward-declared `struct` types can result in one C library having visible differences in Swift where there aren't any in C.
I agree, and I'd be strongly opposed to sinking FileDescriptor. Some sort of FileStream that exposes access to an underlying C FILE * pointer might be reasonable as that, at least, is nominally portable.