Using the C library from Swift: thoughts

[Joe_Groff]

It seems to me that a common "C99" or "POSIX" module doesn't necessarily need to be fully-formed out of the gate to be incrementally useful. We could conceivably work on regularizing the Swift API for some underlying calls (API-noting and/or overlaying them so that they provide as consistent an interface as possible when used from Swift), while still letting developers import the platform-specific module to access interfaces that haven't been improved yet. Over time, the common interface would hopefully grow and reduce the need to reach underneath it. That would allow more codebases to benefit from incremental progress as it's made without the whole thing being gated on a years-long project to do an all-encompassing audit.

[FranzBusch]

I emphasize with the overall problem since I had to write a ton of those compile time checks in NIO and other server related libraries. I would love to see this getting easier but I also expect that for POSIX related APIs NIO is the right place to handle those platform differences right now.

@al45tair

About the only thing in your post where I have a slight worry is <signal.h>, not because I like it (or UNIX signals), but because I think we probably do want to have some kind of API for some of the things they get used for — Ctrl-C handling, window size change notifications and so on. But that could totally live in System or in some kind of terminal handling package and doesn't have to look particularly like signals. In fact, it'd be better if it didn't.

Just to add to that topic we do have the UnixSignals module that is providing this functionality right now. It is built on top of Dispatch but that is just an implementation detail. It is also tied with the ServiceLifecycle package but we would be happy to contribute this someplace else.

[timsneath]

I suspect perfect might be the enemy of the good here. In an ideal world of homogenous Swift code, we'd do everything through Foundation, but we live in a messy world with tons of C interop and usage cases where C code might be swapped out and gradually migrated to more idiomatic replacements. Like UnsafeMutablePointer, this kind of code exists to abstract the boundaries of the world we want to live in from the world that exists.

And so, I'd like to be able to call POSIX APIs like ioctl() and exit() directly even if other alternatives are available.

[scanon (Steve Canon)]

@timsneath

In an ideal world of homogenous Swift code, we'd do everything through Foundation

FWIW, I don't actually think this is true. Even in a perfect future world of homogenous Swift code, there are still plenty of platform-specific distinctions that we won't want to abstract over. Faithfully exposing platform semantics in Swift native bindings is also desirable (and that's why we have System), as is making available platform API that we haven't had a chance to make Swift bindings for yet.

[Joe_Groff]

@timsneath

And so, I'd like to be able to call POSIX APIs like ioctl() and exit() directly even if other alternatives are available.

Yeah, if this module existed, I would expect it to only regularize variations in C headers that tend to be minor in C but affect Swift interfaces a lot (like nullability, long vs long long typedefs, char signedness, etc.) but otherwise expose the interface to Swift as it's standardized by C or POSIX.

[jrose (Jordan Rose)]

I also once thought the System library was going to be this (not the import alias, but the full redo of the POSIX-level stuff), but last time I said something like that I was told it’s not a goal to offer the same APIs on different platforms, so

[grynspan (Jonathan Grynspan)]

I think System and C represent two complementary, but different, API surfaces. System would nominally be the libc-level-but-not-standardized bits, but C would be "if it's in the C standard, it's here."

At least, that's how I model it in my mind—I don't speak for anybody else here.

[grynspan (Jonathan Grynspan)]

@Joe_Groff

Yeah, if this module existed, I would expect it to only regularize variations in C headers that tend to be minor in C but affect Swift interfaces a lot (like nullability, long vs long long typedefs, char signedness, etc.)

char signedness bubbles up into the stdlib as CChar being platform-specific. I don't think I regularly work with any platforms where char is unsigned, but I suspect a lot of the code in our repo would fall over at the sight of it, mostly around pointer conversions needed for C strings.

[Joe_Groff]

@grynspan

char signedness bubbles up into the stdlib as CChar being platform-specific. I don't think I regularly work with any platforms where char is unsigned, but I suspect a lot of the code in our repo would fall over at the sight of it, mostly around pointer conversions needed for C strings.

One thing Swift importer and/or overlays could do is ensure that APIs are imported using the proper typedef names for size_t, off_t, time_t, etc., so that Swift code that consistently uses the matching typedefs rather than C* typedefs should work regardless of what the platform uses for those typedefs. Another step to enforcing portability could be to treat these typedefs more strongly as distinct wrapper types in Swift, like we did with nonportable typedefs like CGFloat in the Apple SDKs. In practice, though, it might be good enough to pin these typedefs to Int/Int32/Int8/etc. based on their expected range just to avoid platform churn in the actual typedef definitions in headers.

[Finagolfin]

@Finagolfin

They do not help at all with implementation code like this:

                #if canImport(Darwin)
                var w: off_t = off_t(count)
                let result: CInt = Darwin.sendfile(fd, descriptor, offset, &w, nil, 0)
                written = w
                return ssize_t(result)
                #elseif os(Linux) || os(FreeBSD) || os(Android)
                var off: off_t = offset
                #if canImport(Glibc)
                let result: ssize_t = Glibc.sendfile(descriptor, fd, &off, count)
                #elseif canImport(Musl)
                let result: ssize_t = Musl.sendfile(descriptor, fd, &off, count)
                #elseif canImport(Android)
                let result: ssize_t = Android.sendfile(descriptor, fd, &off, count)
                #endif

Hmm, I tinkered with that code in NIO a bit and apparently simply changing that last non-Darwin sendfile() repetition to let result: ssize_t = sendfile(descriptor, fd, &off, count) doesn't work because of a naming collision between some other Swift method and this C invocation, @Lukasa, can you confirm?

If so, perhaps providing an alias like platformC would allow it to be written once as ssize_t = platformC.sendfile instead, ie some more worthless repetition that could be avoided by moving forward with one of the simple pitches for now.

[lukasa (Cory Benfield)]

Yeah, that's correct, which is why even though all the Linux platforms agree on what this method is called and what its arguments are, we still have to explicitly say "no, we want the libc one".

[johannesweiss]

@Joe_Groff

Yeah, if this module existed, I would expect it to only regularize variations in C headers that tend to be minor in C but affect Swift interfaces a lot (like nullability, long vs long long typedefs, char signedness, etc.) but otherwise expose the interface to Swift as it's standardized by C or POSIX.

I support the proposal in principle but was also wondering about what @Joe_Groff touches on here: Yes, the C standard provides the type of each function with certain attributes (like restricted) in each version. But additionally there are aspects like nullability that (as per my understanding anyway) aren't part of the standard definitions. For example, I'd think fwrite wants a non-null FILE *stream and I'd assume some platform libcs have it annotated as __nonnull and others won't.

Are we proposing to ship a universal Swift version of these standard C headers which include more than the actual standard?

[al45tair (Alastair Houghton)]

@Finagolfin

moving forward with one of the simple pitches for now.

AFAICT there's no fundamental reason anyone couldn't put up a Swift package that does the necessary import dance and then people could just add it as a package dependency and import it. If it's just the import statements you care about, that would surely be a reasonable fix.

The advantage of that is that there's much less worry about changing such a thing. If we go adding a PlatformC module to Swift itself, and all we do is the "simple" conditional imports of the various platform modules, the interface it exports will be hard for us to fix in future (even though it'll be different on each platform and it'll export all kinds of random things).

[al45tair (Alastair Houghton)]

@johannesweiss

For example, I'd think fwrite wants a non-null FILE *stream and I'd assume some platform libcs have it annotated as __nonnull and others won't.

Yup. There are other fun problems too like whether or not getc() is a macro (if it is, it's a complex macro and won't be imported by Swift, so ideally we'd want to turn it into an inline function).

I don't think it's trivial writing a set of C headers that would work everywhere… I think in reality the way to do something like that would be to have Clang scan the platform C headers for definitions and then emit something Swift compatible that gives us a uniform interface.

[Karl]

I also hoped System would cover this, and minor differences like signed/unsigned char would be effectively hidden by the use of abstractions like a PlatformString type.

Also, a lot of C APIs (including POSIX, and lots of C-only platform-specific APIs) are just awful. Beyond unsafety, the limitations of C mean they have very poor discoverability. C interop is useful, but it shouldn't be the end goal for such widely used, stable interfaces.

The OS provides a lot of really useful information and abilities via C, and the goal should be to make them approachable even for novice developers. That's what I hoped System would be.

Even if there is some divergence by platform, I think it could be manageable. For low-level things like C and POSIX I hope we can hide a lot of the divergence with abstraction. As you get higher up, there are libraries such as libproc which I would like in System and are available on Mac and Linux, with some Mac-specific extensions.

[benrimmington]

C++ interop already supports:

• import CxxStdlib for the platform's default C++ standard library. (This will contain C standard library APIs.)

• #include <swift/bridging> for customization macros to annotate C++ code.

So why can't we have the same conveniences for C interop?

[Joe_Groff]

@johannesweiss

I support the proposal in principle but was also wondering about what @Joe_Groff touches on here: Yes, the C standard provides the type of each function with certain attributes (like restricted) in each version. But additionally there are aspects like nullability that (as per my understanding anyway) aren't part of the standard definitions. For example, I'd think fwrite wants a non-null FILE *stream and I'd assume some platform libcs have it annotated as __nonnull and others won't.

Are we proposing to ship a universal Swift version of these standard C headers which include more than the actual standard?

I think we'd have to figure that out. Although nullability isn't part of standard C, there is at least standards language for many C and POSIX APIs that states whether passing NULL to certain APIs is undefined or supported behavior, so that could be used as a baseline. Or we could be conservative and treat all parameters as nullable. It seems to me like the important thing would be to provide some consistency across platforms for at least a subset of these APIs.

[grynspan (Jonathan Grynspan)]

@Joe_Groff

Or we could be conservative and treat all parameters as nullable.

Naïvely, if we know some C API doesn't fare well with a null value, I would expect a Swift projection of that API to add nullability to ensure Swift callers cannot call it in an unsafe way.

For instance, fopen() expects two C strings as arguments and the C standard doesn't say either can be NULL (nor is it meaningful if either is NULL, at least under the standard.) So I would expect both to be non-optional when projected into Swift.

Other API might have platform-specific, documented behaviour when NULL is passed. I don't think we need to support that functionality in a "C standard library" module, but it might make sense to include an alternate projection of that API in the System module for it.

[compnerd (Saleem Abdulrasool)]

@grynspan

char signedness bubbles up into the stdlib as CChar being platform-specific. I don't think I regularly work with any platforms where char is unsigned, but I suspect a lot of the code in our repo would fall over at the sight of it, mostly around pointer conversions needed for C strings.

Windows uses a unsigned char as CChar.

[grynspan (Jonathan Grynspan)]

@compnerd

Windows uses a unsigned char as CChar.

You sure about that?

print(CChar.self)