[Out of scope] Discussion on general Darwin/GlibC module


(Alex Blewitt) #1

Although out of scope for phase 1, something that keeps cropping up in a variety of Linux/Darwin Swift scripts is the conditional inclusion of Darwin or GlibC per platform. The last point was an observation that creating a 'nice' wrapper for LibC or a cleaned up POSIX API is a non-goal:

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027621.html

I think it makes sense to have a cross platform “libc” which is an alias for darwin, glibc, or whatever, and just leave it at that.

Other proposals for a “POSIX” module have gotten bogged down because inevitably the idea comes up to make the resultant API nicer in various ways: rename creat, handle errno more nicely, make use of multiple return values, … etc. The problem with this approach is that we don’t *want* people using these layer of APIs, we want higher level Foundation-like APIs to be used.

...

I think we should formally decide that a “nice” wrapper for libc is a non-goal. There is too much that doesn’t make sense to wrap at this level - the only Swift code that should be using this is the implementation of higher level API, and such extremely narrow cases that we can live with them having to handle the problems of dealing with the raw APIs directly.

-Chris

I have created a draft for a proposal to create such a module. Comments are welcome.

Alex

···

---

# Libc module for Swift

* Proposal: [SE-NNNN](NNNN-filename.md)
* Authors: [Alex Blewitt](https://github.com/alblue)
* Review Manager: TBD
* Status: **Under discussion**

## Introduction

When running on Darwin, the base module is called `Darwin`. When running
on Linux or other operating systems, it's called `GlibC`.

This repeatedly leads to code such as:

    ````
    #if os(Linux)
      import Glibc
    #else
      import Darwin
    #endif
    ```

As the set of operating systems evolve, one of these conditional imports
needs to be updated. Instead of repeating this, make it available via a
standard `Libc` module in the base Swift library.

Swift-evolution thread: [Discussion thread topic for that proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027621.html)

## Motivation

The [set of platforms](https://github.com/apple/swift/blob/fdf6ee20e4ca1fd32482f4b7b88a97ebdda52cd2/lib/Basic/LangOptions.cpp#L26-L36)
that Swift currently runs on can be divided into two; Darwin and XNU based systems
(macOS, iOS, watchOS, tvOS), Windows, and Unix based systems
(Linux, FreeBSD, Android, PS4).

The base module on Darwin is called `Darwin`, while on Linux and
other Unix systems the base module is called `Glibc`. The base
module is typically conditionally included when working at a lower layer
than Foundation (which has the same detail involved in importing the
base module).

As a result, conditionally importing the right version typically uses
a conditional test based on the operating system, and the same code is
seen in a number of different modules, both internal to Swift and external:

* [Test for mmap in stdlib](https://github.com/apple/swift/blob/07b196d2f9a5facc490b35e3649e18937796239b/test/stdlib/mmap.swift#L4-L9)
* [Validation test for PassIfChildCrashedDuringTestExecution](https://github.com/apple/swift/blob/c3b7709a7c4789f1ad7249d357f69509fb8be731/validation-test/StdlibUnittest/ChildProcessShutdown/PassIfChildCrashedDuringTestExecution.swift#L4-L9)
* [Kitura's Socket definitions](https://github.com/IBM-Swift/BlueSocket/blob/49c5af8b6953cecc8674a7fcf746fa27a72c056a/Sources/Socket.swift#L21-L25)
* [Vapor's HTTP Server](https://github.com/vapor/engine/blob/1f95094ee470408309e98dd56b2251210d6a2a3d/Sources/HTTP/Models/Server/HTTP%2BServer.swift#L1-L5)

Some have already created a `Libc` module that effectively does what this
proposal suggests, such as [Vapor's Core Libc](https://github.com/vapor/core/blob/master/Sources/libc/libc.swift)

    ```
    #if os(Linux)
      @_exported import Glibc
    #else
      @_exported import Darwin.C
    #endif
    ```

Each of these examples has subtly different behaviour; for example,
whether or not the os tests only include Linux (and then fail over to
Darwin), or whether they contain other Unices such as FreeBSD and Android.

## Proposed solution

The solution is to formalise these patterns in the base Swift library
and present a `Libc` module that conditionally imports `Glibc` or `Darwin`
based on the correct platform. Additional operating systems can be added
and kept up to date with the list of supported operating system conditionals
and including a failure message when an unknown operating system is detected.

## Detailed design

This will add a `Libc` module for the standard library that re-exports
the correct import depending on the operating system:

    ```
    #if os(macOS) || os(iOS) || os(tvOS) || os(watchOS)
      @_exported import Darwin
    #elseif os(Linux) || os(FreeBSD) || os(Android) || os(PS4)
      @_exported import Glibc
    #else
      fatalError("Libc not supported on operating system")
    #endif
    ```

As new operating systems are added or become supported (such as Windows)
the standard imports can be added appropriately to this module.

## Source compatibility

There is no impact to source compatibility, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.
However it improves source compatibility going forwards, since as new
operating systems are added this file will be updated, instead of the change
having to be made in multiple open-source projects.

## Effect on ABI stability

There is no impact to ABI compatibility, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.

## Effect on API resilience

There is no impact to ABI resilience, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.

## Alternatives considered

The first alternative is to do nothing. Existing Swift projects already
conditionally import these modules, or import a higher-level module (such
as `Foundation`) that performs the conditional import.

The second alternative is to export sub-modules of the modules. Clang
permits imports of sub-modules, so it could be possible to import only
`Darwin.POSIX` and `GlibC.POSIX`. However, in Swift, importing a sub-module
makes the whole module available anyway, so the difference between importing
a whole module versus a submodule is irrelevant.

The third alternative is to explore creating standard functions (in Swift)
corresponding to POSIX functionality, but where the format of the return
results are known. This would require a per-operating system binding to
expose operating-system details such as the byte ordering of structures
as used in the various `getaddrinfo` calls. These may evolve out of future
evolution proposals and this does not conflict with those goals at this
stage. There are additional clean-ups that this could address, such as the
use of the (thread-local) `errno` which may not be reliably read from within
Swift. However, the (swift-evolution thread)[https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027602.html]
calls this "the perfect being the enemy of the good". Instead of trying to
solve all of these problems, they should be handled by subsequent
proposals (such as (Johannes' proposal)[https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161031/028627.html]
regarding errno handling sent to swift-evolution previously).


(Matt Wright) #2

Although out of scope for phase 1, something that keeps cropping up in a variety of Linux/Darwin Swift scripts is the conditional inclusion of Darwin or GlibC per platform. The last point was an observation that creating a 'nice' wrapper for LibC or a cleaned up POSIX API is a non-goal:

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027621.html

I appreciate the desire to have a combined module for this but I'm not convinced that `Libc` (or `LibC`) is a particularly good choice of name here. The `Lib` prefix feels particularly non-Swifty here, most other instances of lib<something> on Darwin have their `lib` prefix dropped when imported as module. From a hierarchical point of view, the `Darwin` module encompasses a suite of libraries that are larger than libsystem_c.dylib (the Darwin Libc). Confusing the naming with layering here would be unfortunate. There's also a potentially confusing Darwin.C submodule that isn't what you're asking for but does step, somewhat, on the namespace.

Perhaps names more along the lines of `Platform` or `Base` would work better here? On Darwin the all-encompasing base libraries are all under Libsystem, `System` would be another potentially platform-agnostic name.

···

On Nov 9, 2016, at 10:58 AM, Alex Blewitt via swift-evolution <swift-evolution@swift.org> wrote:

I think it makes sense to have a cross platform “libc” which is an alias for darwin, glibc, or whatever, and just leave it at that.

Other proposals for a “POSIX” module have gotten bogged down because inevitably the idea comes up to make the resultant API nicer in various ways: rename creat, handle errno more nicely, make use of multiple return values, … etc. The problem with this approach is that we don’t *want* people using these layer of APIs, we want higher level Foundation-like APIs to be used.

...

I think we should formally decide that a “nice” wrapper for libc is a non-goal. There is too much that doesn’t make sense to wrap at this level - the only Swift code that should be using this is the implementation of higher level API, and such extremely narrow cases that we can live with them having to handle the problems of dealing with the raw APIs directly.

-Chris

I have created a draft for a proposal to create such a module. Comments are welcome.

Alex

---

# Libc module for Swift

* Proposal: [SE-NNNN](NNNN-filename.md)
* Authors: [Alex Blewitt](https://github.com/alblue)
* Review Manager: TBD
* Status: **Under discussion**

## Introduction

When running on Darwin, the base module is called `Darwin`. When running
on Linux or other operating systems, it's called `GlibC`.

This repeatedly leads to code such as:

    ````
    #if os(Linux)
      import Glibc
    #else
      import Darwin
    #endif
    ```

As the set of operating systems evolve, one of these conditional imports
needs to be updated. Instead of repeating this, make it available via a
standard `Libc` module in the base Swift library.

Swift-evolution thread: [Discussion thread topic for that proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027621.html)

## Motivation

The [set of platforms](https://github.com/apple/swift/blob/fdf6ee20e4ca1fd32482f4b7b88a97ebdda52cd2/lib/Basic/LangOptions.cpp#L26-L36)
that Swift currently runs on can be divided into two; Darwin and XNU based systems
(macOS, iOS, watchOS, tvOS), Windows, and Unix based systems
(Linux, FreeBSD, Android, PS4).

The base module on Darwin is called `Darwin`, while on Linux and
other Unix systems the base module is called `Glibc`. The base
module is typically conditionally included when working at a lower layer
than Foundation (which has the same detail involved in importing the
base module).

As a result, conditionally importing the right version typically uses
a conditional test based on the operating system, and the same code is
seen in a number of different modules, both internal to Swift and external:

* [Test for mmap in stdlib](https://github.com/apple/swift/blob/07b196d2f9a5facc490b35e3649e18937796239b/test/stdlib/mmap.swift#L4-L9)
* [Validation test for PassIfChildCrashedDuringTestExecution](https://github.com/apple/swift/blob/c3b7709a7c4789f1ad7249d357f69509fb8be731/validation-test/StdlibUnittest/ChildProcessShutdown/PassIfChildCrashedDuringTestExecution.swift#L4-L9)
* [Kitura's Socket definitions](https://github.com/IBM-Swift/BlueSocket/blob/49c5af8b6953cecc8674a7fcf746fa27a72c056a/Sources/Socket.swift#L21-L25)
* [Vapor's HTTP Server](https://github.com/vapor/engine/blob/1f95094ee470408309e98dd56b2251210d6a2a3d/Sources/HTTP/Models/Server/HTTP%2BServer.swift#L1-L5)

Some have already created a `Libc` module that effectively does what this
proposal suggests, such as [Vapor's Core Libc](https://github.com/vapor/core/blob/master/Sources/libc/libc.swift)

    ```
    #if os(Linux)
      @_exported import Glibc
    #else
      @_exported import Darwin.C
    #endif
    ```

Each of these examples has subtly different behaviour; for example,
whether or not the os tests only include Linux (and then fail over to
Darwin), or whether they contain other Unices such as FreeBSD and Android.

## Proposed solution

The solution is to formalise these patterns in the base Swift library
and present a `Libc` module that conditionally imports `Glibc` or `Darwin`
based on the correct platform. Additional operating systems can be added
and kept up to date with the list of supported operating system conditionals
and including a failure message when an unknown operating system is detected.

## Detailed design

This will add a `Libc` module for the standard library that re-exports
the correct import depending on the operating system:

    ```
    #if os(macOS) || os(iOS) || os(tvOS) || os(watchOS)
      @_exported import Darwin
    #elseif os(Linux) || os(FreeBSD) || os(Android) || os(PS4)
      @_exported import Glibc
    #else
      fatalError("Libc not supported on operating system")
    #endif
    ```

As new operating systems are added or become supported (such as Windows)
the standard imports can be added appropriately to this module.

## Source compatibility

There is no impact to source compatibility, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.
However it improves source compatibility going forwards, since as new
operating systems are added this file will be updated, instead of the change
having to be made in multiple open-source projects.

## Effect on ABI stability

There is no impact to ABI compatibility, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.

## Effect on API resilience

There is no impact to ABI resilience, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.

## Alternatives considered

The first alternative is to do nothing. Existing Swift projects already
conditionally import these modules, or import a higher-level module (such
as `Foundation`) that performs the conditional import.

The second alternative is to export sub-modules of the modules. Clang
permits imports of sub-modules, so it could be possible to import only
`Darwin.POSIX` and `GlibC.POSIX`. However, in Swift, importing a sub-module
makes the whole module available anyway, so the difference between importing
a whole module versus a submodule is irrelevant.

The third alternative is to explore creating standard functions (in Swift)
corresponding to POSIX functionality, but where the format of the return
results are known. This would require a per-operating system binding to
expose operating-system details such as the byte ordering of structures
as used in the various `getaddrinfo` calls. These may evolve out of future
evolution proposals and this does not conflict with those goals at this
stage. There are additional clean-ups that this could address, such as the
use of the (thread-local) `errno` which may not be reliably read from within
Swift. However, the (swift-evolution thread)[https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027602.html]
calls this "the perfect being the enemy of the good". Instead of trying to
solve all of these problems, they should be handled by subsequent
proposals (such as (Johannes' proposal)[https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161031/028627.html]
regarding errno handling sent to swift-evolution previously).
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Drew Crawford) #3

grep -R "import Glibc" ~/Code --include "*.swift" | wc -l
    297

As someone who might be characterized as suffering from the problem this
proposal purports to solve, I am not convinced.

The primary problem here is that "libc" is a misnomer. Did you mean
musl, dietlibc, or glibc? Did you mean "whatever libc my distro likes?"
Swift in practice only supports one per platform, but that is a bug not
a feature, and that bug should not be standardized. We could try to
invent some syntax to specify one but now we are back with the current
system again.

The other problem is that in all my usages, "import Glibc" is not a real
problem I face. The real problems are that "the libcs *plural*" are
*just different*. Darwin has timeval64, glibc does not, and you'd
better check your arch and pick the right one, only on one platform.
SO_REUSEADDR has one type in Brand X and another type in Brand Y. Don't
even get me *started* on poll, EREs, or half a dozen other behavioral
variations.

Taking two different libraries and pretending they are the same is not
the solution, it's the disease. The way out of this swamp for most
developers is to use a real Swift library, the same damn Swift library,
on all platforms (sadly, Foundation today does not meet this
requirement). The way out of this swamp for crazy people like me who
must write to the metal is to actually write to the metal, to the
particular libc being targeted, not to a hypothetical platonic ideal
libc which does not exist.

I realize that four lines at the top of my files is a *visible*
annoyance, but fixing it just promotes it to an invisible one.

Drew

···

--
  Drew Crawford
  drew@sealedabstract.com

On Wed, Nov 9, 2016, at 12:58 PM, Alex Blewitt via swift-evolution wrote:

Although out of scope for phase 1, something that keeps cropping up in
a variety of Linux/Darwin Swift scripts is the conditional inclusion
of Darwin or GlibC per platform. The last point was an observation
that creating a 'nice' wrapper for LibC or a cleaned up POSIX API is a
non-goal:

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027621.html

I think it makes sense to have a cross platform “libc” which is an
alias for darwin, glibc, or whatever, and just leave it at that.
Other proposals for a “POSIX” module have gotten bogged down because
inevitably the idea comes up to make the resultant API nicer in
various ways: rename creat, handle errno more nicely, make use of
multiple return values, … etc. The problem with this approach is
that we don’t *want* people using these layer of APIs, we want higher
level Foundation-like APIs to be used. ...*

* I think we should formally decide that a “nice” wrapper for libc is a
  non-goal. There is too much that doesn’t make sense to wrap at this
  level - the only Swift code that should be using this is the
  implementation of higher level API, and such extremely narrow cases
  that we can live with them having to handle the problems of dealing
  with the raw APIs directly. -Chris

I have created a draft for a proposal to create such a module.
Comments are welcome.

Alex

---

# Libc module for Swift

* Proposal: [SE-NNNN](NNNN-filename.md)
* Authors: [Alex Blewitt](https://github.com/alblue)
* Review Manager: TBD
* Status: **Under discussion**

## Introduction

When running on Darwin, the base module is called `Darwin`.
When running
on Linux or other operating systems, it's called `GlibC`.

This repeatedly leads to code such as:

    ````
    #if os(Linux)
      import Glibc
    #else
      import Darwin
    #endif
    ```

As the set of operating systems evolve, one of these
conditional imports
needs to be updated. Instead of repeating this, make it
available via a
standard `Libc` module in the base Swift library.

Swift-evolution thread: [Discussion thread topic for that proposal]
(https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027621.html)

## Motivation

The [set of platforms]
(https://github.com/apple/swift/blob/fdf6ee20e4ca1fd32482f4b7b88a97ebdda52cd2/lib/Basic/LangOptions.cpp#L26-L36)
that Swift currently runs on can be divided into two; Darwin and XNU
based systems
(macOS, iOS, watchOS, tvOS), Windows, and Unix based systems
(Linux, FreeBSD, Android, PS4).

The base module on Darwin is called `Darwin`, while on Linux and
other Unix systems the base module is called `Glibc`. The base
module is typically conditionally included when working at a
lower layer
than Foundation (which has the same detail involved in importing the
base module).

As a result, conditionally importing the right version typically uses
a conditional test based on the operating system, and the same code is
seen in a number of different modules, both internal to Swift and
external:

* [Test for mmap in stdlib]
  (https://github.com/apple/swift/blob/07b196d2f9a5facc490b35e3649e18937796239b/test/stdlib/mmap.swift#L4-L9)
* [Validation test for PassIfChildCrashedDuringTestExecution]
  (https://github.com/apple/swift/blob/c3b7709a7c4789f1ad7249d357f69509fb8be731/validation-test/StdlibUnittest/ChildProcessShutdown/PassIfChildCrashedDuringTestExecution.swift#L4-L9)
* [Kitura's Socket definitions]
  (https://github.com/IBM-Swift/BlueSocket/blob/49c5af8b6953cecc8674a7fcf746fa27a72c056a/Sources/Socket.swift#L21-L25)
* [Vapor's HTTP Server]
  (https://github.com/vapor/engine/blob/1f95094ee470408309e98dd56b2251210d6a2a3d/Sources/HTTP/Models/Server/HTTP%2BServer.swift#L1-L5)

Some have already created a `Libc` module that effectively does
what this
proposal suggests, such as [Vapor's Core Libc]
(https://github.com/vapor/core/blob/master/Sources/libc/libc.swift)

    ```
    #if os(Linux)
      @_exported import Glibc
    #else
      @_exported import Darwin.C
    #endif
    ```

Each of these examples has subtly different behaviour; for example,
whether or not the os tests only include Linux (and then fail over to
Darwin), or whether they contain other Unices such as FreeBSD and
Android.

## Proposed solution

The solution is to formalise these patterns in the base Swift library
and present a `Libc` module that conditionally imports `Glibc` or
`Darwin`
based on the correct platform. Additional operating systems can
be added
and kept up to date with the list of supported operating system
conditionals
and including a failure message when an unknown operating system is
detected.

## Detailed design

This will add a `Libc` module for the standard library that re-exports
the correct import depending on the operating system:

    ```
    #if os(macOS) || os(iOS) || os(tvOS) || os(watchOS)
      @_exported import Darwin
    #elseif os(Linux) || os(FreeBSD) || os(Android) || os(PS4)
      @_exported import Glibc
    #else
      fatalError("Libc not supported on operating system")
    #endif
    ```

As new operating systems are added or become supported (such as
Windows)
the standard imports can be added appropriately to this module.

## Source compatibility

There is no impact to source compatibility, since this proposal is
additive.
Existing source code will work regardless of if this module is
used or not.
However it improves source compatibility going forwards, since as new
operating systems are added this file will be updated, instead of
the change
having to be made in multiple open-source projects.

## Effect on ABI stability

There is no impact to ABI compatibility, since this proposal is
additive.
Existing source code will work regardless of if this module is
used or not.

## Effect on API resilience

There is no impact to ABI resilience, since this proposal is additive.
Existing source code will work regardless of if this module is
used or not.

## Alternatives considered

The first alternative is to do nothing. Existing Swift projects
already
conditionally import these modules, or import a higher-level
module (such
as `Foundation`) that performs the conditional import.

The second alternative is to export sub-modules of the modules. Clang
permits imports of sub-modules, so it could be possible to import only
`Darwin.POSIX` and `GlibC.POSIX`. However, in Swift, importing a
sub-module
makes the whole module available anyway, so the difference between
importing
a whole module versus a submodule is irrelevant.

The third alternative is to explore creating standard functions
(in Swift)
corresponding to POSIX functionality, but where the format of
the return
results are known. This would require a per-operating system
binding to
expose operating-system details such as the byte ordering of
structures
as used in the various `getaddrinfo` calls. These may evolve out
of future
evolution proposals and this does not conflict with those goals at
this
stage. There are additional clean-ups that this could address,
such as the
use of the (thread-local) `errno` which may not be reliably read
from within
Swift. However, the (swift-evolution thread)[
https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027602.html]
calls this "the perfect being the enemy of the good". Instead of
trying to
solve all of these problems, they should be handled by subsequent
proposals (such as (Johannes' proposal)[
https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161031/028627.html]
regarding errno handling sent to swift-evolution previously).
_________________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Dave Abrahams) #4

Although out of scope for phase 1, something that keeps cropping up
in a variety of Linux/Darwin Swift scripts is the conditional
inclusion of Darwin or GlibC per platform. The last point was an
observation that creating a 'nice' wrapper for LibC or a cleaned up
POSIX API is a non-goal:

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027621.html

I appreciate the desire to have a combined module for this but I'm not
convinced that `Libc` (or `LibC`) is a particularly good choice of
name here. The `Lib` prefix feels particularly non-Swifty here, most
other instances of lib<something> on Darwin have their `lib` prefix
dropped when imported as module. From a hierarchical point of view,
the `Darwin` module encompasses a suite of libraries that are larger
than libsystem_c.dylib (the Darwin Libc). Confusing the naming with
layering here would be unfortunate. There's also a potentially
confusing Darwin.C submodule that isn't what you're asking for but
does step, somewhat, on the namespace.

Perhaps names more along the lines of `Platform` or `Base` would work
better here? On Darwin the all-encompasing base libraries are all
under Libsystem, `System` would be another potentially
platform-agnostic name.

Howzat going to work out on Windows?

···

on Wed Nov 09 2016, Matt Wright <swift-evolution@swift.org> wrote:

On Nov 9, 2016, at 10:58 AM, Alex Blewitt via swift-evolution <swift-evolution@swift.org> wrote:

I think it makes sense to have a cross platform “libc” which is an
alias for darwin, glibc, or whatever, and just leave it at that.

Other proposals for a “POSIX” module have gotten bogged down
because inevitably the idea comes up to make the resultant API
nicer in various ways: rename creat, handle errno more nicely, make
use of multiple return values, … etc. The problem with this
approach is that we don’t *want* people using these layer of APIs,
we want higher level Foundation-like APIs to be used.

...

I think we should formally decide that a “nice” wrapper for libc is
a non-goal. There is too much that doesn’t make sense to wrap at
this level - the only Swift code that should be using this is the
implementation of higher level API, and such extremely narrow cases
that we can live with them having to handle the problems of dealing
with the raw APIs directly.

-Chris

I have created a draft for a proposal to create such a module. Comments are welcome.

Alex

---

# Libc module for Swift

* Proposal: [SE-NNNN](NNNN-filename.md)
* Authors: [Alex Blewitt](https://github.com/alblue)
* Review Manager: TBD
* Status: **Under discussion**

## Introduction

When running on Darwin, the base module is called `Darwin`. When running
on Linux or other operating systems, it's called `GlibC`.

This repeatedly leads to code such as:

    ````
    #if os(Linux)
      import Glibc
    #else
      import Darwin
    #endif
    ```

As the set of operating systems evolve, one of these conditional imports
needs to be updated. Instead of repeating this, make it available via a
standard `Libc` module in the base Swift library.

Swift-evolution thread: [Discussion thread topic for that proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027621.html)

## Motivation

The [set of platforms](https://github.com/apple/swift/blob/fdf6ee20e4ca1fd32482f4b7b88a97ebdda52cd2/lib/Basic/LangOptions.cpp#L26-L36)
that Swift currently runs on can be divided into two; Darwin and XNU based systems
(macOS, iOS, watchOS, tvOS), Windows, and Unix based systems
(Linux, FreeBSD, Android, PS4).

The base module on Darwin is called `Darwin`, while on Linux and
other Unix systems the base module is called `Glibc`. The base
module is typically conditionally included when working at a lower layer
than Foundation (which has the same detail involved in importing the
base module).

As a result, conditionally importing the right version typically uses
a conditional test based on the operating system, and the same code is
seen in a number of different modules, both internal to Swift and external:

* [Test for mmap in stdlib](https://github.com/apple/swift/blob/07b196d2f9a5facc490b35e3649e18937796239b/test/stdlib/mmap.swift#L4-L9)
* [Validation test for PassIfChildCrashedDuringTestExecution](https://github.com/apple/swift/blob/c3b7709a7c4789f1ad7249d357f69509fb8be731/validation-test/StdlibUnittest/ChildProcessShutdown/PassIfChildCrashedDuringTestExecution.swift#L4-L9)
* [Kitura's Socket definitions](https://github.com/IBM-Swift/BlueSocket/blob/49c5af8b6953cecc8674a7fcf746fa27a72c056a/Sources/Socket.swift#L21-L25)
* [Vapor's HTTP Server](https://github.com/vapor/engine/blob/1f95094ee470408309e98dd56b2251210d6a2a3d/Sources/HTTP/Models/Server/HTTP%2BServer.swift#L1-L5)

Some have already created a `Libc` module that effectively does what this
proposal suggests, such as [Vapor's Core Libc](https://github.com/vapor/core/blob/master/Sources/libc/libc.swift)

    ```
    #if os(Linux)
      @_exported import Glibc
    #else
      @_exported import Darwin.C
    #endif
    ```

Each of these examples has subtly different behaviour; for example,
whether or not the os tests only include Linux (and then fail over to
Darwin), or whether they contain other Unices such as FreeBSD and Android.

## Proposed solution

The solution is to formalise these patterns in the base Swift library
and present a `Libc` module that conditionally imports `Glibc` or `Darwin`
based on the correct platform. Additional operating systems can be added
and kept up to date with the list of supported operating system conditionals
and including a failure message when an unknown operating system is detected.

## Detailed design

This will add a `Libc` module for the standard library that re-exports
the correct import depending on the operating system:

    ```
    #if os(macOS) || os(iOS) || os(tvOS) || os(watchOS)
      @_exported import Darwin
    #elseif os(Linux) || os(FreeBSD) || os(Android) || os(PS4)
      @_exported import Glibc
    #else
      fatalError("Libc not supported on operating system")
    #endif
    ```

As new operating systems are added or become supported (such as Windows)
the standard imports can be added appropriately to this module.

## Source compatibility

There is no impact to source compatibility, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.
However it improves source compatibility going forwards, since as new
operating systems are added this file will be updated, instead of the change
having to be made in multiple open-source projects.

## Effect on ABI stability

There is no impact to ABI compatibility, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.

## Effect on API resilience

There is no impact to ABI resilience, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.

## Alternatives considered

The first alternative is to do nothing. Existing Swift projects already
conditionally import these modules, or import a higher-level module (such
as `Foundation`) that performs the conditional import.

The second alternative is to export sub-modules of the modules. Clang
permits imports of sub-modules, so it could be possible to import only
`Darwin.POSIX` and `GlibC.POSIX`. However, in Swift, importing a sub-module
makes the whole module available anyway, so the difference between importing
a whole module versus a submodule is irrelevant.

The third alternative is to explore creating standard functions (in Swift)
corresponding to POSIX functionality, but where the format of the return
results are known. This would require a per-operating system binding to
expose operating-system details such as the byte ordering of structures
as used in the various `getaddrinfo` calls. These may evolve out of future
evolution proposals and this does not conflict with those goals at this
stage. There are additional clean-ups that this could address, such as the
use of the (thread-local) `errno` which may not be reliably read from within
Swift. However, the (swift-evolution

thread)[https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027602.html]

calls this "the perfect being the enemy of the good". Instead of trying to
solve all of these problems, they should be handled by subsequent
proposals (such as (Johannes'

proposal)[https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161031/028627.html]

regarding errno handling sent to swift-evolution previously).
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
-Dave


(Alexis) #5

I agree that trying to completely unify low-level platforms is usually a mess. That said, I also don’t think accessing platform specific behaviour needs to involve completely throwing away the nice abstractions in Foundation. Wherever possible, we should provide platform-specific extensions to the types in Foundation. For instance, we could expose methods/inits that operate in terms of file descriptors on unix-y systems, and handle_t on windows.

But I also think there should be some opt-in to doing this, so that Foundation users can be confident they’re writing portable software by default. I don’t think imports should be the mechanism for this because this necessarily forces awkward divisions. I’m cautiously optimistic the feature flag system we need to build out for language evolution purposes will provide a good fit here. Opting into platform-specific behaviour is fairly similar to opting into experimental APIs.

(Note: I haven’t actually used Foundation much, so this may be inconsistent with its overarching design)

···

On Nov 10, 2016, at 10:48 PM, Drew Crawford via swift-evolution <swift-evolution@swift.org> wrote:

    grep -R "import Glibc" ~/Code --include "*.swift" | wc -l
    297

As someone who might be characterized as suffering from the problem this proposal purports to solve, I am not convinced.

The primary problem here is that "libc" is a misnomer. Did you mean musl, dietlibc, or glibc? Did you mean "whatever libc my distro likes?" Swift in practice only supports one per platform, but that is a bug not a feature, and that bug should not be standardized. We could try to invent some syntax to specify one but now we are back with the current system again.

The other problem is that in all my usages, "import Glibc" is not a real problem I face. The real problems are that "the libcs plural" are *just different*. Darwin has timeval64, glibc does not, and you'd better check your arch and pick the right one, only on one platform. SO_REUSEADDR has one type in Brand X and another type in Brand Y. Don't even get me *started* on poll, EREs, or half a dozen other behavioral variations.

Taking two different libraries and pretending they are the same is not the solution, it's the disease. The way out of this swamp for most developers is to use a real Swift library, the same damn Swift library, on all platforms (sadly, Foundation today does not meet this requirement). The way out of this swamp for crazy people like me who must write to the metal is to actually write to the metal, to the particular libc being targeted, not to a hypothetical platonic ideal libc which does not exist.

I realize that four lines at the top of my files is a *visible* annoyance, but fixing it just promotes it to an invisible one.

Drew

--
  Drew Crawford
  drew@sealedabstract.com

On Wed, Nov 9, 2016, at 12:58 PM, Alex Blewitt via swift-evolution wrote:

Although out of scope for phase 1, something that keeps cropping up in a variety of Linux/Darwin Swift scripts is the conditional inclusion of Darwin or GlibC per platform. The last point was an observation that creating a 'nice' wrapper for LibC or a cleaned up POSIX API is a non-goal:

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027621.html

I think it makes sense to have a cross platform “libc” which is an alias for darwin, glibc, or whatever, and just leave it at that.

Other proposals for a “POSIX” module have gotten bogged down because inevitably the idea comes up to make the resultant API nicer in various ways: rename creat, handle errno more nicely, make use of multiple return values, … etc. The problem with this approach is that we don’t *want* people using these layer of APIs, we want higher level Foundation-like APIs to be used.

...

I think we should formally decide that a “nice” wrapper for libc is a non-goal. There is too much that doesn’t make sense to wrap at this level - the only Swift code that should be using this is the implementation of higher level API, and such extremely narrow cases that we can live with them having to handle the problems of dealing with the raw APIs directly.

-Chris

I have created a draft for a proposal to create such a module. Comments are welcome.

Alex

---

# Libc module for Swift

* Proposal: [SE-NNNN](NNNN-filename.md)
* Authors: [Alex Blewitt](https://github.com/alblue)
* Review Manager: TBD
* Status: **Under discussion**

## Introduction

When running on Darwin, the base module is called `Darwin`. When running
on Linux or other operating systems, it's called `GlibC`.

This repeatedly leads to code such as:

    ````
    #if os(Linux)
      import Glibc
    #else
      import Darwin
    #endif
    ```

As the set of operating systems evolve, one of these conditional imports
needs to be updated. Instead of repeating this, make it available via a
standard `Libc` module in the base Swift library.

Swift-evolution thread: [Discussion thread topic for that proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027621.html)

## Motivation

The [set of platforms](https://github.com/apple/swift/blob/fdf6ee20e4ca1fd32482f4b7b88a97ebdda52cd2/lib/Basic/LangOptions.cpp#L26-L36)
that Swift currently runs on can be divided into two; Darwin and XNU based systems
(macOS, iOS, watchOS, tvOS), Windows, and Unix based systems
(Linux, FreeBSD, Android, PS4).

The base module on Darwin is called `Darwin`, while on Linux and
other Unix systems the base module is called `Glibc`. The base
module is typically conditionally included when working at a lower layer
than Foundation (which has the same detail involved in importing the
base module).

As a result, conditionally importing the right version typically uses
a conditional test based on the operating system, and the same code is
seen in a number of different modules, both internal to Swift and external:

* [Test for mmap in stdlib](https://github.com/apple/swift/blob/07b196d2f9a5facc490b35e3649e18937796239b/test/stdlib/mmap.swift#L4-L9)
* [Validation test for PassIfChildCrashedDuringTestExecution](https://github.com/apple/swift/blob/c3b7709a7c4789f1ad7249d357f69509fb8be731/validation-test/StdlibUnittest/ChildProcessShutdown/PassIfChildCrashedDuringTestExecution.swift#L4-L9)
* [Kitura's Socket definitions](https://github.com/IBM-Swift/BlueSocket/blob/49c5af8b6953cecc8674a7fcf746fa27a72c056a/Sources/Socket.swift#L21-L25)
* [Vapor's HTTP Server](https://github.com/vapor/engine/blob/1f95094ee470408309e98dd56b2251210d6a2a3d/Sources/HTTP/Models/Server/HTTP%2BServer.swift#L1-L5)

Some have already created a `Libc` module that effectively does what this
proposal suggests, such as [Vapor's Core Libc](https://github.com/vapor/core/blob/master/Sources/libc/libc.swift)

    ```
    #if os(Linux)
      @_exported import Glibc
    #else
      @_exported import Darwin.C
    #endif
    ```

Each of these examples has subtly different behaviour; for example,
whether or not the os tests only include Linux (and then fail over to
Darwin), or whether they contain other Unices such as FreeBSD and Android.

## Proposed solution

The solution is to formalise these patterns in the base Swift library
and present a `Libc` module that conditionally imports `Glibc` or `Darwin`
based on the correct platform. Additional operating systems can be added
and kept up to date with the list of supported operating system conditionals
and including a failure message when an unknown operating system is detected.

## Detailed design

This will add a `Libc` module for the standard library that re-exports
the correct import depending on the operating system:

    ```
    #if os(macOS) || os(iOS) || os(tvOS) || os(watchOS)
      @_exported import Darwin
    #elseif os(Linux) || os(FreeBSD) || os(Android) || os(PS4)
      @_exported import Glibc
    #else
      fatalError("Libc not supported on operating system")
    #endif
    ```

As new operating systems are added or become supported (such as Windows)
the standard imports can be added appropriately to this module.

## Source compatibility

There is no impact to source compatibility, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.
However it improves source compatibility going forwards, since as new
operating systems are added this file will be updated, instead of the change
having to be made in multiple open-source projects.

## Effect on ABI stability

There is no impact to ABI compatibility, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.

## Effect on API resilience

There is no impact to ABI resilience, since this proposal is additive.
Existing source code will work regardless of if this module is used or not.

## Alternatives considered

The first alternative is to do nothing. Existing Swift projects already
conditionally import these modules, or import a higher-level module (such
as `Foundation`) that performs the conditional import.

The second alternative is to export sub-modules of the modules. Clang
permits imports of sub-modules, so it could be possible to import only
`Darwin.POSIX` and `GlibC.POSIX`. However, in Swift, importing a sub-module
makes the whole module available anyway, so the difference between importing
a whole module versus a submodule is irrelevant.

The third alternative is to explore creating standard functions (in Swift)
corresponding to POSIX functionality, but where the format of the return
results are known. This would require a per-operating system binding to
expose operating-system details such as the byte ordering of structures
as used in the various `getaddrinfo` calls. These may evolve out of future
evolution proposals and this does not conflict with those goals at this
stage. There are additional clean-ups that this could address, such as the
use of the (thread-local) `errno` which may not be reliably read from within
Swift. However, the (swift-evolution thread)[https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161003/027602.html]
calls this "the perfect being the enemy of the good". Instead of trying to
solve all of these problems, they should be handled by subsequent
proposals (such as (Johannes' proposal)[https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161031/028627.html]
regarding errno handling sent to swift-evolution previously).
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Alex Blewitt) #6

    grep -R "import Glibc" ~/Code --include "*.swift" | wc -l
    297

As someone who might be characterized as suffering from the problem this proposal purports to solve, I am not convinced.

The primary problem here is that "libc" is a misnomer. Did you mean musl, dietlibc, or glibc? Did you mean "whatever libc my distro likes?" Swift in practice only supports one per platform, but that is a bug not a feature, and that bug should not be standardized. We could try to invent some syntax to specify one but now we are back with the current system again.

We're at the current system to start off with, though. When you do "import Darwin" or "import GlibC" you're getting whatever the platform has, regardless of what you call it. You could call it something else, like "Platform" or "Base" but it doesn't change the suggestion itself.

The other problem is that in all my usages, "import Glibc" is not a real problem I face. The real problems are that "the libcs plural" are *just different*. Darwin has timeval64, glibc does not, and you'd better check your arch and pick the right one, only on one platform. SO_REUSEADDR has one type in Brand X and another type in Brand Y. Don't even get me *started* on poll, EREs, or half a dozen other behavioral variations.

Yes, these are issues. Some of them will be worked out with the swift server workgroup, or at least standardising Socket as a type which abstracts the platform away. But we're at that position at the moment, whether or not there's a standard module to represent Darwin/Glibc.

Taking two different libraries and pretending they are the same is not the solution, it's the disease. The way out of this swamp for most developers is to use a real Swift library, the same damn Swift library, on all platforms (sadly, Foundation today does not meet this requirement). The way out of this swamp for crazy people like me who must write to the metal is to actually write to the metal, to the particular libc being targeted, not to a hypothetical platonic ideal libc which does not exist.

I realize that four lines at the top of my files is a *visible* annoyance, but fixing it just promotes it to an invisible one.

Not necessarily, it can be a starting point to fix some of the other problems. In any case, the four lines at the top of your files are almost certainly inconsistent on other platforms; for example, do you test for freebsd? Or ps4?

https://github.com/drewcrawford/Caroline/blob/26cd0d71e57a62fac6258e4e13dfd6849a1945c6/caroline-static-tool/FileUtils.swift

#if os(OSX)
import Darwin
#elseif os(Linux)
import Glibc
#endif

So your test framework doesn't work on FreeBSD by default. Yet they've still got the same 'write' method. It also doesn't seem to support some of the other platforms that might be desirable in a test framework, such as iOS, watchOS or tvOS. You'll just get silent errors on those when it's used on those platforms. And as new platforms get added, your code will slowly drift further away from supporting everything to supporting a few known values.

Now granted, some of these may have yet more incompatible versions for 'write' which needs handling specifically. That's bad, and it should be something that can be worked on. But most of the other functions (like 'close') don't need handling specifically. Or, as used in https://github.com/drewcrawford/Caroline/blob/edd8aefef44717ecfa03c629100baf095fab983a/caroline-static-tool/main.swift to just get access to the exit() function, which is the same across all platforms.

Other proposals - such as Johannes' treatment of how to handle errno - will help work around these problems. Perhaps we end up with a generic write function that wraps the platform specific one to abstract that away as well, which reduces these issues one by one.

Alex

···

On 11 Nov 2016, at 03:48, Drew Crawford <drew@sealedabstract.com> wrote:


(Drew Crawford) #7

Thanks for using specific examples, as they are illustrative.

Or, as used in https://github.com/drewcrawford/Caroline/blob/edd8aefef44717ecfa03c629100baf095fab983a/caroline-static-tool/main.swift to just get access to the exit() function, which is the same across all platforms.

exit is an interesting case. Believe it or not, it is *not* portable, as used here. The C standard defines two constants for its parameter: EXIT_SUCCESS (defined to be zero) and EXIT_FAILURE (undefined). You can pass other values (trivia time: only values less than 0377, because the high bits of this integer are reserved on some platforms), but the "fail-iarity" or "successiness" of values other than the defined two are not specified and may take on different meanings on different platforms. C programmers commonly pick a nonzero value to indicate failure (as I did here) – sometimes multiple distinct values to indicate different kinds of failure – but this is actually non-portable.

The reason I am being pedantic here is because it demonstrates a broader point: people do not actually write portable code; they target either their current platform or some vague amalgamation of 2-3 platforms in their immediate vicinity. All the platforms I cared about do something sane with exit(1) so it's "portable enough". But "portable enough" isn't portable.

In any case, the four lines at the top of your files are almost certainly inconsistent on other platforms; for example, do you test for freebsd? Or ps4?

This is a feature, not a bug. We *like* compiler errors because they give us early warning we got something wrong.

I *want* the compiler to stop me when compiling this for OpenVMS. When I wrote this code, I had only Darwin/Glibc in mind, and on that basis, I used exit(1). The working assumption is now violated, and the compiler is correct to remind me about it. I don't know, and would need to go find out, what kind of exit is sensible for VMS. Or to move to EXIT_FAILURE like a language lawyer.

But the compile error itself is not a silly annoyance that I would like to "solve" by blindly putting more libcs in the list etc. This is a guard rail where I indicated the platforms I had in mind when writing this code, and when I wake up one morning in a drunken stupor and try to do something else, we fire a warning shot.

It's a safety feature, like force-unwrap. If you're sure it won't blow up, you can put an exclamation mark in there and shut up the compiler. If you're sure it's portable to bsd, add bsd to the list and shut up the compiler. But the resolution to this compile error is not to import more libcs. The resolution is to *consider carefully if that is a good idea*.

The assumption embedded in the proposal is that of course we want the program to compile, and the question is merely to calculate the set of import statements that will achieve compilation, for which the proposal offers an algorithm. My argument is that actually we do not want the program to compile unless we are reasonably sure it will work as intended, and the question is what syntax allows the author to reasonably encode their assumptions so that when they are violated we provide a moment of quiet reflection to consider e.g. if exit(1) is sensible on VMS.

The existing system is imperfect (and very ugly) but does a surprisingly good job in this dimension.

It also doesn't seem to support some of the other platforms that might be desirable in a test framework, such as iOS, watchOS or tvOS.
I do support iOS (not in this component – it's a command-line tool, so the omission of iOS in this file is deliberate, and the compatibility issues go far beyond exit). I don't yet support watchOS or tvOS because I don't have CI for those platforms yet and in my view supporting a platform is more than adjusting an import statement and wondering if it will compile.

So in summary:

1. I would not use this feature in the cited examples

2. I would prefer it if others did not use this feature. When I see "import Glibc" at the top of a file I know what I am signing up for. When I see "import libc" for all I know the developer used Windows.

Finally, this is more of a detail, but I still do not understand how this would be implemented for a Linux platform without Glibc, such as Arch. The current proposal has

#if os(Linux)

  @\_exported import Glibc

which is obviously not going to compile on Arch. So if the goal is to have syntax that is portable the current proposal does not do it.

I do believe there is some room for a more moderate reform on the libc problem. For example instead of the traditional ifdefs, we could have

import? Glibc

import? Darwin

Where the "import?" keyword imports the module if available or otherwise has no effect.

This preserves the majority of desireable properties discussed above (clearly indicates the intended libcs, provides guard rails similar to the present regime) while being significantly shorter than the present regime. It also contemplates problems that the current proposal does not, such as Arch (we can continue to list libcs until we find one, and Swift already has rules for resolving multiple imports), and it is more broadly applicable to other kinds of import problems than libc imports narrowly.

One problem with the import? idea is that it glosses e.g. watchOS/tvOS/macOS as "Darwin" when that is not strictly true (posix_spawn for example) but it's at least a far more sensible unification than VMS/Darwin. We do have a clear path to check Darwin portability in the compiler (e.g. require @availability(macOS) for posix_spawn) while no similar path exists to check Darwin/VMS portability. So I think doing something moderate is at least plausible.

Still, it may be a solution in search of a problem. If you are writing against something as badly fractured as POSIX, it should feel like it, and even a small unification may be a step too far.

Of course, the real solution to writing portable code is to target a portable standard, such as Foundation or some alternative high-level library.

···

On November 15, 2016 at 10:27:14 AM, Alex Blewitt (alblue@apple.com) wrote:

On 11 Nov 2016, at 03:48, Drew Crawford <drew@sealedabstract.com> wrote:

grep \-R &quot;import Glibc&quot; \~/Code \-\-include &quot;\*\.swift&quot; | wc \-l
297

As someone who might be characterized as suffering from the problem this proposal purports to solve, I am not convinced.

The primary problem here is that "libc" is a misnomer. Did you mean musl, dietlibc, or glibc? Did you mean "whatever libc my distro likes?" Swift in practice only supports one per platform, but that is a bug not a feature, and that bug should not be standardized. We could try to invent some syntax to specify one but now we are back with the current system again.

We're at the current system to start off with, though. When you do "import Darwin" or "import GlibC" you're getting whatever the platform has, regardless of what you call it. You could call it something else, like "Platform" or "Base" but it doesn't change the suggestion itself.

The other problem is that in all my usages, "import Glibc" is not a real problem I face. The real problems are that "the libcs plural" are *just different*. Darwin has timeval64, glibc does not, and you'd better check your arch and pick the right one, only on one platform. SO_REUSEADDR has one type in Brand X and another type in Brand Y. Don't even get me *started* on poll, EREs, or half a dozen other behavioral variations.

Yes, these are issues. Some of them will be worked out with the swift server workgroup, or at least standardising Socket as a type which abstracts the platform away. But we're at that position at the moment, whether or not there's a standard module to represent Darwin/Glibc.

Taking two different libraries and pretending they are the same is not the solution, it's the disease. The way out of this swamp for most developers is to use a real Swift library, the same damn Swift library, on all platforms (sadly, Foundation today does not meet this requirement). The way out of this swamp for crazy people like me who must write to the metal is to actually write to the metal, to the particular libc being targeted, not to a hypothetical platonic ideal libc which does not exist.

I realize that four lines at the top of my files is a *visible* annoyance, but fixing it just promotes it to an invisible one.

Not necessarily, it can be a starting point to fix some of the other problems. In any case, the four lines at the top of your files are almost certainly inconsistent on other platforms; for example, do you test for freebsd? Or ps4?

https://github.com/drewcrawford/Caroline/blob/26cd0d71e57a62fac6258e4e13dfd6849a1945c6/caroline-static-tool/FileUtils.swift

#if
os(OSX)

import
Darwin

#elseif
os(Linux)

import
Glibc

#endif

So your test framework doesn't work on FreeBSD by default. Yet they've still got the same 'write' method. It also doesn't seem to support some of the other platforms that might be desirable in a test framework, such as iOS, watchOS or tvOS. You'll just get silent errors on those when it's used on those platforms. And as new platforms get added, your code will slowly drift further away from supporting everything to supporting a few known values.

Now granted, some of these may have yet more incompatible versions for 'write' which needs handling specifically. That's bad, and it should be something that can be worked on. But most of the other functions (like 'close') don't need handling specifically. Or, as used in https://github.com/drewcrawford/Caroline/blob/edd8aefef44717ecfa03c629100baf095fab983a/caroline-static-tool/main.swift to just get access to the exit() function, which is the same across all platforms.

Other proposals - such as Johannes' treatment of how to handle errno - will help work around these problems. Perhaps we end up with a generic write function that wraps the platform specific one to abstract that away as well, which reduces these issues one by one.

Alex


(Alex Blewitt) #8

OpenVMS isn't in the supported list of Swift packages, so comparing it to that is pointless.

https://github.com/apple/swift/blob/c3b7709a7c4789f1ad7249d357f69509fb8be731/lib/Basic/LangOptions.cpp#L26-L36

static const StringRef SupportedConditionalCompilationOSs[] = {
  "OSX",
  "tvOS",
  "watchOS",
  "iOS",
  "Linux",
  "FreeBSD",
  "Windows",
  "Android",
  "PS4",
};

Look, this code is already battle tested in Swift. It's even used in the test case for the interpreter:

https://github.com/apple/swift/blob/c3b7709a7c4789f1ad7249d357f69509fb8be731/test/Interpreter/SDK/libc.swift#L10-L14

#if os(OSX) || os(iOS) || os(watchOS) || os(tvOS)
  import Darwin
#elseif os(Linux) || os(FreeBSD) || os(PS4) || os(Android)
  import Glibc
#endif

Alex

···

On 15 Nov 2016, at 19:51, Drew Crawford <drew@sealedabstract.com> wrote:

Thanks for using specific examples, as they are illustrative.

Or, as used in https://github.com/drewcrawford/Caroline/blob/edd8aefef44717ecfa03c629100baf095fab983a/caroline-static-tool/main.swift to just get access to the exit() function, which is the same across all platforms.

exit is an interesting case. Believe it or not, it is *not* portable, as used here. The C standard defines two constants for its parameter: EXIT_SUCCESS (defined to be zero) and EXIT_FAILURE (undefined). You can pass other values (trivia time: only values less than 0377, because the high bits of this integer are reserved on some platforms), but the "fail-iarity" or "successiness" of values other than the defined two are not specified and may take on different meanings on different platforms. C programmers commonly pick a nonzero value to indicate failure (as I did here) – sometimes multiple distinct values to indicate different kinds of failure – but this is actually non-portable.

The reason I am being pedantic here is because it demonstrates a broader point: people do not actually write portable code; they target either their current platform or some vague amalgamation of 2-3 platforms in their immediate vicinity. All the platforms I cared about do something sane with exit(1) so it's "portable enough". But "portable enough" isn't portable.

In any case, the four lines at the top of your files are almost certainly inconsistent on other platforms; for example, do you test for freebsd? Or ps4?

This is a feature, not a bug. We *like* compiler errors because they give us early warning we got something wrong.

I *want* the compiler to stop me when compiling this for OpenVMS. When I wrote this code, I had only Darwin/Glibc in mind, and on that basis, I used exit(1). The working assumption is now violated, and the compiler is correct to remind me about it. I don't know, and would need to go find out, what kind of exit is sensible for VMS. Or to move to EXIT_FAILURE like a language lawyer.

But the compile error itself is not a silly annoyance that I would like to "solve" by blindly putting more libcs in the list etc. This is a guard rail where I indicated the platforms I had in mind when writing this code, and when I wake up one morning in a drunken stupor and try to do something else, we fire a warning shot.

It's a safety feature, like force-unwrap. If you're sure it won't blow up, you can put an exclamation mark in there and shut up the compiler. If you're sure it's portable to bsd, add bsd to the list and shut up the compiler. But the resolution to this compile error is not to import more libcs. The resolution is to *consider carefully if that is a good idea*.

The assumption embedded in the proposal is that of course we want the program to compile, and the question is merely to calculate the set of import statements that will achieve compilation, for which the proposal offers an algorithm. My argument is that actually we do not want the program to compile unless we are reasonably sure it will work as intended, and the question is what syntax allows the author to reasonably encode their assumptions so that when they are violated we provide a moment of quiet reflection to consider e.g. if exit(1) is sensible on VMS.

The existing system is imperfect (and very ugly) but does a surprisingly good job in this dimension.

It also doesn't seem to support some of the other platforms that might be desirable in a test framework, such as iOS, watchOS or tvOS.

I do support iOS (not in this component – it's a command-line tool, so the omission of iOS in this file is deliberate, and the compatibility issues go far beyond exit). I don't yet support watchOS or tvOS because I don't have CI for those platforms yet and in my view supporting a platform is more than adjusting an import statement and wondering if it will compile.

So in summary:

1. I would not use this feature in the cited examples

2. I would prefer it if others did not use this feature. When I see "import Glibc" at the top of a file I know what I am signing up for. When I see "import libc" for all I know the developer used Windows.

Finally, this is more of a detail, but I still do not understand how this would be implemented for a Linux platform without Glibc, such as Arch. The current proposal has

  #if os(Linux)

      @_exported import Glibc

which is obviously not going to compile on Arch. So if the goal is to have syntax that is portable the current proposal does not do it.

I do believe there is some room for a more moderate reform on the libc problem. For example instead of the traditional ifdefs, we could have

import? Glibc

import? Darwin

Where the "import?" keyword imports the module if available or otherwise has no effect.

This preserves the majority of desireable properties discussed above (clearly indicates the intended libcs, provides guard rails similar to the present regime) while being significantly shorter than the present regime. It also contemplates problems that the current proposal does not, such as Arch (we can continue to list libcs until we find one, and Swift already has rules for resolving multiple imports), and it is more broadly applicable to other kinds of import problems than libc imports narrowly.

One problem with the import? idea is that it glosses e.g. watchOS/tvOS/macOS as "Darwin" when that is not strictly true (posix_spawn for example) but it's at least a far more sensible unification than VMS/Darwin. We do have a clear path to check Darwin portability in the compiler (e.g. require @availability(macOS) for posix_spawn) while no similar path exists to check Darwin/VMS portability. So I think doing something moderate is at least plausible.

Still, it may be a solution in search of a problem. If you are writing against something as badly fractured as POSIX, it should feel like it, and even a small unification may be a step too far.

Of course, the real solution to writing portable code is to target a portable standard, such as Foundation or some alternative high-level library.
On November 15, 2016 at 10:27:14 AM, Alex Blewitt (alblue@apple.com <mailto:alblue@apple.com>) wrote:

On 11 Nov 2016, at 03:48, Drew Crawford <drew@sealedabstract.com <mailto:drew@sealedabstract.com>> wrote:

    grep -R "import Glibc" ~/Code --include "*.swift" | wc -l
    297

As someone who might be characterized as suffering from the problem this proposal purports to solve, I am not convinced.

The primary problem here is that "libc" is a misnomer. Did you mean musl, dietlibc, or glibc? Did you mean "whatever libc my distro likes?" Swift in practice only supports one per platform, but that is a bug not a feature, and that bug should not be standardized. We could try to invent some syntax to specify one but now we are back with the current system again.

We're at the current system to start off with, though. When you do "import Darwin" or "import GlibC" you're getting whatever the platform has, regardless of what you call it. You could call it something else, like "Platform" or "Base" but it doesn't change the suggestion itself.

The other problem is that in all my usages, "import Glibc" is not a real problem I face. The real problems are that "the libcs plural" are *just different*. Darwin has timeval64, glibc does not, and you'd better check your arch and pick the right one, only on one platform. SO_REUSEADDR has one type in Brand X and another type in Brand Y. Don't even get me *started* on poll, EREs, or half a dozen other behavioral variations.

Yes, these are issues. Some of them will be worked out with the swift server workgroup, or at least standardising Socket as a type which abstracts the platform away. But we're at that position at the moment, whether or not there's a standard module to represent Darwin/Glibc.

Taking two different libraries and pretending they are the same is not the solution, it's the disease. The way out of this swamp for most developers is to use a real Swift library, the same damn Swift library, on all platforms (sadly, Foundation today does not meet this requirement). The way out of this swamp for crazy people like me who must write to the metal is to actually write to the metal, to the particular libc being targeted, not to a hypothetical platonic ideal libc which does not exist.

I realize that four lines at the top of my files is a *visible* annoyance, but fixing it just promotes it to an invisible one.

Not necessarily, it can be a starting point to fix some of the other problems. In any case, the four lines at the top of your files are almost certainly inconsistent on other platforms; for example, do you test for freebsd? Or ps4?

https://github.com/drewcrawford/Caroline/blob/26cd0d71e57a62fac6258e4e13dfd6849a1945c6/caroline-static-tool/FileUtils.swift

#if
os(OSX)

import
Darwin

#elseif
os(Linux)

import
Glibc

#endif

So your test framework doesn't work on FreeBSD by default. Yet they've still got the same 'write' method. It also doesn't seem to support some of the other platforms that might be desirable in a test framework, such as iOS, watchOS or tvOS. You'll just get silent errors on those when it's used on those platforms. And as new platforms get added, your code will slowly drift further away from supporting everything to supporting a few known values.

Now granted, some of these may have yet more incompatible versions for 'write' which needs handling specifically. That's bad, and it should be something that can be worked on. But most of the other functions (like 'close') don't need handling specifically. Or, as used in https://github.com/drewcrawford/Caroline/blob/edd8aefef44717ecfa03c629100baf095fab983a/caroline-static-tool/main.swift to just get access to the exit() function, which is the same across all platforms.

Other proposals - such as Johannes' treatment of how to handle errno - will help work around these problems. Perhaps we end up with a generic write function that wraps the platform specific one to abstract that away as well, which reduces these issues one by one.

Alex