C Macros and Variadic functions


(Ryan Lovelett) #1

I'm beginning to try to port some of my existing C code over to Swift
(you have no idea how excited that makes me). Unfortunately, I've hit a
stumbling block.

When interacting with some of the system/hardware interfaces in Linux it
sometimes (often?) becomes necessary to interact with I/O Control
(ioctl). This introduces two complexities that at least appear to me as
needing resolution.

The first is C macros. A lot ioctl interfaces use headers which define
macros that are specific ioctl request operations. These macros
typically take the form of `_IO`, `_IOR` or `_IOW` (there are more but
this should give you a picture). These values are typically bound to a
specific kernel version and thus are subject to change. I point that out
because obviously one could just calculate the value and hard-code it
into the Swift source (which is what I plan to do in the short term) but
this isn't really a portable solution across kernels. The question is
what, if anything, is planned around this?

The second is variadic C functions. In my case I went ahead and
calculated the result of a C macro and hard coded it into a Swift
variable (e.g., `let DMX_SET_BUFFER_SIZE: CUnsignedInt = 28461`). This
was so I could send it to an ioctl call (e.g., `ioctl(fileReference!,
DMX_SET_BUFFER_SIZE, DVBDVR.DVR_BUFFER_SIZE)`). Unfortunately, this died
in the compiler with `error: 'ioctl' is unavailable: Variadic function
is unavailable`. The interesting part of the compiler error was this
though `SwiftGlibc.ioctl:2:13: note: 'ioctl' has been explicitly marked
unavailable here`. This seems to imply that it _could_ be on and it just
isn't. I searched around a bit and found a few PRs [1] [2] [3] that seem
to indicate that Swift does support variadic C functions. Is there a
reason why ioctl is not?

Sorry if this has been asked or addressed before but I'm at a loss as to
how to search these mailing lists.

[1] https://github.com/apple/swift/pull/196
[2] https://github.com/apple/swift/pull/413
[3] https://github.com/apple/swift-evolution/pull/38


(Xi Ge) #2

I'm beginning to try to port some of my existing C code over to Swift
(you have no idea how excited that makes me). Unfortunately, I've hit a
stumbling block.

When interacting with some of the system/hardware interfaces in Linux it
sometimes (often?) becomes necessary to interact with I/O Control
(ioctl). This introduces two complexities that at least appear to me as
needing resolution.

The first is C macros. A lot ioctl interfaces use headers which define
macros that are specific ioctl request operations. These macros
typically take the form of `_IO`, `_IOR` or `_IOW` (there are more but
this should give you a picture). These values are typically bound to a
specific kernel version and thus are subject to change. I point that out
because obviously one could just calculate the value and hard-code it
into the Swift source (which is what I plan to do in the short term) but
this isn't really a portable solution across kernels. The question is
what, if anything, is planned around this?

The second is variadic C functions. In my case I went ahead and
calculated the result of a C macro and hard coded it into a Swift
variable (e.g., `let DMX_SET_BUFFER_SIZE: CUnsignedInt = 28461`). This
was so I could send it to an ioctl call (e.g., `ioctl(fileReference!,
DMX_SET_BUFFER_SIZE, DVBDVR.DVR_BUFFER_SIZE)`). Unfortunately, this died
in the compiler with `error: 'ioctl' is unavailable: Variadic function
is unavailable`. The interesting part of the compiler error was this
though `SwiftGlibc.ioctl:2:13: note: 'ioctl' has been explicitly marked
unavailable here`. This seems to imply that it _could_ be on and it just
isn't. I searched around a bit and found a few PRs [1] [2] [3] that seem
to indicate that Swift does support variadic C functions. Is there a
reason why ioctl is not?

Though generally unavailable as variadic C functions (Jordan might be the better person to explain why),
we still have other reasons to import them as unavailable functions instead of ignoring them. One example is for printing
the complete interface of the underlying Clang module. This message does NOT imply the variadic function is available
in certain circumstances. Do you any suggestions about better error message?

···

On Jan 5, 2016, at 11:54 AM, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:

Sorry if this has been asked or addressed before but I'm at a loss as to
how to search these mailing lists.

[1] https://github.com/apple/swift/pull/196
[2] https://github.com/apple/swift/pull/413
[3] https://github.com/apple/swift-evolution/pull/38
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Ryan Lovelett) #3

Response inline:

>
> I'm beginning to try to port some of my existing C code over to Swift
> (you have no idea how excited that makes me). Unfortunately, I've hit a
> stumbling block.
>
> When interacting with some of the system/hardware interfaces in Linux it
> sometimes (often?) becomes necessary to interact with I/O Control
> (ioctl). This introduces two complexities that at least appear to me as
> needing resolution.
>
> The first is C macros. A lot ioctl interfaces use headers which define
> macros that are specific ioctl request operations. These macros
> typically take the form of `_IO`, `_IOR` or `_IOW` (there are more but
> this should give you a picture). These values are typically bound to a
> specific kernel version and thus are subject to change. I point that out
> because obviously one could just calculate the value and hard-code it
> into the Swift source (which is what I plan to do in the short term) but
> this isn't really a portable solution across kernels. The question is
> what, if anything, is planned around this?
>
> The second is variadic C functions. In my case I went ahead and
> calculated the result of a C macro and hard coded it into a Swift
> variable (e.g., `let DMX_SET_BUFFER_SIZE: CUnsignedInt = 28461`). This
> was so I could send it to an ioctl call (e.g., `ioctl(fileReference!,
> DMX_SET_BUFFER_SIZE, DVBDVR.DVR_BUFFER_SIZE)`). Unfortunately, this died
> in the compiler with `error: 'ioctl' is unavailable: Variadic function
> is unavailable`. The interesting part of the compiler error was this
> though `SwiftGlibc.ioctl:2:13: note: 'ioctl' has been explicitly marked
> unavailable here`. This seems to imply that it _could_ be on and it just
> isn't. I searched around a bit and found a few PRs [1] [2] [3] that seem
> to indicate that Swift does support variadic C functions. Is there a
> reason why ioctl is not?

Though generally unavailable as variadic C functions (Jordan might be the
better person to explain why),
we still have other reasons to import them as unavailable functions
instead of ignoring them. One example is for printing
the complete interface of the underlying Clang module. This message does
NOT imply the variadic function is available
in certain circumstances. Do you any suggestions about better error
message?

Not exactly the take-away that I was expecting to my question. That
having been said, the part that made my feel like it _could_ and wasn't
was the note. What is the point of the note, if not to say, "Hey we know
this is a variadic function and that's why we've disabled it."? Which to
me implicitly says this could be enabled in the future just, for
whatever reason, not right now. Therefore, my recommendation would be
just to drop the note. The compiler error was enough to say variadic C
functions are unavailable.

>
> Sorry if this has been asked or addressed before but I'm at a loss as to
> how to search these mailing lists.
>
> [1] https://github.com/apple/swift/pull/196
> [2] https://github.com/apple/swift/pull/413
> [3] https://github.com/apple/swift-evolution/pull/38
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev

Just to be clear though the intent of my question was not to quibble
with compiler error messages. My real question is how are we meant to do
systems programming with Swift on Linux if we cannot call ioctl?

···

On Tue, Jan 5, 2016, at 03:12 PM, Xi Ge wrote:

> On Jan 5, 2016, at 11:54 AM, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:


(Kate Stone) #4

In the absence of an automatic mechanism for importing the definition of variadic functions you can still define your own prototypes and bind them to the system implementation. For example, this declaration:

@_silgen_name("ioctl") func ioctl(fildes: CInt, request: UInt64, result: UnsafePointer<Int>) -> Int

… gives you a non-variadic interface to ioctl that you can use for invocations that conform to this specific convention. You can define as many overloads as you wish, and so long as you’re cautious about which one you’re using for a given request you should be able to make progress.

The same basic strategy can be applied to any variadic functional interfaces. Ideally you’d want to hide this implementation detail behind a more Swift-friendly API where the request type is implied to create a more type-safe interface.

Kate Stone k8stone@apple.com <mailto:k8stone@apple.com>
 Xcode Low Level Tools

···

On Jan 5, 2016, at 12:32 PM, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:

Just to be clear though the intent of my question was not to quibble
with compiler error messages. My real question is how are we meant to do
systems programming with Swift on Linux if we cannot call ioctl?


(Ryan Lovelett) #5

First off thanks for the help.

No doubt I'm doing something wrong. I wish I could help debug my own
problem but I don't really get what @_silgen_name does. That seems like
black magic to me.

That compiled though now it won't link. I've put the code up as a
Gist here[1].

$ swift build Linking Executable: .build/debug/CastableLive /home/ryan/Source/castable-
live/.build/debug/CastableLive.o/main.swift.o: In function
`_TFC12CastableLive6DVBDVRcfT7adapterSi6numberSi_GSqS0__': /home/ryan/Source/castable-
live/main.swift:40: undefined reference to `ioctl' /usr/bin/ld: /home/ryan/Source/castable-
live/.build/debug/CastableLive: hidden symbol `ioctl' isn't defined
/usr/bin/ld: final link failed: Bad value clang-3.7: error: linker
command failed with exit code 1 (use -v to see invocation) <unknown>:0:
error: link command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: build had 1 command failures error: exit(1): ["/usr/bin/swift-build-
tool", "-f", "/home/ryan/Source/castable-
live/.build/debug/CastableLive.o/llbuild.yaml"]

···

On Tue, Jan 5, 2016, at 04:57 PM, Kate Stone wrote:

On Jan 5, 2016, at 12:32 PM, Ryan Lovelett via swift-dev <swift- >> dev@swift.org> wrote:

Just to be clear though the intent of my question was not to quibble
with compiler error messages. My real question is how are we meant to
do systems programming with Swift on Linux if we cannot call ioctl?

In the absence of an automatic mechanism for importing the definition
of variadic functions you can still define your own prototypes and
bind them to the system implementation. For example, this
declaration:

@_silgen_name("ioctl") func ioctl(fildes: CInt, request: UInt64,
result: UnsafePointer<Int>) -> Int

… gives you a non-variadic interface to ioctl that you can use for
invocations that conform to this specific convention. You can define
as many overloads as you wish, and so long as you’re cautious about
which one you’re using for a given request you should be able to make
progress.

The same basic strategy can be applied to any variadic functional
interfaces. Ideally you’d want to hide this implementation detail
behind a more Swift-friendly API where the request type is implied to
create a more type-safe interface.

Kate Stonek8stone@apple.com  Xcode Low Level Tools

Links:

  1. https://gist.github.com/anonymous/4861aba2280251fe57c6


(Joe Groff) #6

Don't do this. User code never has any business using underscored attributes. @_silgen_name produces an external reference to a Swift function, with Swift's calling convention. It cannot be safely used to refer to C functions, especially not variadic ones, since the convention for variadics in C is also different from non-variadic C functions. For variadic functions, you should write non-variadic wrappers in C (which can be static inline, to avoid wrapping overhead in production builds) and import them as Clang modules. See how open and fcntl are exported by the Darwin and Glibc overlays for an example.

-Joe

···

On Jan 5, 2016, at 1:57 PM, Kate Stone via swift-dev <swift-dev@swift.org> wrote:

On Jan 5, 2016, at 12:32 PM, Ryan Lovelett via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Just to be clear though the intent of my question was not to quibble
with compiler error messages. My real question is how are we meant to do
systems programming with Swift on Linux if we cannot call ioctl?

In the absence of an automatic mechanism for importing the definition of variadic functions you can still define your own prototypes and bind them to the system implementation. For example, this declaration:

@_silgen_name("ioctl") func ioctl(fildes: CInt, request: UInt64, result: UnsafePointer<Int>) -> Int

… gives you a non-variadic interface to ioctl that you can use for invocations that conform to this specific convention. You can define as many overloads as you wish, and so long as you’re cautious about which one you’re using for a given request you should be able to make progress.

The same basic strategy can be applied to any variadic functional interfaces. Ideally you’d want to hide this implementation detail behind a more Swift-friendly API where the request type is implied to create a more type-safe interface.


(Fastmail) #7

It seems to me that an interesting tool in the swift toolchain would be one that could generate implementations of methods in circumstances like this, using, for instance, CPP macros as hints for function prototypes. Then again, I might be entirely wrong and I'd be happy to be corrected!

Tom

···

Sent from my iPhone

On 5 Jan 2016, at 16:57, Kate Stone <katherine_stone@apple.com> wrote:

On Jan 5, 2016, at 12:32 PM, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:

Just to be clear though the intent of my question was not to quibble
with compiler error messages. My real question is how are we meant to do
systems programming with Swift on Linux if we cannot call ioctl?

In the absence of an automatic mechanism for importing the definition of variadic functions you can still define your own prototypes and bind them to the system implementation. For example, this declaration:

@_silgen_name("ioctl") func ioctl(fildes: CInt, request: UInt64, result: UnsafePointer<Int>) -> Int

… gives you a non-variadic interface to ioctl that you can use for invocations that conform to this specific convention. You can define as many overloads as you wish, and so long as you’re cautious about which one you’re using for a given request you should be able to make progress.

The same basic strategy can be applied to any variadic functional interfaces. Ideally you’d want to hide this implementation detail behind a more Swift-friendly API where the request type is implied to create a more type-safe interface.

Kate Stone k8stone@apple.com
 Xcode Low Level Tools


(Kate Stone) #8

Apologies for the poor guidance. I had success with this path much earlier in Swift’s evolution and I’m clearly not up to date on the latest recommendations. Joe’s approach is definitely the right way to go.

Kate Stone k8stone@apple.com <mailto:k8stone@apple.com>
 Xcode Low Level Tools

···

On Jan 6, 2016, at 11:18 AM, Joe Groff via swift-dev <swift-dev@swift.org> wrote:

On Jan 5, 2016, at 1:57 PM, Kate Stone via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

On Jan 5, 2016, at 12:32 PM, Ryan Lovelett via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Just to be clear though the intent of my question was not to quibble
with compiler error messages. My real question is how are we meant to do
systems programming with Swift on Linux if we cannot call ioctl?

In the absence of an automatic mechanism for importing the definition of variadic functions you can still define your own prototypes and bind them to the system implementation. For example, this declaration:

@_silgen_name("ioctl") func ioctl(fildes: CInt, request: UInt64, result: UnsafePointer<Int>) -> Int

… gives you a non-variadic interface to ioctl that you can use for invocations that conform to this specific convention. You can define as many overloads as you wish, and so long as you’re cautious about which one you’re using for a given request you should be able to make progress.

The same basic strategy can be applied to any variadic functional interfaces. Ideally you’d want to hide this implementation detail behind a more Swift-friendly API where the request type is implied to create a more type-safe interface.

Don't do this. User code never has any business using underscored attributes. @_silgen_name produces an external reference to a Swift function, with Swift's calling convention. It cannot be safely used to refer to C functions, especially not variadic ones, since the convention for variadics in C is also different from non-variadic C functions. For variadic functions, you should write non-variadic wrappers in C (which can be static inline, to avoid wrapping overhead in production builds) and import them as Clang modules. See how open and fcntl are exported by the Darwin and Glibc overlays for an example.

-Joe

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Jordan Rose) #9

Hi, all. I'll try to shed some light on both macros and variadic functions.

Macros are the easy ones. Obviously, a macro like this can't be imported as a function:

#define UNIQUE_NAME(prefix) prefix##__COUNTER__

But here's a trickier one:

#define NAME(windowStruct) windowStruct.attrs->name

In this case, there's probably a pair of structs in mind, one with an 'attrs' field, and one with a 'name' field. But the compiler can't know that just from the macro. In fact, the macro may be intended for use over many structs, all with an 'attrs' field (much like C++ templates). Swift can't express this.

We'd definitely be glad to improve our handling of macros that are unambiguous, but importing all macros is not a goal we can ever reach. In most cases, the best workaround is to write a static inline C function that just calls through to the macro.

···

---

Okay, C variadics. The main problem with C variadics is that they're antithetical to Swift's idea of memory safety. I mean, sure, calling any C function means you're stepping outside Swift's safety, but at least the call is well-typed, and if you're trafficking in UnsafePointer, the function says so (and you probably had to make or get one at the call site). With a variadic function, you can pass total garbage and not be aware of it, and the function will just crash.

A practical problem with C variadics is that they're at odds with Swift's variadics model. Swift variadics, like C# or Java, specify a particular type for all of the variadic arguments—they're all Strings, or Ints, or AnyObjects. C variadics, on the other hand, allow a mix of types, and they're all passed differently. Now, we can certainly do this—we do have Clang, after all—but it's Yet Another Difference between Swift and C that people might have to think about. (For variadics with null sentinels <https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-g_t_0040code_007bsentinel_007d-function-attribute-3219>, does the compiler insert the sentinel? Does it check whether one of the values is nil? What if it's alternating value-key pairs <https://developer.apple.com/library/mac/documentation/Cocoa/Reference/Foundation/Classes/NSDictionary_Class/index.html#//apple_ref/occ/instm/NSDictionary/initWithObjectsAndKeys:>?)

That's mostly it. They're even less safe than the rest of C, and they make things more complicated in Swift. As of Swift 2.2 we do at least import them as unavailable instead of missing entirely, so you get a reasonable error message (as Ryan noticed). It's a lot easier to do that than to actually implement it, though; we basically fake it with a Swift variadic function that has the same signature.

We do import the 'va_list' type, and there's a helper in the standard library, 'withVaList', to safely build a va_list that you can pass to a C function. That's mostly intended for providing a Swift variadic interface that calls through to the underlying implementation. (It's very platform-specific code.) If a function doesn't have a va_list variant, however, Joe's right that the only correct workaround today is, again, a static inline C function. (If you look at the particular pull requests Ryan called out, you'll see they all shim to a C function to do exactly this.)

I hope this clears up some of the issues in both areas. They're not happy answers, and like everything there's probably room for improvement, but they are areas with enough complexity that we'd need to be careful about changing them.

Jordan

On Jan 6, 2016, at 16:28 , Thomas Catterall via swift-dev <swift-dev@swift.org> wrote:

It seems to me that an interesting tool in the swift toolchain would be one that could generate implementations of methods in circumstances like this, using, for instance, CPP macros as hints for function prototypes. Then again, I might be entirely wrong and I'd be happy to be corrected!

Tom

Sent from my iPhone

On 5 Jan 2016, at 16:57, Kate Stone <katherine_stone@apple.com <mailto:katherine_stone@apple.com>> wrote:

On Jan 5, 2016, at 12:32 PM, Ryan Lovelett via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Just to be clear though the intent of my question was not to quibble
with compiler error messages. My real question is how are we meant to do
systems programming with Swift on Linux if we cannot call ioctl?

In the absence of an automatic mechanism for importing the definition of variadic functions you can still define your own prototypes and bind them to the system implementation. For example, this declaration:

@_silgen_name("ioctl") func ioctl(fildes: CInt, request: UInt64, result: UnsafePointer<Int>) -> Int

… gives you a non-variadic interface to ioctl that you can use for invocations that conform to this specific convention. You can define as many overloads as you wish, and so long as you’re cautious about which one you’re using for a given request you should be able to make progress.

The same basic strategy can be applied to any variadic functional interfaces. Ideally you’d want to hide this implementation detail behind a more Swift-friendly API where the request type is implied to create a more type-safe interface.

Kate Stone k8stone@apple.com <mailto:k8stone@apple.com>
 Xcode Low Level Tools

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev