Libdispatch epoll port


(Dzianis Fedarenka) #1

I have some experience in event multiplexing programming for linux. So it looks like interesting project for me. There is some conceptual questions which I think should be discussed:

1) Obviously, kqueue and epoll have a little different semantics. For example: in linux timers, signals and socket can be presented as file descriptor and processed uniformly. Is there any chance that community will agree to develop separate API for linux?
2) Does anyone have long term vision about how to inject platform specific code into current implementation of dispatch_source? As far as I’ve read the source it’s heavily bound with kqueue semantics and «#ifdef»-way seems to be completely messy..(

···

On Dec 10, 2015, at 12:42 AM, Joakim Hassila via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:

Hi,

On 8 dec. 2015, at 16:56, Pierre Habouzit <pierre at habouzit.net> wrote:

FWIW, this is my personal, let’s call it enlightened, opinion, based on my knowledge of dispatch and my past extensive system programming experience with Linux before I joined Apple.

I think that long term, the best way to maintain a Linux libdispatch port is to go away from the libkqueue that tries to emulate kqueue fully, where dispatch only needs a small subset of the surface of kqueue. Given how source.c is written today, this is not a very small undertaking, but eventually dispatch source map to epoll_ctl(EPOLLONESHOT) very very well.

That makes sense, could simplify the implementation (and keep thing cleaner). Then the follow up question is of course how to split/manage source.c (as Daniel pointed out there is the merging issue).

we can decide when/if someone tries to tackle it. I humbly recognize that I have no great idea of how to do so.


(Tony Parker) #2

Hi Dzianis,

Hi,

FWIW, this is my personal, let’s call it enlightened, opinion, based on my knowledge of dispatch and my past extensive system programming experience with Linux before I joined Apple.

I think that long term, the best way to maintain a Linux libdispatch port is to go away from the libkqueue that tries to emulate kqueue fully, where dispatch only needs a small subset of the surface of kqueue. Given how source.c is written today, this is not a very small undertaking, but eventually dispatch source map to epoll_ctl(EPOLLONESHOT) very very well.

That makes sense, could simplify the implementation (and keep thing cleaner). Then the follow up question is of course how to split/manage source.c (as Daniel pointed out there is the merging issue).

we can decide when/if someone tries to tackle it. I humbly recognize that I have no great idea of how to do so.

I have some experience in event multiplexing programming for linux. So it looks like interesting project for me. There is some conceptual questions which I think should be discussed:

1) Obviously, kqueue and epoll have a little different semantics. For example: in linux timers, signals and socket can be presented as file descriptor and processed uniformly. Is there any chance that community will agree to develop separate API for linux?

For what it’s worth, we went ahead and based CFRunLoop.c on Linux on top of epoll:

https://github.com/apple/swift-corelibs-foundation/blob/master/CoreFoundation/RunLoop.subproj/CFRunLoop.c

https://github.com/apple/swift-corelibs-foundation/commit/d594de1bdd7f10a558e30b92809420303ded0a6a#diff-9739b4f43fc59b19e677f9e3f835d159

I think it makes total sense for dispatch’s SPI for CF to simply return an eventfd.

- Tony

···

On Dec 17, 2015, at 12:36 PM, Dzianis Fedarenka via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 10, 2015, at 12:42 AM, Joakim Hassila via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:

On 8 dec. 2015, at 16:56, Pierre Habouzit <pierre at habouzit.net> wrote:

2) Does anyone have long term vision about how to inject platform specific code into current implementation of dispatch_source? As far as I’ve read the source it’s heavily bound with kqueue semantics and «#ifdef»-way seems to be completely messy..(
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev


(Pierre Habouzit) #3

it’s exactly what we want for runloop tied queues. The mach port that is used for this on Darwin receives messages only to break out of the mach_msg() call, but the handler of the message is a void function: _dispatch_wakeup_runloop_thread().

The good news is that a mach_port is an uint32_t and eventfd would be an int, so as far as storage is concerned, everything is fine.

I would have the _dispatch_get_main_queue_port_4CF / _dispatch_runloop_root_queue_get_port_4CF return an eventfd, and adapt the code that disposes of it. This is a IMO straightforward patch that should be written e.g. that way:

#if HAVE_MACH
// current OS X Code
#elif HAVE_EVENTFD
// linux port
#else
#error should not happen
#endif

And also have:

DISPATCH_COCOA_COMPAT be set to one on linux (until it is, you don’t get the main queue and runloop tied queues).

The one murky thing is that someone has to *consume* what’s in that eventfd, today, it’s implicit with mach because MiG will call dispatch’s _dispatch_wakeup_runloop_thread() for it (corresponding to the wakeup_runloop_thread routine in protocol.defs), but for linux, it’s probably best if CF knows that it’s an eventfd and it has to eventfd_read() from it to consume the event before it’s calling _dispatch_runloop_root_queue_perform_4CF(). The alternative is for _dispatch_runloop_root_queue_perform_4CF() to do that read in a non blocking way, but for the cases when several things have been queued on the runloop queue and have been coalesced in a single eventfd delivery, it’s a bit dumb to pay a syscall per dequeue.

On Mach the coalescing happens because the port has a queue width of 1 and incoming messages are dropped when the port is full.

-Pierre

···

On Dec 17, 2015, at 12:40 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Hi Dzianis,

On Dec 17, 2015, at 12:36 PM, Dzianis Fedarenka via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 10, 2015, at 12:42 AM, Joakim Hassila via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:

Hi,

On 8 dec. 2015, at 16:56, Pierre Habouzit <pierre at habouzit.net> wrote:

FWIW, this is my personal, let’s call it enlightened, opinion, based on my knowledge of dispatch and my past extensive system programming experience with Linux before I joined Apple.

I think that long term, the best way to maintain a Linux libdispatch port is to go away from the libkqueue that tries to emulate kqueue fully, where dispatch only needs a small subset of the surface of kqueue. Given how source.c is written today, this is not a very small undertaking, but eventually dispatch source map to epoll_ctl(EPOLLONESHOT) very very well.

That makes sense, could simplify the implementation (and keep thing cleaner). Then the follow up question is of course how to split/manage source.c (as Daniel pointed out there is the merging issue).

we can decide when/if someone tries to tackle it. I humbly recognize that I have no great idea of how to do so.

I have some experience in event multiplexing programming for linux. So it looks like interesting project for me. There is some conceptual questions which I think should be discussed:

1) Obviously, kqueue and epoll have a little different semantics. For example: in linux timers, signals and socket can be presented as file descriptor and processed uniformly. Is there any chance that community will agree to develop separate API for linux?

For what it’s worth, we went ahead and based CFRunLoop.c on Linux on top of epoll:

https://github.com/apple/swift-corelibs-foundation/blob/master/CoreFoundation/RunLoop.subproj/CFRunLoop.c

https://github.com/apple/swift-corelibs-foundation/commit/d594de1bdd7f10a558e30b92809420303ded0a6a#diff-9739b4f43fc59b19e677f9e3f835d159

I think it makes total sense for dispatch’s SPI for CF to simply return an eventfd.


(Pierre Habouzit) #4

-Pierre

Hi Dzianis,

Hi,

FWIW, this is my personal, let’s call it enlightened, opinion, based on my knowledge of dispatch and my past extensive system programming experience with Linux before I joined Apple.

I think that long term, the best way to maintain a Linux libdispatch port is to go away from the libkqueue that tries to emulate kqueue fully, where dispatch only needs a small subset of the surface of kqueue. Given how source.c is written today, this is not a very small undertaking, but eventually dispatch source map to epoll_ctl(EPOLLONESHOT) very very well.

That makes sense, could simplify the implementation (and keep thing cleaner). Then the follow up question is of course how to split/manage source.c (as Daniel pointed out there is the merging issue).

we can decide when/if someone tries to tackle it. I humbly recognize that I have no great idea of how to do so.

I have some experience in event multiplexing programming for linux. So it looks like interesting project for me. There is some conceptual questions which I think should be discussed:

1) Obviously, kqueue and epoll have a little different semantics. For example: in linux timers, signals and socket can be presented as file descriptor and processed uniformly. Is there any chance that community will agree to develop separate API for linux?

For what it’s worth, we went ahead and based CFRunLoop.c on Linux on top of epoll:

https://github.com/apple/swift-corelibs-foundation/blob/master/CoreFoundation/RunLoop.subproj/CFRunLoop.c

https://github.com/apple/swift-corelibs-foundation/commit/d594de1bdd7f10a558e30b92809420303ded0a6a#diff-9739b4f43fc59b19e677f9e3f835d159

I think it makes total sense for dispatch’s SPI for CF to simply return an eventfd.

it’s exactly what we want for runloop tied queues. The mach port that is used for this on Darwin receives messages only to break out of the mach_msg() call, but the handler of the message is a void function: _dispatch_wakeup_runloop_thread().

The good news is that a mach_port is an uint32_t and eventfd would be an int, so as far as storage is concerned, everything is fine.

I would have the _dispatch_get_main_queue_port_4CF / _dispatch_runloop_root_queue_get_port_4CF return an eventfd, and adapt the code that disposes of it. This is a IMO straightforward patch that should be written e.g. that way:

#if HAVE_MACH
// current OS X Code
#elif HAVE_EVENTFD
// linux port
#else
#error should not happen
#endif

And also have:

DISPATCH_COCOA_COMPAT be set to one on linux (until it is, you don’t get the main queue and runloop tied queues).

The one murky thing is that someone has to *consume* what’s in that eventfd, today, it’s implicit with mach because MiG will call dispatch’s _dispatch_wakeup_runloop_thread() for it (corresponding to the wakeup_runloop_thread routine in protocol.defs), but for linux, it’s probably best if CF knows that it’s an eventfd and it has to eventfd_read() from it to consume the event before it’s calling _dispatch_runloop_root_queue_perform_4CF(). The alternative is for _dispatch_runloop_root_queue_perform_4CF() to do that read in a non blocking way, but for the cases when several things have been queued on the runloop queue and have been coalesced in a single eventfd delivery, it’s a bit dumb to pay a syscall per dequeue.

On Mach the coalescing happens because the port has a queue width of 1 and incoming messages are dropped when the port is full.

Actually alternatively this could be done (no error handling for clarity, but it should have some!):

static bool
_dispatch_runloop_queue_drain_one(dispatch_queue_t dq)
{
  if (!dq->dq_items_tail) {
#ifdef __linux__
    eventfd_read((int)dq->do_ctxt, &(eventfd_t)0);
    if (!dq->dq_items_tail) {
      return false;
    }
#else
    return false;
#endif
  }
  ...
}

IOW: consume the eventfd when we think the queue is empty, and check again it really is, that way you even have the eventfd break you out of the epoll while the queue is full which is probably nice (even if CF is supposed to already track this anyway, but I don’t know CFRunloop well so Tony should tell which alternative is better between my former mail or that one).

-Pierre

···

On Dec 17, 2015, at 1:35 PM, Pierre Habouzit <pierre@habouzit.net> wrote:

On Dec 17, 2015, at 12:40 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 17, 2015, at 12:36 PM, Dzianis Fedarenka via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 10, 2015, at 12:42 AM, Joakim Hassila via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:

On 8 dec. 2015, at 16:56, Pierre Habouzit <pierre at habouzit.net> wrote:


(Thomas Greany) #5

What about libev? http://software.schmorp.de/pkg/libev.html

···

On Thu, Dec 17, 2015 at 12:36 PM, Dzianis Fedarenka via swift-corelibs-dev < swift-corelibs-dev@swift.org> wrote:

>> On Dec 10, 2015, at 12:42 AM, Joakim Hassila via swift-corelibs-dev > <swift-corelibs-dev at swift.org> wrote:
>>
>> Hi,
>>
>>> On 8 dec. 2015, at 16:56, Pierre Habouzit <pierre at habouzit.net> > wrote:
>>>
>>> FWIW, this is my personal, let’s call it enlightened, opinion, based
on my knowledge of dispatch and my past extensive system programming
experience with Linux before I joined Apple.
>>>
>>> I think that long term, the best way to maintain a Linux libdispatch
port is to go away from the libkqueue that tries to emulate kqueue fully,
where dispatch only needs a small subset of the surface of kqueue. Given
how source.c is written today, this is not a very small undertaking, but
eventually dispatch source map to epoll_ctl(EPOLLONESHOT) very very well.
>>
>> That makes sense, could simplify the implementation (and keep thing
cleaner). Then the follow up question is of course how to split/manage
source.c (as Daniel pointed out there is the merging issue).
> we can decide when/if someone tries to tackle it. I humbly recognize
that I have no great idea of how to do so.

I have some experience in event multiplexing programming for linux. So it
looks like interesting project for me. There is some conceptual questions
which I think should be discussed:

1) Obviously, kqueue and epoll have a little different semantics. For
example: in linux timers, signals and socket can be presented as file
descriptor and processed uniformly. Is there any chance that community will
agree to develop separate API for linux?
2) Does anyone have long term vision about how to inject platform specific
code into current implementation of dispatch_source? As far as I’ve read
the source it’s heavily bound with kqueue semantics and «#ifdef»-way seems
to be completely messy..(
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev


(Pierre Habouzit) #6

libev model isn’t compatible with dispatch model, and retrofiting one on top of the other will put you in a world of hurt, and would diverge from the OS X port greatly, which we’d like to avoid.

I originally thought you were replying to the thread about having the Runloop queues code in dispatch ported (for some reason your reply wasn’t attached to a mail thread for me and I got confused), and that one needs a raw interface with epoll to interact with CFRunlopp and that’s what I “replied” to.

I don’t think that moving away from likqueue is a good idea yet. Technically, dispatch is really interleaved with kqueue code, and libkqueue overhead is smaller than I originally thought, so I’ve kind of revisited that argument.

The one thing that will be painful for the port to an extent, is that kevents are now uniquified in kernel based on the ident + udata, said another way, if you watch the same file descriptor twice with two different udata’s, then it’s two different kevents in kernel. That is *NOT* the case with epoll, where uniquing is only based on the file-descriptor: if you EPOLL_CTL_ADD two different “udata” for the same fd, you get EEXIST, which means that you need to do multiplexing in userland (which dispatch used to do before kevent started taking udata into account).

That has impacts on the userland parallelism as all source registering and deregistering have to be serialized wrt each other. This is the hard part, not abstracting epoll.

-Pierre

···

On Dec 17, 2015, at 1:44 PM, Thomas Greany via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

What about libev? http://software.schmorp.de/pkg/libev.html

On Thu, Dec 17, 2015 at 12:36 PM, Dzianis Fedarenka via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:
>> On Dec 10, 2015, at 12:42 AM, Joakim Hassila via swift-corelibs-dev <swift-corelibs-dev at swift.org <http://swift.org/>> wrote:
>>
>> Hi,
>>
>>> On 8 dec. 2015, at 16:56, Pierre Habouzit <pierre at habouzit.net <http://habouzit.net/>> wrote:
>>>
>>> FWIW, this is my personal, let’s call it enlightened, opinion, based on my knowledge of dispatch and my past extensive system programming experience with Linux before I joined Apple.
>>>
>>> I think that long term, the best way to maintain a Linux libdispatch port is to go away from the libkqueue that tries to emulate kqueue fully, where dispatch only needs a small subset of the surface of kqueue. Given how source.c is written today, this is not a very small undertaking, but eventually dispatch source map to epoll_ctl(EPOLLONESHOT) very very well.
>>
>> That makes sense, could simplify the implementation (and keep thing cleaner). Then the follow up question is of course how to split/manage source.c (as Daniel pointed out there is the merging issue).
> we can decide when/if someone tries to tackle it. I humbly recognize that I have no great idea of how to do so.

I have some experience in event multiplexing programming for linux. So it looks like interesting project for me. There is some conceptual questions which I think should be discussed:

1) Obviously, kqueue and epoll have a little different semantics. For example: in linux timers, signals and socket can be presented as file descriptor and processed uniformly. Is there any chance that community will agree to develop separate API for linux?
2) Does anyone have long term vision about how to inject platform specific code into current implementation of dispatch_source? As far as I’ve read the source it’s heavily bound with kqueue semantics and «#ifdef»-way seems to be completely messy..(
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev


(Daniel A. Steffen) #7

-Pierre

Hi Dzianis,

Hi,

FWIW, this is my personal, let’s call it enlightened, opinion, based on my knowledge of dispatch and my past extensive system programming experience with Linux before I joined Apple.

I think that long term, the best way to maintain a Linux libdispatch port is to go away from the libkqueue that tries to emulate kqueue fully, where dispatch only needs a small subset of the surface of kqueue. Given how source.c is written today, this is not a very small undertaking, but eventually dispatch source map to epoll_ctl(EPOLLONESHOT) very very well.

That makes sense, could simplify the implementation (and keep thing cleaner). Then the follow up question is of course how to split/manage source.c (as Daniel pointed out there is the merging issue).

we can decide when/if someone tries to tackle it. I humbly recognize that I have no great idea of how to do so.

I have some experience in event multiplexing programming for linux. So it looks like interesting project for me. There is some conceptual questions which I think should be discussed:

1) Obviously, kqueue and epoll have a little different semantics. For example: in linux timers, signals and socket can be presented as file descriptor and processed uniformly. Is there any chance that community will agree to develop separate API for linux?

For what it’s worth, we went ahead and based CFRunLoop.c on Linux on top of epoll:

https://github.com/apple/swift-corelibs-foundation/blob/master/CoreFoundation/RunLoop.subproj/CFRunLoop.c

https://github.com/apple/swift-corelibs-foundation/commit/d594de1bdd7f10a558e30b92809420303ded0a6a#diff-9739b4f43fc59b19e677f9e3f835d159

I think it makes total sense for dispatch’s SPI for CF to simply return an eventfd.

it’s exactly what we want for runloop tied queues. The mach port that is used for this on Darwin receives messages only to break out of the mach_msg() call, but the handler of the message is a void function: _dispatch_wakeup_runloop_thread().

The good news is that a mach_port is an uint32_t and eventfd would be an int, so as far as storage is concerned, everything is fine.

I would have the _dispatch_get_main_queue_port_4CF / _dispatch_runloop_root_queue_get_port_4CF return an eventfd, and adapt the code that disposes of it. This is a IMO straightforward patch that should be written e.g. that way:

#if HAVE_MACH
// current OS X Code
#elif HAVE_EVENTFD
// linux port
#else
#error should not happen
#endif

And also have:

DISPATCH_COCOA_COMPAT be set to one on linux (until it is, you don’t get the main queue and runloop tied queues).

The one murky thing is that someone has to *consume* what’s in that eventfd, today, it’s implicit with mach because MiG will call dispatch’s _dispatch_wakeup_runloop_thread() for it (corresponding to the wakeup_runloop_thread routine in protocol.defs)

actually that is never called, the only thing that is used is the mig server routine _dispatch_send_wakeup_runloop_thread, the client routine is just there so that the mig client code links…

, but for linux, it’s probably best if CF knows that it’s an eventfd and it has to eventfd_read() from it to consume the event before it’s calling _dispatch_runloop_root_queue_perform_4CF(). The alternative is for _dispatch_runloop_root_queue_perform_4CF() to do that read in a non blocking way, but for the cases when several things have been queued on the runloop queue and have been coalesced in a single eventfd delivery, it’s a bit dumb to pay a syscall per dequeue.

On Mach the coalescing happens because the port has a queue width of 1 and incoming messages are dropped when the port is full.

Actually alternatively this could be done (no error handling for clarity, but it should have some!):

static bool
_dispatch_runloop_queue_drain_one(dispatch_queue_t dq)
{
  if (!dq->dq_items_tail) {
#ifdef __linux__
    eventfd_read((int)dq->do_ctxt, &(eventfd_t)0);
    if (!dq->dq_items_tail) {
      return false;
    }
#else
    return false;
#endif
  }
  ...
}

IOW: consume the eventfd when we think the queue is empty, and check again it really is, that way you even have the eventfd break you out of the epoll while the queue is full which is probably nice (even if CF is supposed to already track this anyway, but I don’t know CFRunloop well so Tony should tell which alternative is better between my former mail or that one).

for the main queue, the current runloop / libdispatch interaction model is that __CFRunLoopRun() dequeues the message from the port obtained from _dispatch_get_main_queue_port_4CF(), and that _dispatch_main_queue_callback_4CF() consumes the message in question (aside from its storage).
If we wanted to call the mig client function mentioned above, this is where we would do it.
The main queue is then drained partially or completely, with another wakeup message generated in the partial case.

It seems that this model would make sense for eventfd as well and would keep the divergence to a minimum, though you’d likely need to emulate the coalescing behavior of the qlimit 1 wakeup port by dequeueing all additional wakeup messages on the eventfd, probably in _dispatch_main_queue_callback_4CF

for the runloop queues, the model is slightly different, the wakeup messages are supposed to be consumed by __CFRunLoopRun() and _dispatch_runloop_queue_drain_one called in a loop until it returns false, without any additional wakeup messages being generated.

Daniel

···

On Dec 17, 2015, at 13:41, Pierre Habouzit via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 17, 2015, at 1:35 PM, Pierre Habouzit <pierre@habouzit.net> wrote:

On Dec 17, 2015, at 12:40 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 17, 2015, at 12:36 PM, Dzianis Fedarenka via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 10, 2015, at 12:42 AM, Joakim Hassila via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:

On 8 dec. 2015, at 16:56, Pierre Habouzit <pierre at habouzit.net> wrote:


(Daniel A. Steffen) #8

-Pierre

Hi Dzianis,

Hi,

FWIW, this is my personal, let’s call it enlightened, opinion, based on my knowledge of dispatch and my past extensive system programming experience with Linux before I joined Apple.

I think that long term, the best way to maintain a Linux libdispatch port is to go away from the libkqueue that tries to emulate kqueue fully, where dispatch only needs a small subset of the surface of kqueue. Given how source.c is written today, this is not a very small undertaking, but eventually dispatch source map to epoll_ctl(EPOLLONESHOT) very very well.

That makes sense, could simplify the implementation (and keep thing cleaner). Then the follow up question is of course how to split/manage source.c (as Daniel pointed out there is the merging issue).

we can decide when/if someone tries to tackle it. I humbly recognize that I have no great idea of how to do so.

I have some experience in event multiplexing programming for linux. So it looks like interesting project for me. There is some conceptual questions which I think should be discussed:

1) Obviously, kqueue and epoll have a little different semantics. For example: in linux timers, signals and socket can be presented as file descriptor and processed uniformly. Is there any chance that community will agree to develop separate API for linux?

For what it’s worth, we went ahead and based CFRunLoop.c on Linux on top of epoll:

https://github.com/apple/swift-corelibs-foundation/blob/master/CoreFoundation/RunLoop.subproj/CFRunLoop.c

https://github.com/apple/swift-corelibs-foundation/commit/d594de1bdd7f10a558e30b92809420303ded0a6a#diff-9739b4f43fc59b19e677f9e3f835d159

I think it makes total sense for dispatch’s SPI for CF to simply return an eventfd.

it’s exactly what we want for runloop tied queues. The mach port that is used for this on Darwin receives messages only to break out of the mach_msg() call, but the handler of the message is a void function: _dispatch_wakeup_runloop_thread().

The good news is that a mach_port is an uint32_t and eventfd would be an int, so as far as storage is concerned, everything is fine.

I would have the _dispatch_get_main_queue_port_4CF / _dispatch_runloop_root_queue_get_port_4CF return an eventfd, and adapt the code that disposes of it. This is a IMO straightforward patch that should be written e.g. that way:

#if HAVE_MACH
// current OS X Code
#elif HAVE_EVENTFD
// linux port
#else
#error should not happen
#endif

And also have:

DISPATCH_COCOA_COMPAT be set to one on linux (until it is, you don’t get the main queue and runloop tied queues).

The one murky thing is that someone has to *consume* what’s in that eventfd, today, it’s implicit with mach because MiG will call dispatch’s _dispatch_wakeup_runloop_thread() for it (corresponding to the wakeup_runloop_thread routine in protocol.defs)

actually that is never called, the only thing that is used is the mig server routine _dispatch_send_wakeup_runloop_thread, the client routine is just there so that the mig client code links…

, but for linux, it’s probably best if CF knows that it’s an eventfd and it has to eventfd_read() from it to consume the event before it’s calling _dispatch_runloop_root_queue_perform_4CF(). The alternative is for _dispatch_runloop_root_queue_perform_4CF() to do that read in a non blocking way, but for the cases when several things have been queued on the runloop queue and have been coalesced in a single eventfd delivery, it’s a bit dumb to pay a syscall per dequeue.

On Mach the coalescing happens because the port has a queue width of 1 and incoming messages are dropped when the port is full.

Actually alternatively this could be done (no error handling for clarity, but it should have some!):

static bool
_dispatch_runloop_queue_drain_one(dispatch_queue_t dq)
{
  if (!dq->dq_items_tail) {
#ifdef __linux__
    eventfd_read((int)dq->do_ctxt, &(eventfd_t)0);
    if (!dq->dq_items_tail) {
      return false;
    }
#else
    return false;
#endif
  }
  ...
}

IOW: consume the eventfd when we think the queue is empty, and check again it really is, that way you even have the eventfd break you out of the epoll while the queue is full which is probably nice (even if CF is supposed to already track this anyway, but I don’t know CFRunloop well so Tony should tell which alternative is better between my former mail or that one).

for the main queue, the current runloop / libdispatch interaction model is that __CFRunLoopRun() dequeues the message from the port obtained from _dispatch_get_main_queue_port_4CF(), and that _dispatch_main_queue_callback_4CF() consumes the message in question (aside from its storage).
If we wanted to call the mig client function mentioned above, this is where we would do it.
The main queue is then drained partially or completely, with another wakeup message generated in the partial case.

It seems that this model would make sense for eventfd as well and would keep the divergence to a minimum, though you’d likely need to emulate the coalescing behavior of the qlimit 1 wakeup port by dequeueing all additional wakeup messages on the eventfd, probably in _dispatch_main_queue_callback_4CF

for the runloop queues, the model is slightly different, the wakeup messages are supposed to be consumed by __CFRunLoopRun() and _dispatch_runloop_queue_drain_one

err, s/_dispatch_runloop_queue_drain_one/_dispatch_runloop_root_queue_perform_4CF/ of course

···

On Dec 17, 2015, at 21:16, Daniel A. Steffen via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 17, 2015, at 13:41, Pierre Habouzit via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 17, 2015, at 1:35 PM, Pierre Habouzit <pierre@habouzit.net> wrote:

On Dec 17, 2015, at 12:40 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 17, 2015, at 12:36 PM, Dzianis Fedarenka via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

On Dec 10, 2015, at 12:42 AM, Joakim Hassila via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:

On 8 dec. 2015, at 16:56, Pierre Habouzit <pierre at habouzit.net> wrote:

called in a loop until it returns false, without any additional wakeup messages being generated.

Daniel

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev