Is anyone working on FileHandle's wait for data in background and notify?

I have a macOS application that I was wanting to port to Linux. It makes use of FileHandle#waitForDataInBackgroundAndNotify which apparently is not yet implemented. I wanted to go ahead and implement that function.

I started reading the CONTRIBUTING.md guide and it says that I should check with the community before working on it to avoid duplication.

Is anyone working on this?

EDIT: I would have liked the title to have been "Is anyone working on FileHandle's waitForDataInBackgroundAndNotify?". This seems to be impossible as it does not pass the complete sentence filter. I get an error, "Title seems unclear, is it a complete sentence?"

1 Like

I was just talking to someone about this earlier today!

As far as I'm aware, noone has started implementing it yet, so it would be a great contribution to the project.

I guess the implemention would use inotify() [1] on Linux? (I haven't checked what waitForDataInBackgroundAndNotify needs, but inotify() handles a lot of notification type functionality)

One strategy would be to add the sys/inotify.h header to the Glibc modulemap [2], then implement the functionality in Swift, calling the inotify() syscalls as needed.

Another strategy would be to write the inotify() using code in C as part of CoreFoundation, then call that from FileHandle.

I'm sure someone else will be along with better ideas than me soon :-)

Thanks,
Ian

[1] inotify(7) - Linux manual page
[2] swift/glibc.modulemap.gyb at main · apple/swift · GitHub

1 Like

AFAIK, NSFileHandle is a file handle that is already opened (ie. an open file descriptor). Therefore I'd suggest epoll on Linux/kqueue on macOS and not inotify as the API. Said that, DispatchSources already provide a platform-agnostic way of doing that so no need to mess with epoll/kqueue directly.

I had together a proof-of-concept implementation based on select. Though I'm interested in any approach that is best.

I'll check out inotify.

I was also not really clear on how I could write the tests. If there would be some good example tests that I should look at.

Thanks, Johannes - I had you in mind when I said "I’m sure someone else will be along with better ideas than me soon :-)"

I was just chatting with Ian about this. I'm in need of readabilityHandler to be implemented as well and we think that the implementation of both things should be quite similar. Maybe we can work together on this. I'm also inexperienced with unit tests.

oh actually, I might be misunderstanding the API. So kqueue/epoll work great on sockets/pipes but not so much on files. If you wanted to wait until a file grows or something, that'd be indeed available with inotify/kqueue.

Ok, I probably should've sorted out my thoughts before replying here, apologies.

So from reading the API docs, I think the main point of this API is to read the file descriptor in the background and report back with the data that was available to be read. It's not waiting for files on disk to grow, so inotify won't be necessary and DispatchSource (or kqueue/epoll directly) will work just fine. Just for the record, kqueue/epoll/select/poll/... will always return readable for regular on-disk files. They only really do a thing for sockets, pipes, etc. But because the FileHandle#waitForDataInBackgroundAndNotify API supports all those things, it should obviously use an eventing mechanism, just to be able to use the same code for files, sockets, pipes, ...

So I guess the easiest possible implementation will be to use DispatchSource.makeReadSource to get notified when something's available to be read. Then on some background thread/queue actually read the data and then report back on the calling 'run loop'. It seems that this API is still defined it terms of NSRunLoops and not DispatchQueues.

On Darwin, this implementation uses dispatch_io to get notification of activity, then signals the run loop to wake it up and indicate activity (because that is indeed the API contract).

Thanks @Tony_Parker, DispatchIO (which is built on top of DispatchSource) is actually even easier than using DispatchSource directly so should be preferred imho.

@Tony_Parker that is good advice. I've already scaffolded something and have been playing with it. Seems to be working the way I'd expect.

import Dispatch

let global = DispatchQueue.global(qos: .background)
let stdinChannel = DispatchIO(type: .stream, fileDescriptor: STDIN_FILENO, queue: global) { (error) in
  guard error == 0 else {
    fatalError("TODO: Actually understand the errors that can happen here and where to put them.")
  }
}

// We want notify when any data is available
stdinChannel.setLimit(lowWater: 1)

stdinChannel.read(offset: 0, length: Int.max, queue: global) { (done, data, error) in
  guard error == 0 else {
    fatalError("TODO: Actually understand the errors that can happen here and where to put them.")
  }
  //  probably send the notification
}

@johannesweiss is there any documentation on what waitForDataInBackgroundAndNotify is supposed to do when an error is encountered? I tried reading the documentation but it seems sparse.

Also, is it true the notification that should be sent, NSFileHandleDataAvailable, really contains no data from the read operation?

It looks like errors from errno are turned into an NSNumber and posted as part of the userInfo dictionary of the notification that is posted:

FOUNDATION_EXPORT NSNotificationName const NSFileHandleReadCompletionNotification;
FOUNDATION_EXPORT NSNotificationName const NSFileHandleReadToEndOfFileCompletionNotification;
FOUNDATION_EXPORT NSNotificationName const NSFileHandleConnectionAcceptedNotification;
FOUNDATION_EXPORT NSNotificationName const NSFileHandleDataAvailableNotification;

with a key of NSFileHandleError. An odd pattern, but this is one of the older classes. I don't see any other way to access it from a brief look.

@rlovelett I think that's all. My interpretation of that is:

  • readInBackgroundAndNotify uses some eventing mechanism and when data is available reads it in the background and calls back with the data (which is pretty much what DispatchIO does)
  • waitForDataInBackgroundAndNotify uses some eventing mechanism and does not read the data from the file descriptor (which is pretty much what DispatchSource (type read) does)

but @Tony_Parker will be able to confirm or correct.

One last question, then I'll start writing code, is there a way to run the compilation of just swift-corelibs-foundation? Or better yet have it generate an Xcode project that I can use to develop with?

I checked the contributing.md file and it was not forthcoming with simple tricks/pointers.

I made an initial PR for this a few days ago. I was hoping for some feedback on the questions I posed in the PR.

Is there something else I need to do get it infront of the right people to receive feedback?

Allright, there’s a lot of things I don’t know in this thread BUT, I can show you two projects that use inotify to get File Events and such on Linux.

The first one’s a fork I made from another project, and I’m not supporting it atm. It works, and it has a small leak, but it works. I’m using it to port SourceKittenDaemon to Linux.

The second one is better and in current development by PonyBoy47. He’s a student like me, and thus he could use some help because of time constraints ;)

PS: I’m glad this topic is on the table. It would be great to have FileWatchers for Swift on Linux :sparkles:

I've already made an implementation that is based on DispatchIO, as suggested by @Tony_Parker. While I'm sure that inotify would work there appears to be a preference towards using DispatchIO.

:thinking: hmm

Well, I’ve never used Mac OS, so I have no idea on how good DispatchIO is.

But to start with... I’d say: whatever works. If you guys add this to CF, however it’s implemented, at least we’ll have the functionality.

That said... If you still want to check how far can inotify go, I’d really suggest getting in touch with PonyBoy47. The guy’s good.