What’s next for Foundation

Development Core Libraries
104

As a general rule, Swift Concurrency should be built on high-performance platform primitives with an eye towards clean interoperation with normal platform code. I am far from an Android expert, but I’m perfectly willing to believe that that means this Looper thing there and should be the default configuration. @compnerd might be able to verify that this is the right thing to do.

There might also be a platform thread pool that we should be using instead of Dispatch. As long as it’s limited, it should be fine.

If you can find a way to rewrite your DispatchQueue.main code in terms of MainActor, you’d be more portable and might be in a good position to cut the Dispatch dependency entirely, which I imagine would be a boon for you.

6 Likes
105

That's about as primitive and high-performance as we can get on Android AFAIK. The alternative would be using the JNI and calling into Java, which I wouldn't be too happy about (especially as a PR on swift).

I'm just not sure about things like Task.sleep or other delayed Jobs – does Swift Concurrency deal with that transparently? Or would the implementation here need to check the desired execution time of job and delay its execution "manually"?

There is no way to do MainActor.sync AFAIK – that's the functionality we'd be missing. Other than that we've been able to remove Dispatch almost entirely (although we have had some weird cases where e.g. Task { @MainActor doesn't appear to run on the main thread, which is concerning).

1 Like
106

If there's some timer facilities you would prefer to use on android generally you may be able to do this by implementing a custom Clock that would be preferable then: swift/Clock.swift at main · apple/swift · GitHub

I've done so on a different C++ concurrency runtime, where we use a custom clock to schedule on an existing thing, rather than using the defaults Swift uses.

PS: This seems like a productive sub-thread but is somewhat off topic for the "what's next foundation", maybe worth splitting it out?

3 Likes
107

Thanks @ktoso for your reply! I’m not sure I understand the idea about using a custom clock. I don’t think I need a specific type of timing, I’d just like to ensure that Task.sleep works with the suggestion I made above (and maybe get some tips on how this works otherwise if it doesn’t). I will have to try it and see I guess.

I agree that it’d be nice to have a separate thread about this. I actually tried starting two separate threads about this recently but didn’t get any response:

Not sure if I posted those in the wrong place or something but if neither of those topics are appropriate I’d be happy to start another :slight_smile: The conversation here has certainly changed my mindset about what the correct approach would be; another difference is that Dispatch is out of the picture now too.

108

To add to this: unfortunately, CF/NS RunLoop was never exposing the "wake up port" to be observed like well established APIs do (libinput, wayland, gtk, qt, etc etc). Most of them are using only public API of periodically run runloop like you've explained earlier and I've never seen anyone using the _dispatch_get_main_queue_port_4CF private API as an alternative.

Because of this for my project I'm using Foundation's runloop as main loop and integrate others into it, since all of those expose the file descriptor to watch whenever events arrive on those loops. This way I don't have to use private API. Unfortunately, on linux (and probably on android too) this would not work properly either because of SR-14920. I've made a PR addressing the issue and porting CFFileDescriptor to make the usage more convenient.

With that said, I don't think what you are looking for can and will be addressed at any point in future. Android's Looper does not expose it's wake file descriptor either, just like CFRunLoop does not. Here's the thread I was asking pretty much the same thing that you need (arguably in opposite way) and reply there is basically same as here: runloop is dead

1 Like
109

Hi @John_McCall, thanks a lot for your ideas so far. I implemented the suggestion I posted above today and it’s all building, taking some inspiration from @kateinoigakukun’s implementation in JavaScriptKit. Basically, the integration between the Android and Swift runtimes looks like it should work.

The issue is, the swift_task_enqueueMainExecutor_hook is not being called. I set that pointer to my override, add a task to the main actor (Task.detached { @MainActor in doSomething() } and the hook is never called (nor is the task of course). Does it only work if I specifically disable Dispatch while building Swift?

1 Like
110

It shouldn't require that, no. You'll have to debug it; if you have more questions, let's take it to a dedicated thread in Development.

1 Like
111

Will conversion of decimal strings into currencies be addressed in the Internationalization API?

112

My understanding is that no new API is being suggested here.

But perhaps the process for including new API in the future Foundation libraries will open up, so everyone could contribute with feature proposals.

2 Likes
114

I'm late to this party, it seems, but this is incredibly exciting. One question I have is: can we make pitches for things to be implemented in these new Foundation-successor frameworks via the standard Evolution process?

Specifically, given this particular quote from the original post:

I've got a working, performant, fully unit-tested progress implementation based on structured concurrency, which I'd love to submit for your consideration if you're interested in it:

3 Likes
115

Thanks @CharlesS, I will take a look. In the short term we’re really focused on the fundamental types. After we get the basics down we will certainly be discussing API like Progress (and beyond).

3 Likes
116

I just remember what I want for next Foundation and keep a record of it. :nerd_face:

I want Foundation to add a protocol such as StreamProtocol that could be like Ruby's IO.
I suppose its conforming types are:

  • Stream class
  • Pipe class
  • Classes related to sockets
  • FileHandle class
  • ...and so on

Backgrounds

5 Likes
118

Try this snippet:

    let url: URL = ...
    Task {
        print("start")
        for try await line in url.lines {
            print(line)
        }
        print("done")
    }

or without an explicit Task if you are inside an async function already.

4 Likes
120

What I usually do is wire up TextOutputStream to the C standard I/O library, as shown by the snippet below.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

extension CommandLine {

    static var stdout = CFileOutputStream(file: Darwin.stdout)
    static var stderr = CFileOutputStream(file: Darwin.stderr)
    
    struct CFileOutputStream: TextOutputStream {
        let file: UnsafeMutablePointer<FILE>
        mutating func write(_ string: String) {
            string.withCString { cStr in
                _ = fputs(cStr, self.file)
            }
        }
    }
}

print("Hello Cruel World!", to: CommandLine.stdout)
5 Likes
122

Logically this:

await url.lines += "\(name), (age), \(hobby)"

but that doesn't exist :grinning:

A tangent about write

Compared to read, write operation could be used in an async "fire and forget" manner even if the API is "sync" looking - you write some bytes, perhaps they will stay in a memory buffer for quite some time and eventually hit the disk. Typically you are not interested in the result of the write operation (I/O error? disk full? permission error? disk is write protected? removable media ejected? network server disconnected? who cares? :rofl:), but, seriously, even if you do care – buffering makes it problematic for you to see some of those errors, unless the error occurs during the buffering phase itself ("not enough memory to put your bytes into the buffer"), or unless you can tolerate knowing about the error later on, e.g. during the next "write" call or via some external delegate/notification mechanism. If you look at it from another angle, even a mere memory access could result into one of those errors on VM page miss and obviously we don't (and can't) check every memory access for those I/O errors.

123

We can generalize a string writer / line scanner if we had, say, StreamProtocol.
(I have a complaint about current TextOutputStream because its func write(_:) is not throwable.)
My humble idea is like below:

public protocol WritableStream {
  mutating func write<T>(contentsOf data: T) throws where T: DataProtocol
  // Other requirements?
}

public protocol ReadableStream {
  mutating func read(upToCount count: Int) throws -> Data?
  // Other requirements?
}

public typealias StreamProtocol = WritableStream & ReadableStream

extension WritableStream {
  public mutating func write(_ string: String, using encoding: String.Encoding = .utf8) throws {
    // Implementation here (able to use `write(contentsOf:)`)
  }
}

public class LineScanner {
  private var _buffer: Data = .init()
  private var _stream: any ReadableStream
  public init(reading stream: any ReadableStream) {
    self._stream = stream
  }

  public func nextLine() throws -> String? {
    // Read bytes to search CR/LF/CR+LF/EOF with `read(upToCount:)` using buffer.
  }
}

Now, something controversial comes to my mind.
I guess the reason why stdin, stdout, and stderr are instances of FileHandle is just because of historical one. Such concept would fit OOP.
However, we can do protocol-oriented programming in Swift.
Can't we reconsider that hierarchy?

For example,

public class StandardInput: ReadableStream {
  public static var default: StandardInput { /* singleton */ }
}
public class StandardOutput: WritableStream {
  public static var default: StandardOutput { /* singleton */ }
}
public class StandardError: WritableStream {
  public static var default: StandardError { /* singleton */ }
}
open class FileHandle: StreamProtocol {
  // :
}

Hmm, I'm not sure what should be better.

124

With an eye on move-only types and consuming functions, I don't think FileHandle should be a class at all. And I'd argue the same for stdin, stdout and stderr. While in current-day Swift, it makes a lot of sense, since Foundation isn't going to be released today I'd consider designs for the features we will most likely have.

4 Likes
125

I'd like to add that I've seen, heard and experience a strong need for a new type, similar to Data, that we can use as a general Bag of Bytes type. I wonder if this is a better for for stdlib as opposed to Foundation, but think that both should be considered and discussed.

5 Likes
127

Being a class does have one benefit for FileHandle though, and that's that it is possible for it to close the file descriptor in its deinit method. Once you take that away and turn FileHandle into a straight wrapper around a file descriptor, it starts looking similar enough to System.FileDescriptor that I'm not certain what purpose it serves to have it be a separate entity.

128

A non-copyable FileHandle struct will have a deinit method.

10 Likes