Coalescing Concurrency Documentation

Jon_Shier · 2023-01-12T22:54:50.926Z

Currently, in depth documentation of Swift concurrency features is limited to the various evolution proposals which make up the feature. By my count there are at least 15 proposals which impact this feature, making it quite difficult to track down all of the interactions. This is especially true for things which are defined in earlier proposals which interact with older proposals, as those early proposals make no mention of the later feature, and the later proposals assumes knowledge of earlier proposal. Additionally, those earlier proposals aren't updated as the overall feature changes, making it difficult to put things together as the language changes.

Given the complexity of the Swift concurrency feature, I'd like to see if there's room for an official document that takes most of the content and examples from the proposals and coalesces them into a single document which can be updated as the concurrency feature changes (new APIs, new defaults, new compiler options for safety checking, etc.).

For example, to answer a question like "Where do Task initialization closures execute, and how does that change when the closure captures a global actor?" you need to refer to multiple proposals. You need to know about actors (SE-0306), global actors (SE-0316), closures in concurrency (SE-0302), structured concurrency (SE-0304), and look at the source code to find the undocumented @_inheritActorContext that plays a key role. Given these proposals aren't interlinked, it's essentially impossible for anyone who wasn't already familiar with those documents to track that information down, and every time I do track it down it takes quite a while.

So might there be a place for official documentation like this?

ktoso · 2023-01-13T05:19:46.093Z

In my personal opinion that "place" should be the following document: Concurrency — The Swift Programming Language (Swift 5.7) within the Swift Book.

The way it is edited and maintained has been a little unclear in the past. Since the Swift book has now moved to an open source process, I believe clarifying this and adding much more content and details could be possible, although the process and how we shall approach that is governed by the @swift-documentation-workgroup.

I know that before accepting any new content @Alex_Martini wanted to complete a cleanup pass of grammar and stylistic changes (see this thread: TSPL review pass). It seems we're still in the phase of moving over, and not really accepting new content AFAICS?

I'd love to understand if and how we can provide more content, of the kind you asked for here, into this documentation. I'm more than willing and happy to add content about the more advanced patterns, features, or things the doc currently does not mention at all. We're also missing documentation for distributed actors there, as well as new (discarding) task groups.

So, if there's a process in the works to make it easier for proposal authors to contribute into this documentation -- I'd like to contribute right along with developing those features -- especially as their specific and tricky bits are still fresh in my (or any other proposal authors) mind

Karl · 2023-01-13T05:35:23.139Z

Also, there's a lot of really useful information in the WWDC talks - often going well beyond the written documentation. It makes some sense - the language development is mostly driven by Apple engineers, so that's the venue they choose to demonstrate and explain new language features in depth. As a result, those WWDC videos are really useful for people learning the Swift language. They are a kind of documentation, IMO.

Yet Apple sometimes remove videos from their own site, and send lawyers after people who mirror them. It would be nice if that content was made permanently available.

Jon_Shier · 2023-01-13T06:15:14.302Z

The Swift book doesn't currently contain documentation on the same level of detail as the proposals themselves. The sort of documentation I'm thinking of doesn't really exist in the project at all as far as I can tell. There's only the book, the inline docs, and the proposals. This hasn't really been a problem until now, but the concurrency feature is so large it's difficult to assemble answers manually. Now, we could put more detail into the book, but it seemed like that was specifically avoided. Or perhaps we can just start by fleshing out the concurrency section in the book and see where that leaves us?

ktoso · 2023-01-13T08:17:36.016Z

The full amount of detail a proposal contains would be too much, but we're missing some important information in the book right now, so it definitely warrants extending it. I recently noticed we don't define what a structured task really is in the Swift book, thus, people are very confused and think Task{} is "structured" while it's not. It's not their fault for misunderstanding this though, we're not doing a good job in the swift book, or api docs at explaining this. The best place would be really additional detail in the Swift book.

We have a nice diagram explaining all this, but hidden deep in WWDC talks, which many people don't find or miss.

tgoyne · 2023-01-13T17:56:10.195Z

ktoso:

The full amount of detail a proposal contains would be too much

I think I disagree with this for the specific case of Concurrency. For most features the proposals go into far more detail than the documentation should, but my experience with trying to use concurrency is that even the proposals aren't detailed enough for me to understand why my code behaves the way it does, and I've had to regularly consult the standard library and compiler source.

Jon_Shier · 2023-01-13T17:59:41.127Z

That is my thought as well. While enhancing the book should be done, what I'd really like to accomplish here is to provide a document that allows users to reason about the concurrency system, set their expectations, and refer back when observed behavior doesn't match.

Currently, Swift's concurrency features are rather low level. To me that means it need rather low level documentation.

taylorswift · 2023-01-13T19:54:07.468Z

ktoso:

The full amount of detail a proposal contains would be too much

swift concurrency is really complicated and a lot of times the only way i can answer questions like:

what happens to an async let if you return without awaiting on it, or
what happens if i use multiple awaits per line:

func foo() async
{
}

async let x:Void = ...
await (x, foo())

for myself is to read through the details of the individual proposals. and sometimes the proposals themselves are underspecified or just plain wrong about the actual behavior of swift concurrency, like how SE-0317 specified implicit cancellation and awaiting on anonymous async lets, but from swift 5.5 through 5.7 there was no such thing as an anonymous async let in the first place.

taylorswift · 2023-01-14T22:13:54.042Z

to yield another anecdote into this stream of feedback, i spent the better part of a day trying to figure out the answer to this question, which i only found after digging up this thread and sifting through a discussion that mostly consists of interpersonal conflict to find the information i needed.

as a low-hanging action item, can we take @John_McCall ’s reply here:

The closure is executed synchronously, without allowing any interleaving on the actor; your first code is correct.

and put it somewhere more accessible?