Where can I learn about how closures are implemented?

THV · 2021-05-07T16:20:31.555Z

I would like to study closures more deeply and was thinking I could find documentation or examine the actual code in the Swift git repository but I wasn’t “quickly” finding a good entry point. Can someone point me in the right direction? Thanks for your patience if this should be easy to locate and I just overlooked it.

In fact, is there any kind of document that maps out the organization of the source code or the overall design of Swift and the architecture of its various features? I’ve read a number of the manifestos and the SE threads and proposals. If the answer is just to wade into the source waters and look around, that’s fine, too.

Appreciate the guidance. Thank you.

Kentzo · 2021-05-07T20:10:18.724Z

This recent thread seems helpful: How do closures work (memory management).

THV · 2021-05-07T20:53:02.728Z

I had read that post and it is helpful. It didn’t get me to the finish line in terms of what I’m looking for but it does help with a piece of this puzzle. Thank you. And thank you @stuchlej for a great discussion on your thread.

I should also note that @lukasa’s comment about the underlying struct was particular insightful: How do closures work (memory management) - #2 by lukasa

This is what I’d like to dig more into. This process of capturing the state. Where does it go? How is it captured? The presence of the optional pointer to AnyObject seems to be abstracting away a lot or black magic.

byaruhaf · 2021-05-07T21:17:59.420Z

Hi @THV , you can check out Gosh Darn Closure Syntax

THV · 2021-05-07T21:28:14.235Z

Really nice resource, thank you. I understand how to write closures, though. I’m trying to understand how they are implemented by the compiler. E.g., what are the underlying data structures that make closures work?

Kentzo · 2021-05-08T00:51:49.310Z

I assume underlying idea and implementation of Swift closures is very close to C blocks. There is an old video from WWDC 2010 and Apple's own documentation on blocks. The clang project also documents them.

lukasa · 2021-05-08T06:40:09.457Z

THV:

This is what I’d like to dig more into. This process of capturing the state. Where does it go? How is it captured? The presence of the optional pointer to AnyObject seems to be abstracting away a lot or black magic.

Per @Joe_Groff’s comment in that thread, the closure context is assembled by the compiler, and it can be thought of as a sort of anonymous class. Let’s leave aside the special case of a closure that closes over either a) nothing or b) only one class object for now.

The compiler puts together the collection of values you close over and effectively assembles a class, for example:

class ClosureContextforClosureFoo {
    weak var self: SomeType
    var iterationCount: Int
    var someOtherCapturedState: SomeOtherType
    var anotherClosureToCall: (SomeOtherType, Int) -> Int

At the point you need to call the closure, the compiler will construct this object, by heap allocating its storage and then calling the copy constructor for each of the elements in turn. For classes this means their pointer is copied in and they are swift_retained, for values they’ll be shallow-copied as usual.

This object is then passed around as part of the closure, including being retain/released as normal. In almost every way that matters it’s a little weird anonymous class, except that it doesn’t really have any identity outside of the closure itself.

Below that level, in the compiler itself, I’m not sure. My low-level Swift understanding stops at “what does the compiler produce for a line of code”, I have much less experience with “what does the compiler do to get to that place”. @Joe_Groff et al are more helpful for that.

lukasa · 2021-05-08T06:41:04.699Z

lukasa:

At the point you need to call the closure, the compiler will construct this object, by heap allocating its storage and then calling the copy constructor for each of the elements in turn.

Oh, I should clarify this: the closure context is only heap-allocated if the closure is escaping. Non-escaping closures get a stack-allocated closure context as a really nice optimization.

Closures that close over a single class also do not allocate a context, and closures that close over nothing don’t allocate one either.

Nickolas_Pohilets · 2021-05-08T11:49:32.809Z

IIRC, closures that close over a single collection (which is also just a pointer under the hood) and don’t mutate it, also don’t allocate a context. I was playing only with built-in collections, not sure if this applies to all structs with a single reference-counted field.

THV · 2021-05-08T18:20:21.328Z

@lukasa @Kentzo and @Nickolas_Pohilets - This information and @Kentzo links regarding C Blocks is just what I need. Thank you all.

lukasa · 2021-05-09T09:41:07.786Z

Nickolas_Pohilets:

I was playing only with built-in collections, not sure if this applies to all structs with a single reference-counted field.

All structs that are only a single refcounted field will see this optimisation.

uliwitness · 2021-05-10T20:09:27.364Z

This is from the ObjC days, but since closures are basically the same as ObjC blocks, the article may still be useful for understanding what closures are at their core:

Orange Juice Liberation Front – 1 Jan 14

What a Block really Is

Uli’s blog on programming, game development, pop culture and other boring things.

THV · 2021-05-10T20:25:00.872Z

Thank you! This is excellent.