Function builders

@John_McCall I finally could catch up the proposal and this thread. So far I'm overwhelmed by the complexity of this feature. Don't get me wrong, I like it a lot, but it was and still is hard to wrap your mind around it.

@gregtitus mentioned on Slack that the compiler executes the builder functions in a depth-first-traversal way, which is not mentioned anywhere in the proposal or in this thread, but this tiny information was crucial in my learning process. Knowing this makes this feature less magical and more predictable.

Here a good example:

@_functionBuilder
struct TestBuilder {
  typealias Component = [Int]
  
  static func buildBlock(_ children: Component...) -> Component {
    let result = children.flatMap { $0 }
    print("buildBlock \(children) -> \(result)")
    return result
  }
  
  static func buildIf(_ children: Component?) -> Component {
    let result = children ?? []
    print("buildIf \(children as Any) -> \(result)")
    return result
  }
}

func testBlock(@TestBuilder makeChildren: () -> [Int]) -> [Int] {
  return makeChildren()
}

testBlock {
  if true {
    [1]
    if true {
      [2]
    }
  }
  [3]
  if true {
    [4]
    if false {
      [42]
    }
  }
  [5, 6]
}

This prints:

buildBlock [[2]] -> [2]
buildIf Optional([2]) -> [2]
buildBlock [[1], [2]] -> [1, 2]
buildIf Optional([1, 2]) -> [1, 2]
buildIf nil -> []
buildBlock [[4], []] -> [4]
buildIf Optional([4]) -> [4]
buildBlock [[1, 2], [3], [4], [5, 6]] -> [1, 2, 3, 4, 5, 6]

I'm worried about one particular thing with the current design. What if you release a module with a builder that has not specified every possible builder functions? Theoretically I as the module user can retroactively extend your builder type with custom builder functions and potentially alter the behaviour of the building behavior that can lead to unexpected results.

I hope you tracked this issue and I'd expect the compiler to emit errors on retroactively added builder functions from outside your own module boundary.

4 Likes

This is, to me, the most important point. We can work around all the limitations of the current proposal but this is a major one that will not allow function builders to go beyond the (rather simple) scope of the composite pattern: for example, any kind of context-aware computation could not be modeled with the current implementation, and that would be a gigantic miss, even for a first iteration.

2 Likes

A question, and apologies if this is already answered somewhere:

How would the builder handle Never-returning functions? Would clients need to write _ = fatalError(…) inside a builder? Or would plain old fatalError(…) work?

The latter seems in keeping with the principle that throws works like normal without builder intervention, but seems like it would require a special exception in the proposal.

An overload of buildExpression that accepts Never and does nothing should do the trick.

Ah but SwiftUI is not the language Swift, it's a framework, and it's a framework that completely transforms the way people makes apps, so I don't we should mind that involves some workarounds that can be removed in due time.

Indeed, implementation is trivial; my question is just whether it is / should be a part of the proposal.

Apologies if this has been considered above already (this is a very long thread and I am still working my way through it)

Have we considered just keeping a stack of some common currency type (for a particular builder), and then passing each statement to a build function which transforms it to that common currency type and places it on the stack. If we have each build function take the stack inout, then optimizations should still be possible.

This should have several advantages:

  1. You should be able to use normal control flow (skipped expressions just won't have a chance to alter the stack)
  2. You don't run into the same 10 children are ok, but 11 are not problem (without needing variadic generics)
  3. You should be able to type check each expression individually (based on whether there is a matching build function for that expression)
  4. Build functions can make optimizations as they go by looking at the stack
  5. It still allows arbitrary nesting
1 Like

I should mention this is the approach I took in my own DSL, and it works really well for my needs. It even allows you to do very strong runtime error handling (It is a parser that can parse around the error and return meaningful human readable error messages along with a range pointing out where the issues are)

I also find the coroutine approach interesting. Perhaps instead of using {: }, we could have an @autoyielding annotation (at the declaration site instead of call site) that would cause it to automatically yield in appropriate contexts.

That said, I still prefer the stack based approach, as it has a good power to complexity ratio in my experience. Basically we would be bringing the power of something like Forth or Factor into our DSLs...

There are a lot of comments in this thread that discuss alternatives that are not capable of capturing the detailed type information this proposal is capable of capturing. It’s fine to prefer these approaches. They work very well in many contexts! But I think it’s important to address the elephant in the room. What would have SwiftUI do? How would you modify the design of SwiftUI such that it is compatible with that kind of DSL support?

3 Likes

I would argue that the Controller is now able to be broken up into lots of reusable pieces (i.e. Combine) instead of being a massive cluster that has to be rewritten for each app.

I had written my own version of this a couple of years ago (I called them Chains), and have been happily using that pattern in most of my code. Was working on open sourcing those, but with Combine doing it better, there is no need. lol.

It should still be able to capture the same type information when needed (see post above), however some of the optimizations might be pushed to runtime (until we get compile-time functions). SwiftUI would look the same at the use-site, but have a slightly different approach for builder declarations (defining functions taking individual lines and only a single build-block). The advantage is that you could have things like variable assignment and control-flow statements natively, and you wouldn't have the 10 item limit. It could also be done in a reasonably short timeframe.

Which post? Are you able to provide examples? I would like to see a sketch of how the feature would be used by a library like SwiftUI so I can better understand how it is capable of meeting the needs of a library like that.

If you mean this post I don’t see how a “common currency type” for the builder could meet the needs of SwiftUI. It’s typing requirements are much more sophisticated than anything I can imagine meeting that description. But maybe I’m missing something.

I suppose it depends on exactly what optimizations it needs, but remember that that common currency type could be a protocol, class, or enum that can easily bucket things based on various optimization opportunities... and since it is internal to the builder type, it can easily be changed in the future when new opportunities arise.

More importantly, a stack lets you lay down hint/control tokens that can affect the way the other components are used/read/etc...

Basically, you can kind of think of it kind of like a tokenizer, and then the build block becomes a parser of those tokens. Very powerful. Should have enough power for SwiftUI (but I could be wrong because I haven't seen the internals).

But doesn't currency type type-erase the underlying expression?
So the type information is lost.

I think it would be helpful for everyone involved in this discussion if folks involved in SwiftUI would be willing to comment further on how preserving type information is important to the design and implementation. @dabrahams is there any way that would be possible?

6 Likes

There is still full type information in the functions which transform it to that currency type, so many optimizations could be performed there. There are also a lot of tricks that you can do with the stack to allow recovery of type information (e.g. all of the statements after this token have the same type), which can allow cross-statement optimization.

I am making an assumption here that most optimizations can be placed in buckets of functionality (e.g. group these things in metal) and shouldn't need full type info beyond which optimization bucket is appropriate.

Yes, fully agreed!

Subscripts can capture type information and it would be arguably simpler to comprehend. It provides good distinctions between eDSL codes with native codes and is just standard Swift syntax. It doesn’t create a context around the arguments but it can be manually added through immediately called blocks.

@anandabits @John_McCall

I also feel like I need to clarify one point that I didn’t make clear before: when I say Stack, I do NOT mean array. With an actual stack it is possible to carry forward type information or to erase it as desired.

Let me give a concrete example where I use this to propagate the type info from the top item of the stack (it should be possible to do the same thing with the entire stack of type info with a related design).

I have a drawing framework that has the concept of a “DrawContext”. This is a pair of protocols that provide access to a CGContext as well as a number of values used in things like animation. The stack nodes are implemented as a bunch of types which adhere to the protocols, and which know how to wrap another node.

There are a series of functions which make the wrapping process much cleaner, and they all return the actual type created as opposed to type erasing to the protocol. In some cases, those functions (as well as initializers) are overloaded so they can take advantage of that type information (e.g. concatenation of transforms into a single layer)

If I want a type erased version, I simply store in a variable with the protocol type. The end result is that I have an efficient functional way to represent a drawing environment (including transforms, clipping, etc...).

It should be possible to capture the whole stack’s type info by making nodes where the type is dependent on what they are wrapping. It should also allow partial type erasure, etc...

The end result should be powerful enough for whatever SwiftUI needs, but that complexity can disappear for simple use cases like HTML Builder.

Does that make more sense?
(I sometimes forget that others don’t have access to the context of my ideas)

1 Like

With the current builder functions, it executes builder functions as it exits scopes. That's a depth-first post-order traversal (but just of the blocks that are executed), but only because we don't yet support any looping control flow.

That's true. We could certainly ignore builder functions that aren't defined in the builder's module, although that seems like the sort of extension that people will end up complaining about.