Function builders

[Paul_Cantrell]

@DevAndArtist

An overload of buildExpression that accepts Never and does nothing should do the trick.

Indeed, implementation is trivial; my question is just whether it is / should be a part of the proposal.

[Jon_Hull]

Apologies if this has been considered above already (this is a very long thread and I am still working my way through it)

Have we considered just keeping a stack of some common currency type (for a particular builder), and then passing each statement to a build function which transforms it to that common currency type and places it on the stack. If we have each build function take the stack inout, then optimizations should still be possible.

This should have several advantages:

1. You should be able to use normal control flow (skipped expressions just won't have a chance to alter the stack)
2. You don't run into the same 10 children are ok, but 11 are not problem (without needing variadic generics)
3. You should be able to type check each expression individually (based on whether there is a matching build function for that expression)
4. Build functions can make optimizations as they go by looking at the stack
5. It still allows arbitrary nesting

[Jon_Hull]

I should mention this is the approach I took in my own DSL, and it works really well for my needs. It even allows you to do very strong runtime error handling (It is a parser that can parse around the error and return meaningful human readable error messages along with a range pointing out where the issues are)

[Jon_Hull]

I also find the coroutine approach interesting. Perhaps instead of using {: }, we could have an @autoyielding annotation (at the declaration site instead of call site) that would cause it to automatically yield in appropriate contexts.

That said, I still prefer the stack based approach, as it has a good power to complexity ratio in my experience. Basically we would be bringing the power of something like Forth or Factor into our DSLs...

[anandabits (Matthew Johnson)]

@Jon_Hull

That said, I still prefer the stack based approach, as it has a good power to complexity ratio in my experience. Basically we would be bringing the power of something like Forth or Factor into our DSLs...

There are a lot of comments in this thread that discuss alternatives that are not capable of capturing the detailed type information this proposal is capable of capturing. It’s fine to prefer these approaches. They work very well in many contexts! But I think it’s important to address the elephant in the room. What would have SwiftUI do? How would you modify the design of SwiftUI such that it is compatible with that kind of DSL support?

[Jon_Hull]

@Karl

But most importantly, SwiftUI changes the paradigm from MVC to just MV. Had a talk with an Apple Engineer about that yesterday and he confirmed. Logic goes into the view. States go into the view. The view is inside the view. Everything is in the view.

This reinforces my concern above about these builders looking too much like invalid/bad Swift code in real large-scale applications, even though it might look fine in a tutorial app or code snippet on a slide.

I would argue that the Controller is now able to be broken up into lots of reusable pieces (i.e. Combine) instead of being a massive cluster that has to be rewritten for each app.

I had written my own version of this a couple of years ago (I called them Chains), and have been happily using that pattern in most of my code. Was working on open sourcing those, but with Combine doing it better, there is no need. lol.

@anandabits

There are a lot of comments in this thread that discuss alternatives that are not capable of capturing the detailed type information this proposal is capable of capturing. It’s fine to prefer these approaches. They work very well in many contexts! But I think it’s important to address the elephant in the room. What would have SwiftUI do? How would you modify the design of SwiftUI such that it is compatible with that kind of DSL support?

It should still be able to capture the same type information when needed (see post above), however some of the optimizations might be pushed to runtime (until we get compile-time functions). SwiftUI would look the same at the use-site, but have a slightly different approach for builder declarations (defining functions taking individual lines and only a single build-block). The advantage is that you could have things like variable assignment and control-flow statements natively, and you wouldn't have the 10 item limit. It could also be done in a reasonably short timeframe.

[anandabits (Matthew Johnson)]

@Jon_Hull

It should still be able to capture the same type information when needed (see post above)

Which post? Are you able to provide examples? I would like to see a sketch of how the feature would be used by a library like SwiftUI so I can better understand how it is capable of meeting the needs of a library like that.

[anandabits (Matthew Johnson)]

If you mean this post I don’t see how a “common currency type” for the builder could meet the needs of SwiftUI. It’s typing requirements are much more sophisticated than anything I can imagine meeting that description. But maybe I’m missing something.

[Jon_Hull]

I suppose it depends on exactly what optimizations it needs, but remember that that common currency type could be a protocol, class, or enum that can easily bucket things based on various optimization opportunities... and since it is internal to the builder type, it can easily be changed in the future when new opportunities arise.

More importantly, a stack lets you lay down hint/control tokens that can affect the way the other components are used/read/etc...

Basically, you can kind of think of it kind of like a tokenizer, and then the build block becomes a parser of those tokens. Very powerful. Should have enough power for SwiftUI (but I could be wrong because I haven't seen the internals).

[Lantua]

But doesn't currency type type-erase the underlying expression?
So the type information is lost.

[anandabits (Matthew Johnson)]

@Jon_Hull

Should have enough power for SwiftUI (but I could be wrong because I haven't seen the internals).

I think it would be helpful for everyone involved in this discussion if folks involved in SwiftUI would be willing to comment further on how preserving type information is important to the design and implementation. @dabrahams is there any way that would be possible?

[Jon_Hull]

@Lantua

But doesn't currency type type-erase the underlying expression?
So the type information is lost.

There is still full type information in the functions which transform it to that currency type, so many optimizations could be performed there. There are also a lot of tricks that you can do with the stack to allow recovery of type information (e.g. all of the statements after this token have the same type), which can allow cross-statement optimization.

I am making an assumption here that most optimizations can be placed in buckets of functionality (e.g. group these things in metal) and shouldn't need full type info beyond which optimization bucket is appropriate.

@anandabits

I think it would be helpful for everyone involved in this discussion if folks involved in SwiftUI would be willing to comment further on how preserving type information is important to the design and implementation. @dabrahams is there any way that would be possible?

Yes, fully agreed!

[yesleon (Li-Heng Hsu)]

Subscripts can capture type information and it would be arguably simpler to comprehend. It provides good distinctions between eDSL codes with native codes and is just standard Swift syntax. It doesn’t create a context around the arguments but it can be manually added through immediately called blocks.

[Jon_Hull]

@anandabits @John_McCall

I also feel like I need to clarify one point that I didn’t make clear before: when I say Stack, I do NOT mean array. With an actual stack it is possible to carry forward type information or to erase it as desired.

Let me give a concrete example where I use this to propagate the type info from the top item of the stack (it should be possible to do the same thing with the entire stack of type info with a related design).

I have a drawing framework that has the concept of a “DrawContext”. This is a pair of protocols that provide access to a CGContext as well as a number of values used in things like animation. The stack nodes are implemented as a bunch of types which adhere to the protocols, and which know how to wrap another node.

There are a series of functions which make the wrapping process much cleaner, and they all return the actual type created as opposed to type erasing to the protocol. In some cases, those functions (as well as initializers) are overloaded so they can take advantage of that type information (e.g. concatenation of transforms into a single layer)

If I want a type erased version, I simply store in a variable with the protocol type. The end result is that I have an efficient functional way to represent a drawing environment (including transforms, clipping, etc...).

It should be possible to capture the whole stack’s type info by making nodes where the type is dependent on what they are wrapping. It should also allow partial type erasure, etc...

The end result should be powerful enough for whatever SwiftUI needs, but that complexity can disappear for simple use cases like HTML Builder.

Does that make more sense?
(I sometimes forget that others don’t have access to the context of my ideas)

[John_McCall]

@DevAndArtist

@gregtitus mentioned on Slack that the compiler executes the builder functions in a depth-first-traversal way, which is not mentioned anywhere in the proposal or in this thread, but this tiny information was crucial in my learning process. Knowing this makes this feature less magical and more predictable.

With the current builder functions, it executes builder functions as it exits scopes. That's a depth-first post-order traversal (but just of the blocks that are executed), but only because we don't yet support any looping control flow.

@DevAndArtist

I'm worried about one particular thing with the current design. What if you release a module with a builder that has not specified every possible builder functions? Theoretically I as the module user can retroactively extend your builder type with custom builder functions and potentially alter the behaviour of the building behavior that can lead to unexpected results.

That's true. We could certainly ignore builder functions that aren't defined in the builder's module, although that seems like the sort of extension that people will end up complaining about.

[John_McCall]

@ExFalsoQuodlibet

This is, to me, the most important point. We can work around all the limitations of the current proposal but this is a major one that will not allow function builders to go beyond the (rather simple) scope of the composite pattern: for example, any kind of context-aware computation could not be modeled with the current implementation, and that would be a gigantic miss, even for a first iteration.

I think I'm leaning towards the same conclusion: there's really no reason not to use instance methods for building, since clients like SwiftUI that don't need state tracking can of course just use an empty type. It would require every builder type to provide a static startFunction() method (or maybe an init?) that creates the builder instance, but that seems like a fairly modest imposition.

Builders like SwiftUI that want to propagate typed values for each block can define finishBlock methods that then cause the results from the block to be collected into locals (thus requiring buildConditional and so on) so they can be passed finishBlock later, while builders that just want to internally collect values as they come can just define buildExpression to collect the value.

This would capture most of the expressible behavior of receiver closures except for the ability to initialize the builder context dynamically by passing it in as a parameter. I don't know how important that is in practice to its use cases in Kotlin, and the implicit contextual data dependency seems pretty unfortunate in some ways.

[John_McCall]

@Paul_Cantrell

How would the builder handle Never -returning functions? Would clients need to write _ = fatalError(…) inside a builder? Or would plain old fatalError(…) work?

If we can teach the builder to ignore Void-typed expressions, we can teach it to ignore uninhabited types like Never as well, and that seems reasonable.

I don't think we can reasonably do things like have the builder ignore results that appear in unreachable code.

[John_McCall]

SwiftUI wants to propagate information about component types into the type of the built result, not just propagate information about component types temporarily into the builder instead of erasing at an earlier stage. At a very low level, this allows the in-memory representation of a stack of, say, an A, a B, and a C to be essentially an (A, B, C) tuple instead of having to store type information and potentially boxing at each level, which is important for the memory performance of the system given its heavy use of type-recursion and decorators; but more importantly it provides a guarantee that the type structure won't change (except at specific places that request it with AnyView), which makes structural analysis and value-comparison far more efficient.

At least, that's how I understand it; I am a language developer, not a SwiftUI expert.

[Jon_Hull]

Here is some very simple code to show it is possible to store all type information:

protocol Stack {
    func topValue<V>(of type:V.Type) -> V?
    var popErased:Stack? {get}
}

extension Stack {
    func push<V>(_ value:V) -> Node<Self,V> {
        return Node(parent: self, value: value)
    }
}

struct Root<T>:Stack {
    let topValue:T
    
    init(value:T) {
        self.topValue = value
    }
    
    func topValue<V>(of type: V.Type) -> V? {
        return topValue as? V
    }
    
    var popErased:Stack? {
        return nil
    }
}

struct Node<P:Stack,T>:Stack {
    let pop:P
    let topValue:T
    
    init(parent:P, value:T) {
        self.pop = parent
        self.topValue = value
    }
    
    func topValue<V>(of type: V.Type) -> V? {
        return topValue as? V
    }
    
    var popErased:Stack? {
        return pop
    }
}

let nodeA = Root(value: "One")   // Type: Root<String>
let nodeB = nodeA.push(2)        // Type: Node<Int, Root<String>>
let rootAgain = nodeB.pop        // Type: Root<String>
let val = nodeB.topValue         // Type: Int
let rootVal = rootAgain.topValue // Type: String
let erased:Stack = nodeB         // Type: Stack

[John_McCall]

Sure, I don't disagree that it's possible to simulate some effects of variadic generics through your own generic pair types. The Either approach used in this proposal in fact simulates a variadic sum type by folding to a tree of pairwise sums. I think this would be much more awkward for some DSL clients, but I don't know if there's any reason we couldn't use this for SwiftUI.