[Pitch #2] Function builders

Hi all,

It has been a long time since the first function builders pitch last year. Since then, we've completely transformed the implementation to get the support for language constructs like local let bindings, if let, switch, and if #available that we wanted, as well as implementing a new semantic model that makes efficient type checking possible.

Last year's pitch thread involved a lot of potential expansions to the idea of function builders---things like virtualizing execution, or programmatically inspecting the AST---to broaden their applicability into other domains. None of those have made any visible progress. On the other hand, people have built a number of cool DSLs on top of the experimental function builders implementation, so we feel pretty good that we have a well-rounded feature that makes Swift more expressive. The ad hoc nature of the interaction between the language and function builder types allows us to treat the system we have as the basis, which can be further extended when those ideas come to fruition.

To that end, here is a revised proposal for function builders. It captures the actual state of the implementation on master (any trunk development snapshot will do), most of which is also in Swift 5.3.

Whaddya think?

Doug

31 Likes

Just to be sure. Generic function builders which propagate their generic parameters to their build functions would not work under this proposal? I didn't see any mention of that in the proposal, but this used to work in the first version of function builders and lately it doesn't. This capability is allows a super convenient class of DSLs which are not possible anymore. If it's indeed the case that they will not work, would be nice to have its justification be addressed in the proposal.

-- edit

I saw mention of back propagation, but if I understand correctly, having a generic function builder that feeds its generic parameters to their build functions would not violate the "no back propagation" rule. Am I misunderstanding this?

One can propagate from generic arguments of the function builder type to their build functions; that should work. We had a bit of a discussion a few months ago about this. Is that what you had in mind? Do you have a specific example we can work through?

Doug

(To balance the nit-picky nature of this comment, I’ve expressed my pickiest nits in the form of a PR)

If a function builder defines buildEither(first:) and buildEither(second:), but not buildOptional(_:), is it able to process conditionals where all branches produce a value? The exposition under Selection statements seems to say yes, while the summary of buildOptional suggests no.

In Statement blocks:

Within a statement block, the individual statements are separately transformed into sequences of statements which are then concatenated. Each such sequence may optionally produce a single partial result , which is an expression (typically a reference to a local variable) which can be used later in the block.

The relationship between the “typically a reference to a local variable” here and “The statement-expression is used to initialize a unique variable of the statement-expression's result type” in Expression statements is perhaps a bit too subtle

If the statement block is the top-level body of the function being transformed, the final statement in the transformed block is a return with the combined result expression as its operand.

This should say something about buildFinalResult, right?

Example:

We're not using an injection tree because the function builder type doesn't declare those methods

It does, though.

Really like the additions of switch and if let!

One unexpected behaviour i encountered (with the Xcode 12 Beta 4) is that the functionbuilder transform doesn't seem to consider protocols and/or protocol extensions.

My example is a "default" function-builder.
The idea was to provide a protocol with default implementations for most of the build methods.

indirect enum FunctionBuilder<Expression> {
    case expression(Expression)
    case block([FunctionBuilder])
    case either(Either<FunctionBuilder, FunctionBuilder>)
    case optional(FunctionBuilder?)
}

protocol FunctionBuilderProtocol {
    associatedtype Expression
    typealias Component = FunctionBuilder<Expression>
    associatedtype Return
    
    static func buildExpression(_ expression: Expression) -> Component
    static func buildBlock(_ components: Component...) -> Component
    static func buildDo(_ components: Component...) -> Component
    static func buildOptional(_ optional: Component?) -> Component
    static func buildArray(_ components: [Component]) -> Component
    static func buildLimitedAvailability(_ component: Component) -> Component
    
    static func buildFinalResult(_ components: Component) -> Return
}

and provide default implementations as protocol extensions

extension FunctionBuilderProtocol {
    static func buildExpression(_ expression: Expression) -> Component { .expression(expression) }
    static func buildBlock(_ components: Component...) -> Component { .block(components) }
    static func buildDo(_ components: Component...) -> Component { .block(components) }
    static func buildOptional(_ optional: Component?) -> Component { .optional(optional) }
    static func buildArray(_ components: [Component]) -> Component { .block(components) }
    static func buildLimitedAvailability(_ component: Component) -> Component { component }
}

With this in place the only thing that needs to be implemented is buildFinalResult.

@_functionBuilder
enum ArrayBuilder<E>: FunctionBuilderProtocol {
    typealias Expression = E
    typealias Return = [E]
    
    static func buildFinalResult(_ components: Component) -> Return {
        switch components {
        case .expression(let e): return [e]
        case .block(let children): return children.flatMap(buildFinalResult)
        case .either(.left(let child)): return buildFinalResult(child)
        case .either(.right(let child)): return buildFinalResult(child)
        case .optional(let child?): return buildFinalResult(child)
        case .optional(nil): return []
        }
    }
}

But this doesn't compile

func buildArray(@ArrayBuilder<String> build: () -> [String]) -> [String] {
    return build()
}


let a = buildArray {
    "1"
    "2"
    if Bool.random() {
        "maybe 3"
    }
}


// error: closure containing control flow statement cannot be used with function builder 'ArrayBuilder'
//    if Bool.random() {
//    ^
//
// note: generic enum 'ArrayBuilder' declared here
// enum ArrayBuilder<E>: FunctionBuilderProtocol {
//      ^
//
// error: cannot convert value of type 'String' to expected argument type 'FunctionBuilder<ArrayBuilder<String>.Expression>' (aka 'FunctionBuilder<String>')
//    "1"
//     ^
// error: cannot convert value of type 'String' to expected argument type 'FunctionBuilder<ArrayBuilder<String>.Expression>' (aka 'FunctionBuilder<String>')
//    "2"

1 Like

Is it possible to have non-result-producing case in switch, or is it always result-producing due to exhaustivity? I feel that the latter could be a little problematic if you want to ignore some branches.

I'm very excited to see this proposal evolve. I currently have a couple of high-level thoughts based on my experimentation with the current @_functionBuilder support:

:one: As I've been working with this, I've never felt like I've been "building functions" as @functionBuilder implies. Nor do I even feel like I've been defining a "builder of functions", because "function" in swift has the implied meaning of an entire method (name + signature + implementation), whereas the building going on is only about defining the implementation.

In my mind, every time I've used @functionBuilder, I've been attempting to "define a DSL", and to that end I believe naming the attribute @dsl or @abstractSyntaxTree or even @statementBuilder (or similar) would be a better description of what it's doing and how it's expected to be used.

:two: My other piece of feedback is related to @anreitersimon's suggestion. When implementing one of these builders, I struggled to know what I actually needed to implement in order to get things to work. I like the idea of making @dsl/@functionBuilder imply conformance to a standard-library-provided protocol, so that implementors have something they can go look at to see what they need to implement. All too often I had to go dig around in the Swift sources, and I don't wish that experience on anyone. :sweat_smile:

15 Likes

Please include some links — or, if that's not possible, a description of some of those examples.

Overall, I'm not convinced that that the feature is worth the added complexity (and the perceived violation of the evolution process). I know that won't matter, and function builders are here to stay, but still it would be nice to have some proof that this a more than a one-trick pony for SwiftUI.

You can find some examples here. A couple more examples are:

6 Likes
struct Component<Root> {}

@_functionBuilder
struct Builder<Root> {
    static func buildBlock(_ component: Component<Root>) -> [Component<Root>] {
        [component]
    }
    
    static func buildBlock(_ components: [Component<Root>]) -> [Component<Root>] {
        components
    }
}

struct Schema<Root> {
    init(_ content: [Component<Root>]) {}
    init(@Builder<Root> content: () -> [Component<Root>]) {}
}

// This works

Schema<Bool>([
    Component()
])

Schema<String>([
    Component(),
    Component()
])

// This doesn't

Schema<Bool> {
    Component() // Generic parameter 'Root' could not be inferred.
}

Schema<String> {
    Component() // Generic parameter 'Root' could not be inferred.
    Component() // Generic parameter 'Root' could not be inferred.
}

IMO, the function builder initializer example should work exactly like the array initializer. Having to specify the generic parameters of Component makes function builders not the best option for this class of DSLs.

A real-life example of this kind of DSL is Graphiti which allows you to write GraphQL schemas in Swift.

This is what the API used to look like:

self.schema = try Schema<MessageRoot, MessageContext> {
    Type(Message.self) {
        Field("content", at: \.content)
    }

    Query {
        Field("message", at: MessageRoot.message)
    }
}

Now it has to be:

self.schema = try Schema<MessageRoot, MessageContext> {
    Type<MessageRoot, MessageContext, Message>(Message.self) {
        Field<Message, MessageContext, String, NoArguments>("content", at: \.content)
    }

    Query<MessageRoot, MessageContext> {
        Field<MessageRoot, MessageContext, String, NoArguments>("message", at: MessageRoot.message)
    }
}

As you can see that's really not a good API for developers. In the end we had to go with variadic arguments:

self.schema = try Schema<MessageRoot, MessageContext>(
    Type(Message.self,
        Field("content", at: \.content)
    ),

    Query(
        Field("message", at: MessageRoot.message)
    )
)

This is exactly the kind of API that function builders should solve and unfortunately they did not help us here.

3 Likes

Excellent write-up! I can’t wait until they become an official part of the language; I’m already using builders extensively in some projects of mine.

Is there a reason why the builder methods are static? I don’t think it’s unreasonable to instantiate a builder at the point of the annotation (just like property wrappers) and let it have state or configuration. It would also make them more consistent with the mental model of property wrappers: the @ instantiates a value that does special things, be it function building or property wrapping. The transformations still work with the returned values of the methods, and the builder is discarded once the function transformation has completed.

@ViewBuilder probably doesn’t have a reason to have state/configuration so it would be a struct without properties, but I don’t think there’s a significant overhead.

4 Likes

For a first step function builders are really good. Nonetheless, I’ve had some time to play around with them and while SwiftUI doesn’t really need to constraint the count of components (Views) it accepts in @ViewBuilder closure, I think it’s a really important feature. Having some way to inform the function builder about the count of components currently accepted to improve error messages would be much appreciated.

One possible design:

extension ArrayBuilder {
    static var acceptedBuildComponents: UInt { 
        Count.acceptedCount 
    } 
}

This way the compiler would know that only 2 components are accepted in the current buildBlock overloads. Therefore if there are more components:

@ArrayBuilder<String, Two>
var strings: [String] {
    “first”

    “second”

    “third” ❌ ArrayBuilder<String, Two> accepts only two components!
}

Adding a protocol would mean that function builders would only work with standard libraries that were new enough to include that protocol (because we can’t backwards-deploy types the way we can backwards-deploy functions). It would also mean that you could only provide one implementation for each of the requirements, rather than being able to provide multiple overloads and having the compiler choose the most appropriate one, and that buildBlock(...) would need to either have a specific, fixed set of arities or be (non-generic) variadic.

(The second concern is why StringInterpolationProtocol doesn’t formally model appendInterpolation(...) as a requirement. The first is why we made sure to redesign string interpolation in time for ABI stability, which ensured that these protocols would always be available; if we’d redesigned post-ABI stability, we’d probably have been stuck using attributes or something.)

Fortunately, at a minimum people will be able to look at the approved function builder proposal to see what they should implement. I assume there will be other documentation, too, and there’s no particular reason that SourceKit couldn’t provide code completion assistance on @functionBuilder types if we wanted it to.

3 Likes

Perhaps this can be solved using the existing compile time evaluation machinery and/or a new language feature that allows you to declare fixed-size arrays.

1 Like

I think this is something that could probably be handled with the right diagnostic work in the type-checker rather than extra language support. In the code that would otherwise diagnose that a call has too many arguments for all the possible overloads, notice that the call is a synthesized call to buildBlock and use a different diagnostic.

1 Like

The answer should be "no". I've clarified the document (and fixed the comment about why we aren't using an injection tree), thank you!

Yes, added.

Sure, clarified.

Thanks!

Doug

1 Like

Switches are always exhaustive and must have a statement in each case, so function builders inherits that and therefore switches are always result-producing. Your DSL could add a value to mean "ignore" or "nothing", such as () or something like SwiftUI's EmptyView.

Doug

1 Like

I referenced SwiftSyntax DSL and an alternative SwiftPM manifest format in the proposal already, and now I've added the Shortcuts and CSS DSLs you've linked to.

Doug

1 Like

Hi Doug,
awesome work on all the recent improvements in function builders, great to see the proposal :slight_smile:

I'd love to bring back the discussion we had about declarations in function builders (there's a writeup with a mock example, though I have a complete DSL which is hurting because of this I can share privately): Function builders and "including" let declarations in built result

Long story short, when trying to port/represent declarative APIs defined in JSON/YAML/XML that often have to "refer to" previously declared elements, it is hard/annoying to express these in today's function builders. Examples of such DSLs would include:

  • "schemas" and their use where a table is first declared, and then other elements need to say "I'm using that one
  • html where one wants to express "when X is clicked show that other element that I declared previously"

More specifically, today I have to do:

// TODAY:
let thingSchema = Schema(id: "thing") // to have the variable
thingSchema // to make it "visible" to function builders
// forgetting to repeat this `thingSchema` statement leads to it missing 
// in the generated schemas, whoops! And it's super easy to miss.

Table(schema: thingSchema)

one proposed workaround was to use nesting, like so:

// TODAY / workaround:
Schema(id: "thing") { schema in 
  Table(schema: schema)
}

and that's okey for simple or "a few" examples but in the DSLs I deal with there can be tens of schemas, and tables may need to use a few of them, leading to unwieldly crazy nesting levels. And more importantly, the nesting does not represent how one thinks about those domains when declaring them.

What one really would want to write is the following:

// WHAT-IF:
let thingSchema = Schema(id: "thing")
let otherSchema = Schema(id: "other")

Table(schemas: thingSchema, otherSchema) { ... }
Table(schema: thingSchema) { ... }
Table(schema: otherSchema) { ... }

So... I fully understand that one may not want to encourage this style, because most declarations should be pure and just for organization purposes, and not accidentally "emitting" things into the function builder. I do argue that for some DSLs this would be a life-changer.

Also, function builders already are not "normal swift" because if and switch statements do emit values already (and to be honest I'd love them to do so anyway in real swift, and not just function builders but that's a another discussion entirely :wink:).

More details are in the linked thread, but that's the TL;DR; and the proposed addition would be some form of:

public static func buildDeclaration(_ element: ThingElementConvertible) -> ThingElement? {
    if let def = element as? Def {
        return .importDef(def) // great, include it
    } else {
        return nil
    }
}

which allows the function builder to inspect and by type include or not declarations. E.g. tables I want to import, but I would not import any random let int = 2 since users may want to do those ad-hoc after all.

Looking forward to see what you think about that!

2 Likes

One thing that I would love to do is to change the final result produced by an existing function builder expression: for example, it could be possible to generate a suite of UI tests that trigger every button, every NavigationLink and every other control in all possible (non-trivial) combinations for a SwiftUI app, hopefully with minimal effort.

Not sure how to achieve this though. Syntax brainstorming:

@UITesting(maxCycleDepth: 3) let view = SomeView().body

In this case the UITesting property wrapper would be able to “inject” additional logic in every buildBlock, buildExpression function inside the original @ViewBuilder body of the SomeView struct.

This is a very rough idea, and I’m probably horribly wrong. Please correct me! (And bikeshed everything) :pray::pray:

1 Like