Add @ArrayBuilder to the standard library

Hi everyone! We have leveraged @ArrayBuilder-like result builders at Airbnb for many years and have found it to really improve the ergonomics of constructing array values. This is such a common operation that it would be great to include a first-party solution for it in the standard library. Here’s a pitch, what do you think?

Will call attention to the slightly orthogonal mini-pitch within this pitch: it would be great to support @ArrayBuilderattributes on properties and functions, rather than require the fully spelled out @ArrayBuilder<Element> syntax that is required today. Compiler implementation here. Even if we decide @ArrayBuilder doesn’t belong in the standard library, this change seems like a great addition.

Introduction

We should add a @ArrayBuilder result builder to the standard library, and a static method for using an @ArrayBuilder to create an array value. We should also improve the ergonomics of working with generic @resultBuilder types.

Motivation

Since being introduced with the release of SwiftUI in 2019, and being formalized in SE-0289, result builders have set the standard for defining lists and trees in code. Compared to imperative code to define these structures, result builders are less verbose and easier to evolve.

Take this example of constructing a [String] array representing arguments for a swift-format invocation:

var arguments: [String] {
    var arguments = [String]()
    arguments.append("format")

    arguments.append("--configuration")
    arguments.append(configurationPath)

    arguments.append("--in-place")

    if recursive {
        arguments.append("--recursive")
    }

    if let cachePath {
        arguments.append("--cache-path")
        arguments.append(cachePath)
    }

    return arguments
}

Compared to the declarative simplicity we are used to from result builders, this code is imperative and verbose.

If instead written using an @ArrayBuilder, the code is much simpler and focuses exclusively on the business logic rather than the imperative array manipulation.

@ArrayBuilder
var arguments: [String] {
    "format"

    "--configuration"
    configurationPath

    "--in-place"

    if recursive {
        "--recursive"
    }

    if let cachePath {
        "--cache-path"
        cachePath
    }
}

Proposed solution

We should add an ArrayBuilder result builder to the standard library, and a static method for using an @ArrayBuilder to create an array value.

While it is straightforward to define an ArrayBuilder type in your own project, this is a strong candidate to include in the standard library:

  • Initializing array values is such a common operation that this addition will be relevant to the vast majority of Swift projects.
  • Including @ArrayBuilder in the standard library enables it to become a standard best practice across the Swift ecosystem.
  • Standardizing on a single implementation reduces fragmentation.

Detailed design

ArrayBuilder

The ArrayBuilder type should accept all of the relevant @resultBuilder functionality, including:

  • Element values, using append
  • Sequence<Element> values, using append(contentsOf:)
  • Conditional values (if statements, optional Element and Sequence values)
  • For loops

A future draft will include the full public API of the proposed ArrayBuilder type.

This enables the following @ArrayBuilder method:

@ArrayBuilder
var arguments: [String] {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
}

Array.build(_:)

We should add a static func build(@ArrayBuilder _ makeArray: () -> [Element]) -> [Element] method to the Array type.

let arguments = Array.build {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
}

// Above example uses type inference to infer `Element` type.
// Other valid syntax includes:
let arguments = [String].build {
    ...
}

let arguments: [String] = .build {
    ...
}

Improving ergonomics of generic @resultBuilders

Today, when using a generic @resultBuilder, you must explicitly specify the generic arguments. Otherwise, you get an error like "reference to generic type 'ArrayBuilder' requires arguments in <...>".

@ArrayBuilder<String>
var arguments2: [String] {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
}

This is redundant since the ArrayBuilder.Result.Element type can be inferred to be the Element type of the attached declaration's return type. Instead, we should allow this to be inferred so that a simpler syntax can be used:

@ArrayBuilder
var arguments2: [String] {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
}

This is useful regardless of whether or not we add ArrayBuilder to the standard library, since the enhancement would be available for user-defined ArrayBuilder types.

Source compatibility

There is always the possibility of source breaks when adding new types or declarations to the standard library, however this is unlikely given user-defined types take precedence over standard library types during symbol lookup.

ABI compatibility

Since this proposal is purely additive, it has no ABI impact.

Implications on adoption

Since result builders have no runtime component, it will be possible to backport @ArrayBuilder support to existing operating system versions.

Future directions

Add .build to other types

We could also add .build static function to other sequence / collection types that currently have an init that accepts an Array / Sequence. Set is probably the best example. However, a @SetBuilder that can create a set in a single pass (rather than building an array and then converting it to a set) may be preferrable.

Alternatives considered

CollectionBuilder

Instead of only supporting arrays, we could create a way to construct any Swift collection. However, the Collection type provides no way to construct and manipulate collection values. We need at a minimum RangeReplaceableCollection, which provides init and append requirements.

Array, ArraySlice, ContiguousArray, Slice, String, and Substring are the only standard library types that conform to RangeReplaceableCollection. It seems potentially undesirable to support String, due to the Unicode-related implications.

Given the Array use case would be multiple orders of magnitude more common than ArraySlice, ContiguousArray, and Slice, it seems reasonable to anchor on the Array use case. ArrayBuilder is a much better name for the Array use case than CollectionBuilder.

Array.init(build:)

This proposal includes an Array.build(_:) method. Instead, we could name that method Array.init(build:).

Including build in the name seems like a good idea to indicate that this is using a @ArrayBuilder / @resultBuilder. This method will commonly be called with a trailing closure, so the only way to ensure the name is visible is to use a static function:

let arguments = Array(build: {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
})

// Trailing closure: no `build`.
let arguments = Array {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
}

Trailing closure syntax is also not currently supported when using [Element] syntax:

// error: 'let' declarations cannot be computed properties
let arguments = [String] {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
}

While this can be worked around by using let arguments: [String] = .init { ... } syntax instead, ideally we select a syntax that supports type inference from the RHS value.

Array.build results in idiomatic call sites that include the build name as a signal to the @ArrayBuilder semantics.

let arguments = Array.build {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
}

let arguments = [String].build {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
}

let arguments: [String] = .build {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
}
29 Likes

These are good reasons for why ArrayBuilder could ship in Standard Library… but I'm still not sure I see the reason for why ArrayBuilder must ship in Standard Library. Because this requires no private or internal symbols I see no technical reasons why this could not ship in an adjacent package. Yes we are potentially fragmenting work across the ecosystem… but the same critique could be made of the work being done in swift-algorithms or swift-collections.

It's a good idea… but IMO the benefits of incubating this idea out in the community outweigh the benefits of codifying this in the Standard Library right at this specific moment in time.

Could Airbnb open source that package? Could we then bring that to the community and see what adoption looks like?

3 Likes

The implementation of an @ArrayBuilder is indeed trivial. There are a few on GitHub.

Having a canonical official implementation would be great to reduce fragmentation and endorse this as a best practice.

swift-collections certainly seems like a reasonable place for this to live, and I would find that satisfactory.

However, “constructing arrays” is an almost universal need. To me this is a step above the more “niche” data structures in swift-collections. If we think of this as becoming a standard best practice for constructing complex arrays, then the standard library seems like the most fitting home.

12 Likes

I’m using the Array.init(_ build:) alternative for my projects, so I favor that syntax, otherwise big +1.

I also think this fits in the standard library. The Dart language, for example, features a similar capability on their website:

2 Likes

+1. This is a neat idea, and something I’ve wished for a long time, even if it’s not that hard (but still not quite trivial either) to write down when needed.

While the above may be true for @resultBuilder types of generic type, nothing forces to define ArrayBuilder with a type parameter. You can sidestep the need by making the static functions generic instead. Here’s a minimal (but unoptimised[1]) draft implementation:

@resultBuilder
public struct ArrayBuilder {
    public static func buildOptional<T>(_ optional: [T]?) -> [T] { optional ?? [] }
    public static func buildEither<T>(first: [T]) -> [T] { first }
    public static func buildEither<T>(second: [T]) -> [T] { second }
    public static func buildPartialBlock<T>(first: T) -> [T] { [first] }
    public static func buildPartialBlock<T>(accumulated: [T], next: T) -> [T] { accumulated + [next] }
    public static func buildPartialBlock<T>(accumulated: [T], next: [T]) -> [T] { accumulated + next }
}

…which is compatible out of the box with the examples of the pitch, such as this:

@ArrayBuilder
var arguments: [String] {
    "format"
    "--in-place"

    if recursive {
        "--recursive"
    }
}

  1. I’d probably throw in some consuming builder function parameters at least, maybe also see about avoiding excessive heap allocations with the aid of an intermediate component type and an ultimate buildFinalResult step to construct the Array. ↩︎

9 Likes

I agree that having it in an official package is a good idea and I also agree with the swift-collections package being a more appropriate package to be host to this functionality.

This is a nice tip, thanks :)

More complex examples seem to be better supported by a ArrayBuilder<Element> implementation.

I had trouble adapting this starter code to support all of these cases:

e.g. Element, Element?, [Element], [Element?], [Element?]?

But this is all trivial to support in the ArrayBuilder<Element>version:

Sample code
@resultBuilder
public struct ArrayBuilder<Element> {

    public static func buildExpression(_ element: Element) -> [Element] {
        [element]
    }

    public static func buildExpression(_ element: Element?) -> [Element] {
        element.map { [$0] } ?? []
    }

    public static func buildExpression(_ sequence: some Sequence<Element>) -> [Element] {
        Array(sequence)
    }

    public static func buildExpression(_ sequence: (some Sequence<Element>)?) -> [Element] {
        sequence.map(Array.init) ?? []
    }

    public static func buildExpression(_ sequence: some Sequence<Element?>) -> [Element] {
        sequence.compactMap { $0 }
    }

    public static func buildExpression(_ sequence: (some Sequence<Element?>)?) -> [Element] {
        sequence?.compactMap { $0 } ?? []
    }

    public static func buildBlock(_ components: [Element]...) -> [Element] {
        components.flatMap { $0 }
    }

    public static func buildOptional(_ component: [Element]?) -> [Element] {
        component ?? []
    }

    public static func buildEither(first: [Element]) -> [Element] {
        first
    }

    public static func buildEither(second: [Element]) -> [Element] {
        second
    }

    public static func buildArray(_ components: [[Element]]) -> [Element] {
        components.flatMap { $0 }
    }
}

let optionalString: String? = "opt"
let nilString: String? = nil

let array = ["a", "b"]
let optionalArray: [String]? = ["c", "d"]
let nilArray: [String]? = nil

let optionalElements: [String?] = ["e", nil, "f"]
let optionalElementsOptional: [String?]? = ["g", nil, "h"]

let recursive = true

@ArrayBuilder<String>
var arguments: [String] {

    // Element
    "format"

    // Element?
    optionalString
    nilString

    // [Element]
    array

    // [Element]?
    optionalArray
    nilArray

    // [Element?]
    optionalElements

    // [Element?]?
    optionalElementsOptional

    // if
    if recursive {
        "--recursive"
    }

    // if let (Element)
    if let value = optionalString {
        value
    }

    // if let (Sequence)
    if let values = optionalArray {
        values
    }

    // for loop producing Element
    for i in 1...3 {
        "--level-\(i)"
    }

    // for loop producing optional Element
    for value in [optionalString, nilString] {
        value
    }

    // for loop producing sequence
    for chunk in [["x", "y"], ["z"]] {
        chunk
    }
}

print(arguments)

Debatable whether or not we’d support all of these out of the box in the standard library, but an implementation that is extensible to support these types of cases is more robust.

I was also initially surprised to find the stdlib didn't ship with Array.Builder or Set.Builder. But having worked with them, I do lean more towards placing them in a separate package (and swift-collections sounds like the right call, but consider perhaps whether Foundation would be appropriate, too).
FWIW, I initially opted for a separate package (GitHub - coenttb/swift-builders: A Swift package with result builders for Array, Dictionary, Set, String, and Markdown.), but for swift-standards, builders are part of the StandardLibraryExtensions target in swift-standards.

Also, I have opted to do Array.Builder, so the generics from parent Array are available for Builder. Perhaps this is something to consider for the proposed stdlib/swift-collection builders, too?

This is pretty neat and I would use it. I would probably use other similar functionality like an eventual DataBuilder, provided that we can make it fast. I think that one of the benefits of a robust implementation of ArrayBuilder would be that it can do a good job of estimating the number of elements the array will include and avoid allocation traffic.

One of the benefits of shipping things in the standard library is that the surface it defines can be used in SDK libraries. One of the problems is that except in pretty specific circumstances, it ties these APIs to an OS version on Apple platforms, which is a common cause of extreme developer sadness. I don’t expect that we would see @ArrayBuilder in SDK API surface because you might as well just take an Array; I think this points to preferring leaving this out of the standard library.

6 Likes

Is this relevant for @ArrayBuilder? I’d expect it to be very easy to back-deploy given result builders have no runtime component.

That would need an implementation-specific assessment to tell. Result builders don’t guarantee having or not having a runtime component.

2 Likes

The normal course of events, IIUC, is that new functions and methods can be freely back-deployed if they depend only on other earlier-available entrypoints, but types, protocols, and conformances cannot be back-deployed. Substantial efforts and design decisions were put in to make such back-deployment possible for the Concurrency library.

4 Likes

Good to know, thanks! Didn’t know that.

The individual static funcs in an enum ArrayBuilder should be back-deployable today.

Since the enum in this case is “just a namespace” and serves no purpose at runtime, what are the technical obstacles to back deploying it, other than that this isn’t implemented yet? Naively it seems trivially back deployable if all the associated static functions are back-deployed and / or emitted into the client.

I’m not sure exactly what workarounds exist for such a case, someone more familiar with the details of back-deployment would have to weigh in! Technically even an uninhabited enum introduces an (inhabited) metatype, but I’m not sure if that poses the same issue for back-deployment. In any case, such matters are typically out of scope for evolution discussion!

1 Like

There might be some clues for you from the support to back-deploy Span et al:

2 Likes

I put together a compiler implementation for inferring result builder generics. It works well and feels really natural :smiley: Add support for inferring generic parameters of result builder attributes by calda · Pull Request #86209 · swiftlang/swift · GitHub

@resultBuilder
struct ArrayBuilder<Element> {
  static func buildBlock(_ elements: Element...) -> [Element] {
    elements
  }
}

@ArrayBuilder // ✅
var stringArray: [String] {
  "foo"
  "bar"
}

For the implementation of ArrayBuilder itself, would appreciate input on what exactly we should support. I tend to think we should support all of:

which gives us an implementation something like:

@resultBuilder
public enum ArrayBuilder<Element> {
  public static func buildExpression(_ element: Element) -> [Element] {
    [element]
  }

  public static func buildExpression(_ element: Element...) -> [Element] {
    element
  }

  public static func buildExpression(_ element: Element?) -> [Element] {
    guard let element else { return [] }
    return [element]
  }

  public static func buildExpression(_ array: [Element]) -> [Element] {
    array
  }

  public static func buildExpression(_ array: [Element]?) -> [Element] {
    array ?? []
  }

  public static func buildExpression(_ sequence: some Sequence<Element>) -> [Element] {
    Array(sequence)
  }

  public static func buildExpression(_ sequence: (some Sequence<Element>)?) -> [Element] {
    guard let sequence else { return [] }
    return Array(sequence)
  }

  public static func buildExpression(_ sequence: some Sequence<Element?>) -> [Element] {
    sequence.compactMap { $0 }
  }

  public static func buildExpression(_ sequence: (some Sequence<Element?>)?) -> [Element] {
    guard let sequence else { return [] }
    return sequence.compactMap { $0 }
  }

  public static func buildOptional(_ component: [Element]?) -> [Element] {
    component ?? []
  }

  public static func buildEither(first: [Element]) -> [Element] {
    first
  }

  public static func buildEither(second: [Element]) -> [Element] {
    second
  }

  public static func buildArray(_ components: [[Element]]) -> [Element] {
    components.flatMap { $0 }
  }

  public static func buildLimitedAvailability(_ component: [Element]) -> [Element] {
    component
  }

  public static func buildPartialBlock(first: [Element]) -> [Element] {
    first
  }

  public static func buildPartialBlock(
    accumulated: [Element],
    next: [Element],
  ) -> [Element] {
    var result = accumulated
    result.append(contentsOf: next)
    return result
  }
}
6 Likes

Though I agree this is a nice extension to the standard library, I would argue that it fits better to be in a standalone package, for reasons of backward deployment feasibility and whether the functionality is essential.

1 Like

I've attempted to show the result builder transform for your second example. It seems much less efficient than your first example. In particular, each buildExpression call is creating a single-element array.

  • Could you remove all of the buildExpression overloads, and instead have buildPartialBlock overloads with consuming parameters?

  • (Or perhaps the result builder transform should support an inout parameter for the accumulated result?)

var arguments: [String] {
    let $__builder0: [String] = ArrayBuilder.buildExpression("format")

    let $__builder1: [String] = ArrayBuilder.buildExpression("--configuration")
    let $__builder2: [String] = ArrayBuilder.buildExpression(configurationPath)

    let $__builder3: [String] = ArrayBuilder.buildExpression("--in-place")

    let $__builder4: [String]
    if recursive {
        let $__builder5: [String] = ArrayBuilder.buildExpression("--recursive")
        let $__builder6: [String] = ArrayBuilder.buildPartialBlock(first: $__builder5)
        $__builder4 = ArrayBuilder.buildOptional(.some($__builder6))
    } else {
        $__builder4 = ArrayBuilder.buildOptional(.none)
    }

    let $__builder7: [String]
    if let cachePath {
        let $__builder8: [String] = ArrayBuilder.buildExpression("--cache-path")
        let $__builder9: [String] = ArrayBuilder.buildExpression(cachePath)
        let $__builder10: [String] = ArrayBuilder.buildPartialBlock(first: $__builder8)
        let $__builder11: [String] = ArrayBuilder.buildPartialBlock(accumulated: $__builder10, next: $__builder9)
        $__builder7 = ArrayBuilder.buildOptional(.some($__builder11))
    } else {
        $__builder7 = ArrayBuilder.buildOptional(.none)
    }

    let $__builder12: [String] = ArrayBuilder.buildPartialBlock(first: $__builder0)
    let $__builder13: [String] = ArrayBuilder.buildPartialBlock(accumulated: $__builder12, next: $__builder1)
    let $__builder14: [String] = ArrayBuilder.buildPartialBlock(accumulated: $__builder13, next: $__builder2)
    let $__builder15: [String] = ArrayBuilder.buildPartialBlock(accumulated: $__builder14, next: $__builder3)
    let $__builder16: [String] = ArrayBuilder.buildPartialBlock(accumulated: $__builder15, next: $__builder4)
    let $__builder17: [String] = ArrayBuilder.buildPartialBlock(accumulated: $__builder16, next: $__builder7)

    return $__builder17
}
1 Like

This makes sense to me. This is more efficient because we avoid the intermediate [String] values for each expression, and instead append directly to the Array, which we ensure isn’t copied by using consuming (right?)

That would give us this implementation, with an additional pair of buildPartialBlock(first:)and buildPartialBlock(accumulated:next:) overloads for each element type we want to support. Works for me.

@resultBuilder
public enum ArrayBuilder<Element> {
  public static func buildPartialBlock(
    first element: Element
  ) -> [Element] {
    [element]
  }

  public static func buildPartialBlock(
    accumulated: consuming [Element],
    next: Element
  ) -> [Element] {
    var result = accumulated
    result.append(next)
    return result
  }

  // ...
}

It does seem useful and reasonable for result builders to support an inout variant of buildPartialBlock(accumulated:next:). I suppose we would have to require all buildPartialBlock overloads either be inout or not inout.

In terms of the performance difference between:

public static func buildPartialBlock(
  accumulated: consuming [Element],
  next: Element
) -> [Element] {
  var result = accumulated
  result.append(next)
  return result
}

and:

public static func buildPartialBlock(
  accumulated: inout [Element],
  next: Element
) {
  result.append(next)
}

I suppose it just depends on if / where we get any implicit copies. What does your intuition say?

I’m happy to prototype inout buildPartialBlock support for result builders and then do some benchmarking.