Function builders implementation progress

Evolution Discussion
1

Hi all,

We've been making quite a bit of progress on the implementation of function builders within the Swift compiler's type checker. While I'm not quite ready to reopen the big design discussion, I wanted to call attention to some of the work we're doing (some of which has made it into Swift releases, some that is still landing in master) to bring it closer to the proposal and get this feature to realize its full potential. The hope here is to get the technology into place to allow us to really explore all of the cool ideas that have come up in the design of this feature, improving the development experience along the way.

Here's a summary of what we've been up to:

  • One-way constraints are a change to the way in which we perform type checking for function builder closures. At a high level, this change makes type checking flow from the "top" of a closure down to the "bottom" (but not backwards). This better matches how normal function type checking works and also provides a significant improvement in type checking performance; technical details are in the code base and early pull request. This was shipped as part of Swift 5.1.1.
  • The new diagnostic architecture allows us to properly diagnose errors deep within function builder closures by honing in on the specific problems that prevent a proper type check. This is technically separate from function builders, but is critical to their usability for developers. The bulk of this landed in Swift 5.2, and we hope the difference is night and day.
  • buildExpression support: one can introduce a static method buildExpression into a function builder type, and it will be called to map each raw expression within a function builder closure to a value that will then be passed to buildBlock. buildExpression can be overloaded to provide some contextual translation of expressions that would be unwieldy with buildBlock alone.
  • #warning and #error support, which are small-but-useful additions.
  • Delayed constraint generation for single-expression closure bodies lets the type checker use more contextual information to type check closure bodies. For function builders, it means that single-expression closures will correctly make buildBlock and buildExpression calls (they were previously skipped in this case).
  • Statement-based translation of function builders makes the implementation model in the compiler match what is specified in the proposal. Prior to this change, function builders were implemented by folding the entire closure into a single-expression closure with one giant expression. With this refactor, it becomes easy to support aspects of statements that cannot be described in expressions. Since then, we've added support for many more kinds of statements:

Not coincidentally, a lot of these changes impact on our ability to evolve the language further in the future, particularly in the area of closures. For example, the delayed constraint generation for single-expression closures could allow code like this to compile:

let f: (Int, Int) -> Bool = { $0.isMultiple(of: 2) }

Right now, one would have to add something like (x, _) in to the closure to show that it takes two parameters.

The generalization of function builders to work with more-or-less arbitrary statements opens up the potential for better type inference through multi-statement closures. For example, right now a single-expression closure can infer a closure result type:

let result = someCollection.map { $0.method() }

but a multi-statement closure does not:

let result = someCollection.map { 
  if $0.someProperty {
    return $0.method() 
  }
  return $0.someOtherMethod()
}

and one will have to provide some type annotations to make this code compile. We should be able to infer the result type of the closure to avoid the need for extra type annotations.

No promises, of course, but it's fun to see where improvements in the architecture and implementation of the type checker can take Swift, especially in making function builders more powerful and more expressive.

EDIT: Updated in early May to reflect the additional support for if let, switch, and so on, as well as reflect what went into Swift 5.2 vs. Swift 5.3 vs. only on master.

Doug

75 Likes
2

Inferring result type of multi-expression closures would be a BIG win! Looking forward to the day this ships, if possible :grinning:

27 Likes
3

I wonder if this would allow omitting the return type declaration on functions and computed properties to further reduce the visual noise:

var radius: Double
var diameter { radius * 2 }
4

@Douglas_Gregor that's a great insight into what's been going on and some fantastic progress made by you and the team. The beauty of a lot of these features is that they concentrate on a lot of 'quality of life' improvements which are a huge step forward and massively appreciated in the day to day world of Swift development. Having posts like this to highlight the extraordinary effort needed to make these improvements possible really brings those achievements to the forefront.

9 Likes
5

This feature would be possible, yes. I would oppose such a feature, for both a philosophical reason and a practical one:

  1. When looking at a declaration, it should be obvious what the type of that declaration is, so that you know how to use it. If you have to reason through the implementation of a function to figure out a return type, it affects code readability. This is the reason Swift doesn't allow one to omit the result type of a function in the declaration, even though we could (technically) implement it in the compiler.
  2. Compile-time performance suffers when there is no barrier between the declaration and body of a function, because it means that every source file that makes a use of that function (e.g., to call it) must also type-check the entire body of the function.

Swift does allow

var circumference = Double.pi * diameter

which I regret due to (1) and has been demonstrated to cause compile-time performance problems in real-world Swift projects (2).

Doug

16 Likes
6

It is fun! Thanks for the update.

+1, both this and not having to explicitly discard unused arguments would be awesome!

4 Likes
7

Am I correct to assume that the opaque return types are what is intended to be used when the return type isn’t known?

8

It depends. Opaque result types are the right tool when you don't want to expose the concrete type you're producing to your clients. If the problem is that you don't know how to write out the type... I'd suggest writing the wrong type and letting the type checker tell you.

Doug

9

Have Xcode 11.4 beta 1 toolchain include these changes?

2 Likes
10

Xcode 11.4 contains Swift 5.2. Only those changes that I’ve said are in 5.2 or earlier will be there. “Master” development snapshots from Swift.org will have the latest.

1 Like
11

Exciting! Might this at last lead to a fix for this longstanding issue?

1 Like
12

It should make that easy to fix.

8 Likes
13

@Douglas_Gregor it would be reasonable if we'd have a document with formalized rules on how and which order the compiler will pick up the build* methods. For example I read the proposal but I still don't understand the difference between buildOptional and buildIf. I also don't know if I'm allowed to overload buildExpression and buildBlock in such a way so that the compiler could choose the optimal solution.

protocol P {}
struct AnyP {
  let base: Any
  init(_ base: P) {
    self.base = base
  }
}

@_functionBuilder
enum Builder {
  static func buildExpression<T>(_ content: T) -> T where T: P {
    content
  }

  static func buildExpression<T>(_ content: T) -> AnyP where T: P {
    AnyP(content)
  }

  static func buildBlock<T>(_ contents: T...) -> [T] where T: P {
    contents
  }

  static func buildBlock(_ contents: AnyP...) -> [AnyP] {
    contents
  }
}

In this example AnyP is the common type. In case all children of the builders closure are the same type, I'd expect the compiler to result in [T]. However if one of the children has a different type I'd expect all expressions to be wrapped into AnyP and then build into [AnyP].
Is that reasonable?

For example:

extension Int: P {}
extension String: P {}

func build<T>(@Builder _ closure: () -> T) -> T {
  closure()
}

// can we expect `[Int]` here?
let array_1 = build {
  1
  2
}

// should be `[AnyP]`
let array_2 = build {
  42
  "swift"
}
1 Like
14

They are the same. buildIf is the first name we chose, but buildOptional is far better to describe what's happening here. Use buildOptional going forward.

It's a syntactic translation, so you can do this. However, I'd recommend not overloading buildBlock more than is necessary for arity (if you can't use variadics), because it can lead to large problem spaces that may not be type checked in a reasonable amount of time. My recommendation is to put the variation in overloads of buildExpression if you need to have it, so that buildBlock can remain simpler.

It doesn't work, mainly because your AnyP overloads are technically more specialized (better matches) than your generic ones. The second issue you'll run into is that the choice of buildBlock cannot influence the choice of buildExpression. Each expression in the closure is mapped to a separate statement, e.g., the body of your array_2 closure becomes:

let a1 = Builder.buildExpression(42)
let a2 = Builder.buildExpression("swift")
return Builder.buildBlock(a1, a2)

You can reason about the type checking behavior from the syntactic transformation. The buildBlock picked for buildBuild(a1, a2) cannot influence the types of a1 and a2 from prior statements.

To your original question:

I consider the syntactic transform to be those formalized rules. However, I agree that we should expand and clarify how each of the function builder entry points is used in the syntactic transform. Revising the function builders proposal is on my list, still, but I'd like to have an answer for for..in loops before bringing up a new version of the document.

Doug

4 Likes
15

Just a quick clarification, buildOptional will only be available with Swift 5.3 right?

16

Yes. Looks like we only added it a couple of weeks ago to master. I recommend using snapshots from master if you want to experiment with function builders.

Doug

2 Likes
17

Thanks for clarification, I was a bit afraid of that, but at least I know that it will be possible when variadic generics will be a thing. Then instead of overloading buildExpression for transformations into AnyP I‘d overload buildBlock in heterogenous and homogenous variadic ways, where the former overload returns [AnyP] and the latter [T].

Thanks again, hopefully we can start working on variadic generics soon. :slight_smile:

1 Like
18

All these improvements will make SwiftUI much easier to use, can't wait :)

19

A PR for for in landed.

https://github.com/apple/swift/pull/31543

@Douglas_Gregor would it make sense to call the function buildCollection instead? This would allow us to preserve some flexibility for the future. Or do we want to differentiate between things like buildArray, buildDictionary instead of overloading them like buildBlock or buildExpression?

20

As implemented, it is building and passing an array to buildArray, hence the name. This follows along with the proposal's use of buildOptional and buildEither (if the standard library had an Either type!), as essentially describing the building blocks rather than the specific language constructs.

I think what we're developing here is two different kind of "build" function in function builders. There are language-construct-centric functions like buildExpression, buildFinalResult, and buildBlock; and there are more fundamental building blocks like buildOptional, buildEither, and builldArray that can be used by multiple constructs (if and switch use buildOptional/buildEither, for..in uses buildArray but in theory other loops could).

Of course, all names are subject to revision. I wanted to get the full space of function builders mapped out so we can think about them, and experiment with them, as a whole.

Doug

10 Likes