[Review] SE-0408: Pack Iteration

Hello Swift community,

The review of SE-0408 "Pack Iteration" begins now and runs through September 19th, 2023.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to me as the review manager via the forum messaging feature or email. When contacting the review manager directly, please put "SE-0408" in the subject line.

Try it out!
An implementation of this proposal is provided in PR #67594, behind the flag -enable-experimental-feature PackIteration. Toolchains are provided:

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

What is your evaluation of the proposal?
Is the problem being addressed significant enough to warrant a change to Swift?
Does this proposal fit well with the feel and direction of Swift?
If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

Thank you,
Doug Gregor
Review Manager

23 Likes

+1, no brainer, feels just right. Intuitive syntax!

4 Likes

It is a really necessary feature to fluently use packs!

I have asked about usage something like allUnwrapOrNil in the pitch thread, but afaik I did not receive any response on it. What do you think about this?

1 Like

It's not possible to assign into a single lane of a pack inside of a for-in-repeat loop with this proposal. However, because the compiler knows about the static "shape" of the loop using the shape of the pack expansion, I believe this might be possible as a future extension to definite initialization.

7 Likes

LGTM on face value.

I'm curious how this is implemented. e.g. for:

func stash<T>(_ value: T) { … }

func save<each T>(_ element: repeat each T) {
    for element in repeat each element {
        stash(element)
    }
}

…does that ues runtime dynamism (e.g. existentials), or does it expand at compile time to essentially e.g. for three arguments:

func save<A, B, C>(_ a: A, _ b: B, _ c: C) {
    stash(a)
    stash(b)
    stash(c)
}

I'm guessing it cannot alawys do static expansion, since in cases like public exports from a module it can't know all the type forms at compile time? (Unless pack-using methods are implicitly @inlinable?)

If a type is repeated in the pack (e.g. a pack of {Bool, String, Bool}) does the generated code deduplicate the type-specific bits to any degree?

This is all academic, to be clear. Probably. I am wondering how well this scales (w.r.t. code size, if not runtime performance) when the packs get very long (e.g. imagine SwiftUI using this, with potentially dozens or hundreds of elements).

1 Like

Variadic pack iteration as a feature does not require anything fundamentally new at the implementation level beyond what variadic generics already require.

The basic implementation for variadic generics is the same as for everything else in generics: Swift is capable of running generic functions directly and does not require them to be specialized for a specific generic argument pack. Your function save, if not specialized, will take a bounded array of pointers to type metadata (representing the parameter pack T) and an array of that length of pointers to objects of those types (representing the value parameter pack element). The for loop will iterate over the elements of that pack essentially just like a pack expansion in any other position would.

10 Likes

Even for existentials, if the compiler is able to determine the underlying static type, it can remove the existential and optimise based on that static information.

For instance, I had a model like this:

protocol Schema {
  func decode(_ encoded: some Collection<UInt8>) -> any Collection<UInt8>
}

struct SomeConcreteSchema: Schema {
  func decode(_ encoded: some Collection<UInt8>) -> any Collection<UInt8> {
    canFastPath(encoded) ? FastDecoded(encoded) : SlowDecoded(encoded)
  }
}

extension Collection where Element == UInt8 {
  var someValue: Int { ... }
}

func doSomething(schema: some Schema) {
  let decoded = schema.decode(...)  // technically, an 'any Collection<UInt8>'
  let value = decoded.someValue  
} 

So, Schema.decode returns an existential, allowing conformers to return different concrete types along different paths. When this gets specialised for SomeConcreteSchema, the compiler eliminates all existentials and directly calls FastDecoded<X>.someValue or SlowDecoded<X>.someValue.

I'm just mentioning it as a real-life example that supports what @John_McCall said (at least as I understand his point): generic parameters (and existentials) are conceptual models. The instructions that get generated are a separate question entirely -- things can be fully specialised, fully unspecialised, partially specialised in various ways, etc.

3 Likes

Right. And while variadic generics does need to add some dynamism to handle the dynamic size of a type parameter pack, it doesn’t need to go further. For example, we do not need to box all of the elements of a value parameter pack as if they were existentials, with each value carrying its type dynamically, because we can statically track the element types back to the type parameter pack, which allows the dynamic type of each T to be recovered just as it would be recoverable in non-variadic generics for a type parameter U.

4 Likes

In general would you say that this (just as an example):

func print<each Item>(_ item: repeat each Item, separator: String = " ", terminator: String = "\n")

is more efficient than this?

func print(_ items: Any..., separator: String = " ", terminator: String = "\n")
1 Like

Also, definitely +1 on this pitch, this will make working with variadic generics a lot easier

There’s potential for it to be more efficient. Actual consequences for any one program are hard to predict.

Would it ever be less efficient?

The efficiency of parameter packs is a super interesting topic, but it's not really related to this evolution review except to the extent that this specific proposal encourages wider parameter pack adoption. Can we move this discussion to another thread if you'd like to dig into it further?

6 Likes

I can’t think of any reason it would, except possibly over-eager specialization causing code-size bloat in places that don’t actually improve performance.

But Holly is right, we should take this to a separate thread.

1 Like

In my limited experimentation with parameter packs I think this will nicely smooth off a rough edge with their use. So that’s a big +1 for me.

I think we should be aware of the current bugs with their use though and ensure we address those as part of any future work on this feature e.g Parameter Packs embedded types and structured concurrency - bug or holding it wrong?

1 Like

In my opinion, the for loop-like syntax is unlike other places where variadic generics can be used, where the repeated value is marked by the each keyword. With the for syntax, the repeated value has to be bound to an iteration variable before use. This has the benefit of making it more obvious what value is being repeated, and feeling intuitively similar to a for loop. But it also has the disadvantage that making one variadic value syntactically "privileged" can become arbitrary. For example, there's no reason why this:

for (left, right) in repeat (each lhs, each rhs) {
    guard left == right else { return false }
}

couldn't be written like this:

repeat {
    let (left, right) = (each lhs, each rhs)
    guard left == right else { return false }
}

Going back to what ensan-hcl said earlier in this thread, the allUnwrapOrNil could be written like this:

func allUnwrapOrNil<each Element>(over element: repeat (each Element)?) -> (repeat each Element)? {
    let result: (repeat each Element)
    repeat {
        guard let element = each element else {
            return nil
        }
        each result = element
    }
    return result
}

In this case, the repeated value (each result) is an lvalue. The for loop-like syntax where repeated values are bound at the top would only allow rvalues.

3 Likes

I agree there's a bit of an issue with regards to for() loop being syntactically sugared like this, and I feel it's a more fundamental issue. With the for-loop syntax I have a hard time understanding how you even assign a value to the correct tuple index while enumerating the types that compose that tuple without some painful mirroring or other thing like that.

Also based on the Future directions section of the proposal, it seems that adding support for applying the iteration over existing swift flow control checks will need to be added individually. I feel like it should be possible to make the syntax less sweet and a bit more specialized so it makes expansion more coherent.

I don't think that the syntax can be as simple as repeat { }, not only is this already used in repeat { } while ; which might render parsing difficult, but given how repeat is used in other context, wouldn't it make more sense to define a pack iteration block like so:

func allUnwrapOrNil<each Element>(over element: (repeat each Element)?) -> (repeat each Element)? {
    let result: (repeat each Element)
    repeat each Element {
        guard let value = each element else {
            return nil
        }
        each result = value
    }
    return result
}

The idea is that you have to specify what parameter pack you are iterating over, then inside the block every call concern one element of that pack, if you're using a tuple whose pack expansion comes from the same type pack then you use the syntax each tupleVariable as the lvalue just like proposed by @ellie20 otherwise you deal with the type as whole.

I'll reuse an example from the first proposal to show how you could get the same result as the print function on variadic tuples with this syntax:

func example<each T: CustomStringConvertible>(packElements value: repeat each T, tuple: (repeat each T)) {
  var repeatEachValueComponents: [String] = []
  var repeatEachTupleComponents: [String] = []
  repeat each T {
      repeatEachValueString.append((each value).description)
      repeatEachTupleString.append((each tuple).description)
  }
  print("(", repeatEachValueComponents.joined(", "), ")")
  print("(", repeatEachTupleComponents.joined(", "), ")")

  var repeatEachValueTupleComponents: [String] = []
  repeat each T {
      var repeatEachComponents: [String] = []
      repeat each T {
          repeatEachComponents.append((each tuple).description)
      }
      repeatEachValueTupleComponents.append("(\(each value), (\(repeatEachComponents.joined(", "))))")
  }
  print("(", repeatEachValueTupleComponents.joined(", "), ")")
  

  var repeatEachValueEachTupleComponents: [String] = []
  repeat each T {
      each repeatEachValueEachTupleComponents.append("(\(value), \(tuple))")
  }
  print("(", repeatEachValueTupleComponents.joined(", "), ")")
}

example(packElements: 1, 2, 3, tuple: (4, 5, 6))

// Prints the following output:
// (1, 2, 3)
// (4, 5, 6)
// ((1, (4, 5, 6)), (2, (4, 5, 6)), (3, (4, 5, 6)))
// ((1, 4), (2, 5), (3, 6))

I think this syntax has the advantage of not adding another sugar on the for keyword, in addition it puts the pack being iterated over at the forefront of the enumeration rather than all the way at the back of the first line, it also circumscribes the whole code where "each x" will represent a single type of the type pack also allowing assigning values to corresponding tuple expansions.

Another advantage I thought about while writing this is that it does not require you to enumerate the values of the pack expansions at the outset. If a developer needs to enumerate various types and that one of the values is expensive to get but another check would remove the need to check for that value, it's beneficial to be able to choose when the value is extracted in one iteration.

Here is another convoluted example:

protocol SomeProtocol {
    associatedtype OtherType
    
    func cheapOperation() -> Bool
    func costlyOperation() -> Element?
}

func performCostlyPackIteration<each Element: SomeProtocol>(packElements values: repeat each Element, otherElements otherValues: repeat each Element) -> (repeat each Element.OtherType?) {
    let result: (repeat each Element.OtherType?)
    repeat each Element {
        if !(each values.cheapOperation()) {
            each result = nil
            continue
        }

        let costlyResult = otherValues.costlyOperation()
        // Do something with the result like use switch { } guards and others.
        each result = costlyResult
    } 
    return result
}

The comment in there does a lot of work but the point is that the complexity could go very far and I'm not sure how this could be represented with the for loop syntax while allowing the user to delay the value extraction or even using the pack expansion syntax without making it unreadable.

Under this proposal and the newly added PR #2153, the implementations of Equatable, Comparable and Hashable would look like this:

extension Tuple: Equatable where repeat each Element: Equatable {
    static func == (lhs: repeat each Element, rhs: repeat each Element) -> Bool {
        repeat each Element {
            if each lhs == each rhs {
                continue
            }
            return false
        }
        return true
    }
}

extension Tuple: Hashable where repeat each Element: Hashable {
    func hash(into hasher: inout Hasher) {
        repeat each Element {
            hasher.combine(each hasher)
        }
    }
}

extension Tuple: Comparable where repeat each Element: Comparable {
    static func < (lhs: repeat each Element, rhs: repeat each Element) -> Bool {
        repeat each Element {
            if each lhs == each rhs {
                continue
            }
            return each lhs < each rhs
        }
    }
}

I hope I conveyed my misgivings properly and please do tell me if I'm completely off.
Thank you for your kind attention.

3 Likes

I think that the proposal is really awesome, and I'd love to have support for pack iteration when using variadic generics.

I have experimented with the supplied toolchain for macOS.

I can get the following to compile:

extension Collection {

    func sorted4<each T: Comparable>(by fs: repeat (Element) -> each T) -> [Element] {
        func isLessThan<each X: Comparable>(fs: repeat (Element) -> each X, a: Element, b: Element) -> Bool {
            for f in repeat each fs {
                if f(a) < f(b) {
                    return true
                } else if f(b) < f(a) {
                    return false
                }
            }
            return false
        }

        return self.sorted(by: { (a: Element, b: Element) -> Bool in
            isLessThan(fs: repeat each fs, a: a, b: b)
        })
    }
}

But I would also expect the following (IMO more readable) version to compile:

extension Collection {
    func sorted5<each T: Comparable>(by fs: repeat (Element) -> each T) -> [Element] {
        return self.sorted(by: { (a: Element, b: Element) -> Bool in
            for f in repeat each fs {
                if f(a) < f(b) {
                    return true
                } else if f(b) < f(a) {
                    return false
                }
            }
            return false
        })
    }
}

The compiler error I get is:

.../VariadicGenerics/VariadicGenerics/Variadics.swift:148:21: error: type of expression is ambiguous without a type annotation
        return self.sorted(by: { (a: Element, b: Element) -> Bool in

I assume that the issue has something to do with the extra scope, but I wonder if the latter example is supposed to work in the final version?

Note also that the version that compiles (sorted4) crashes at runtime on an iOS 17 simulator on the line: if f(a) < f(b) { with Thread 1: EXC_BAD_ACCESS

I do have a version without pack iteration working:

struct LessThan: Error {}
struct GreaterThan: Error {}

extension Collection {
    func sorted3<each T: Comparable>(by f: repeat (Element) -> each T) -> [Element] {

        func isLessThan<S: Comparable>(_ f: (Element) -> S, _ left: Element, _ right: Element) throws {
            if (f(left) < f(right)) {
                throw LessThan()
            } else if (f(left) > f(right)) {
                throw GreaterThan()
            }
        }

        return self.sorted { (a: Element, b: Element) in
            do {
                repeat try isLessThan(each f, a, b)
            } catch is LessThan {
                return true
            } catch {}
            return false
        }
    }
}

@Morten_Bek_Ditlevsen Thank you for catching this! We aim to add pack iteration support in closures in the final version. I've added a note in the PR about this.

2 Likes