[Pitch] Parameter Packs

It is also the case for all generic functions, whether it involves variadic generics or plain generics:

func someFunction<C>(_: C) where C: Collection { }

type(of: someFunction)
// error: generic parameter 'C' could not be inferred

let x = someFunction
// error: generic parameter 'C' could not be inferred

let y = someFunction<Array<Int>>
// error: cannot explicitly specialize a generic function

Generic closures are high on the list of things I'd like us to add to Swift.

2 Likes

There's an innate conflict here between the declaration of a pack, where the name stands for a sequence and it feels right to pluralize it (e.g. Widgets), and the uses of a pack, where the name is always part of an expansion and stands for one component and so feels like it ought to be singular (e.g. Widget). This is a place where experience with a similar feature (e.g. from C++) could potentially be very helpful; unfortunately, as someone with that experience, I can tell you that I've never found an appealing resolution to that conflict, so my experience is in fact not helpful at all.

Now, this conflict would be resolved if we allowed the use site to bind a different name to the pack element. I can think of a couple different syntaxes for that:

  • for A in Arguments { A? }
  • each A in Arguments { A? }
  • Arguments.map { A in A? }

I don't love any of them, though. And in context, I worry that people won't understand that this is an expansion, so maybe they also need a trailing ..., which just makes the syntax worse. They're also a little awkward when it comes to expanding multiple packs in parallel — the first two, I can imagine something with parens, but for .map I really don't know.

Maybe the biggest problem is that this feels really heavyweight when you're just trying to expand the pack directly, which is very common. That's especially true in variadic generic functions, where we probably declare the type parameter pack T and then immediately turn around and declare a value parameter pack of type T..., or a single parameter of type Stream<T...>, or whatever. It's interesting to note that the naming issue disappears here, too: there's nothing misleading or confusing about just writing Arguments.... So if we can settle on a binding/mapping expansion syntax, maybe we can hybridize that syntax with ..., so that ... when the operand is directly a pack reference means a direct pack expansion without any mapping, and otherwise you have to use the binding/mapping syntax.

(This would also eliminate the formal ambiguities with ..., since you could only use it directly on pack references, which would otherwise be ill-formed. If you wanted to write a pack expansion of half-open ranges, you could just write each lowerBound in args { lowerBound... }.... It would still be visually ambiguous, of course.)

This would also give us a syntax for the dreaded "type of pack expansion no longer refers to a pack" problem, e.g. the type of collections.count..., where every component is concretely an Int but the expression is still a pack expansion.

3 Likes

Sadly, to me this looks like yet another case where it is considered more important to integrate some new shiny feature as soon as possible instead of striving for the best long-term solution.
Copying what C++ did might be cheap, but I don't think it's a good fit — and imo deciding about some smaller pieces like labels for generic parameters and value generics should happen before landing a huge change which could benefit from such preparation.

1 Like

I have only used variadic generics in other languages for educational purposes, but when writing out the examples in Swift, I found my self almost always using the singular name for type parameter packs, and the plural name for value parameter packs. I think I gravitated toward using plural names for value packs because 1) I didn't think to write a separate argument label and wanted to use the plural name at the call-site, and 2) with iteration, you typically use the singular/plural names together, so it's hard to figure out what to call the local variable when you're iterating over a direct pack expansion, e.g. for value in values.... But otherwise, I have a much easier time understanding the code when patterns use the singular name.

4 Likes

There is a relatively extensive alternatives considered section as far as a first pitch goes, and we've been exploring those alternatives for over a year now. It's fine to leave a drive by comment that you don't like the design, but it sounds like you might have thoughts on what would be a better fit for Swift. If you do, that's great and I would appreciate elaboration.

These features are both orthogonal to the fundamental model of abstracting over a variable number of type parameters.

12 Likes

We could always introduce a language model into name lookup so that Widgets... would implicitly introduce the Widget name as well, and Indices... would properly singularize to Index. :slight_smile:

Yeah, I was thinking about something similar in an early variadic generics thread, though only as a tool for writing constraints:

I'm not totally sold either.

Yep, this is my feeling as well, and I wouldn't be opposed the type of regime you're proposing where we have a shorthand syntax for the simplest case and then fall back to a heavier syntax as soon as you want to do anything more complex. I think I'd still prefer something that makes it clear in elements... that we're working with a pack rather than a range bound, but at least it doesn't have the issue of the pack reference being buried deep inside a complex expression.

I feel like we can only really resolve this with one of two things:

  • Explicit pack syntax to make it visually obvious when you're referring to the full pack
  • Explicit destructuring syntax that allows you to introduce a singular name

As you (and John) noted in your previous post(s), there's a fundamental conflict between different usages of a pack that make a single choice for singular or plural look silly in some contexts.

I think the silliness (totally agree with @hborla, by the way, about being drawn to the plural for values and the singular for types) is a symptom of the underlying cause where there are currently different usages sharing the same syntax, and perhaps @John_McCall's diagnosis is correct here:

2 Likes

This feature has been at the top of many wish lists essentially since Swift was released, so I don't think that's a fair characterisation.

4 Likes

I think I disagree that direct expansion vs expansion+map should have a separate syntax/mental model. The mental model that I find intuitive for an expansion of length N is "repeat this pattern N times". That mental model works for both plain pack expansions and an expansion where the pack is wrapped in some other type or expression.

Take this WeatherKit API as an example, written using parameter packs (under this pitch) instead of 7 overloads. The developer documentation uses dataSet1, dataSet2, etc, as the parameter names in the overloads, so in this case dataSet describes a single value of WeatherQuery:

func weather<T...>(
    for location: CLLocation,
    including dataSet: WeatherQuery<T>...) async throws -> (T...)
)

This makes sense to me as "repeat the dataSet: WeatherQuery<T> parameter N times". This, on the other hand, is way harder for me to understand:

func weather<Ts...>(
    for location: CLLocation,
    including dataSets: for T in Ts { WeatherQuery<T> }) async throws -> (Ts...)
)

I don't know how to explain this. It's hard to describe that dataSets is a parameter pack, i.e. separate parameters, repeated. I also don't know that I can get behind putting a for-in syntax or any other mapping syntax into a parameter list :face_with_spiral_eyes:

I think the parameter label issue I brought up can be solved with an API design guideline that for function parameter packs, you should use a separate argument label from the name of the pack. That really only leaves for-in loops as the problem:

for data in dataSet... {} // ???

for-in loops are kinda hard to think about under the repetition pattern model. I think what you really want is just a body of code that is repeated for each value in the expanded pattern. Maybe that suggests we should find a different way to express iteration, or maybe we should just let you call the binding the same name as the pack if you're expanding a pack directly.

6 Likes

We could also take the position that expansion patterns are permitted to consist of multiple statements, so that you could write:

  {
    result = result && array.isEmpty
    doSomething(with: result)
  }...

or, if we feel that we really need some introduction of the expansion maybe you can expand a value pack into a statement block:

array... {
    result = result && array.isEmpty // within this block, `array` refers to a single element of the pack
    doSomething(with: result)
}

or maybe somehow extend do?

do... {
    result = result && array.isEmpty
    doSomething(with: result)
}

of course, all of these preclude using array as a pack within the repeated block, so you couldn't do something like:

for element in elements... {
  print(element, bestMatchNotEqual(to: element, from: elements))
}

I can't really think of a concrete situation where this would be useful, but the for-in at least makes it possible to do, even if naming the pack/element is a little funky.

This motivation is exactly what I consider to be wrong: Features should be added when they fit into the big picture, and not in the by their rank in some wishlist.
Returning to an old comparison ([planning] [discussion] Schedule for return of closure parameter labels (+ world domination ramble) - #7 by Chris_Lattner):
You usually really want a roof when you build a house, but still you finish the walls first.

I don’t necessarily agree with this. The element type of packs could be considered the union of all types making up the pack. Since the types making up the pack are unknown in the generic context, they could be opaque types that expose the constraints placed on the parameter pack’s elements. In pseudo Swift

typealias H = (Bool, Int, String)

H.Element.self // (Bool | Int | String).self

func hello<pack Values>(values: expand Values) {
  (expand Values).Element.self // some Any
}

To be clear, I’m not proposing a union feature, just exploring a conceptual model for the type of pack elements.

1 Like

Picking an opaque type that maximally generalizes a bunch of concrete types is actually a very difficult problem that’s prone to instability (and thus source incompatibility) around API changes.

Doing it for an abstract pack is stabilized by the generic signature the user specified.

4 Likes

The proposal references zip, which already has variants (zip2, zip3, etc.) in frameworks like Combine. Builder APIs, like SwiftUI has already demonstrated the need for variadic generics. Multi-dimensional arrays would also be a nice performance improvement compared to just nesting arrays. Further, math frameworks, especially those used in ML would significantly benefit from variadic genetics (and integer generic parameters but that’s a different discussion). Overall, I think there are already several examples of how parameter packs will be useful.

Yeah, I was mostly referring to the generic case. I think it’s fair to make the hypothetical Element type of a concrete tuple unutterable or, at least, some Any. Making it unutterable would have the benefit of the user explicitly specifying which constraints they want to enforce, by writing something like the following (which is admittedly a mouthful).

(Int, String) as (expand some pack Hashable)

My point was that it can hardly be claimed that this feature has been rushed. But concretely then, why is now not the right time, what is missing that needs to precede this, that this builds on?

1 Like

I don’t think John was pushing for that sort of spelling as-is, nor I. In the original discussions the distinction was made using prefix rather than postfix ...:

func weather<T...>(
    for location: CLLocation,
    including dataSet: WeatherQuery<...T>) async throws -> (...T)
)

You are severely discounting the problems that the absence of this feature causes. As the pitch explains quite clearly, there’s an entire class of APIs which exist today and cannot be expressed without repetition (and the resultant duplication of metadata).

2 Likes

Yes, that code looks like it should work to me. You're instantiating Factory<Foo, Int, String> (so Built := Foo, Args := {Int, String}), therefore the substituted type of Factory.init becomes ((Int, String) -> Foo) -> Factory.

The type of a pack element is an opaque type whose requirements are the same as the requirements imposed on the pack type parameter.

For example, if you have

func foo<T...>(ts: T...) where T: Collection, T.Element: Equatable, T.Index == Int {
  for t in ts { ... }
}

Then the type of t is opaque, but it conforms to Collection, it's Element associated type is Equatable, and it's Index associated type is equal to Int.

That is, it is as if the body of the for loop was its own function with this generic signature:

func bodyOfForLoop<T>(t: T) where T: Collection, T.Element: Equatable, T.Index == Int {}

It's the same generic signature as foo(), except that T is no longer a pack parameter, it's just a regular old generic parameter.

In the for loop though, the type of t is unutterable -- at the implementation level its a scoped opened archetype, which is the same mechanism as the type of an opened existential. Perhaps one alternative is to allow the user to name it, so you can write

for <T> t in ts {
  // ... do stuff with t ...
  print(t)
  // also you could write stuff like this
  let firstElt: T.Element = t[0]
  // or
  print(T.self)
}

The important thing is that the element type of t is scoped to the body of the for loop and it cannot escape.

2 Likes

It's worth noting this is actually invalid unless the generic context in which localVariable appears in has a pack parameter T and a generic parameter U. What you're doing is referencing withTypePacks after substitution, so perhaps something like

var localVariable: (Int, String, Float) -> Void = justOnePack

As for this, then yeah, the proposed matching rule for function types that is used for computing substitutions would necessarily fail because the function type on one side has more than one pack expansion type.

func withTypePacks(label1 foo: T..., label2 bar: U...) // legal function declaration according to bullet 1

var anotherVar: (T..., U...) -> Void // this is illegal according to the last bullet point, right?
anotherVar = withTypePacks(label1:label2:)

Discord won't let me post more than 3 replies, so I'm also including a response to @DevAndArtist's post:

The syntax to explicitly substitute an empty pack type has to be G<> because G with a generic type means to infer arguments from context.

You mean cases with associated values containing pack expansion types? I think that should be supported, yeah.

The .N syntax is only supported for tuple types and it extracts the Nth "static" element of the tuple. So .2 in the pack expansion type T... is not supported, and .2 on the tuple type (T...) is a compile-time error.

We could however allow for a dynamic indexing operation on pack expansion types. You'd need a dynamic length operation, like foo.count returning an Int. The indexing operation could probably use a different syntax, like a subscript: foo[42], which would do a bounds check at runtime.

I think the for loop in the proposal would lower to getting the dynamic length of a pack and then dynamically indexing each element, but I'm not sure we want to expose it in the surface language. The rule for the type of a subscript into a pack is funny, because it becomes an opened archetype, but then it would have to be scoped to something. With a for loop the scope is clear because it's the body of the loop, but with a subscript, it would need to be scoped to the lexical lifetime of the result expression or something funny like that.

Sure.

No, because the length is not a compile-time integer, it's an abstract shape. You can't utter an abstract shape directly, you can only say that two pack type parameters have the same shape.

1 Like