That’s exactly what I’m in favor of. The compiler magic could be replaced with proper support for type values (fixed-length arrays, etc.) in the future.
2 Likes
Would the Element type be an opaque type?
typealias ElementOf<X...> = (X...).Element = some Any
typealias ElementOf<X... : Equatable> = (X...).Element = some Equatable
…actually, I guess not, because that would imply they all share the same concrete type.
Maybe, in "code", documentation, warnings, etc. it would be appear as any Whatever, but in implementation it's not actually a sequence existential values. After all, the compiler would be optimizing away all that loop-like code since it knows the number of elements statically.
While I do believe that for loops over packs should be allowed, I don't think they should be treated as the standard sugar over a sequence. I think they should be something akin to the template for syntax that nearly made it in to C++20. It should treat the loop as if you had written the code out separately for each type in the pack.
This would be doable as a library extension if we had a macro system (possibly with slightly different syntax), but one of the most important lessons to be learned from C++ is that while moving things into the library level is nice when it's possible; language features shouldn't be forced into being library features just for the sake of it.
Besides, if we ever do get such a code generation or macro system (the hygienic kind), then a for loop over a parameter pack will become sugar over code you can write yourself naturally.
I think one of the strongest arguments against the "superpower tuples" model is the ambiguity that you pointed out later in your post:
The issue with modeling variadic generics with tuples rather than a new pack concept is that a tuple type is a valid type for one of the constituent generic arguments. When you pass a variadic tuple as an argument to a variadic generic parameter, is the tuple type itself a single generic argument, or does it splat its element types? We could disambiguate by applying ... or some other operator to a tuple value to splat its elements, but that's easy to forget, and if/when tuples gain the ability to conform to protocols, forgetting to write ... would possibly typecheck in a way that the programmer didn't intend.
12 Likes
Maybe there could be a specific compile-time error when that ambiguity emerges, requiring you to specify which you intended?
It’s hardly an elegant solution, but it still seems preferable to adding an entirely novel kind. Unconstrained generics like that aren’t common in practice, either.
Actually, now that I think about it, you could also refuse to unpack tuple literals in variadics. There’s no actual reason to use them there, after all.
let input = (1, 2, 3)
struct Foo {
vals: [Int]
init(vals: Int...) {
self.vals = Array(vals)
}
}
// ✅
Foo(vals: input)
// ✅
Foo(vals: 1, 2, 3)
// ❌: prohibit tuple literal unpacking at call site of variadic
Foo(vals: (1, 2, 3))
2 Likes
Right - it couldn't be an opaque type, because opaque types are, well, opaque "non-names" for a single underlying type.
But I do think there is significant overlap between existentials (and especially collections of existentials) and variadic generics, and the language is currently not well-equipped to deal with either of them comfortably because we don't have a concept of introducing new generic type parameters in the middle of a function. That's why, in the previous thread, I suggested that adding this would be a good step towards variadic generics:
So I think it would be reasonable for the element type to be an existential, which the optimiser could then resolve during generic specialisation.
Also, I don't think it's true that the compiler always statically knows the number of elements. You can have unspecialised generic types today, and presumably variadic generic types may also be unspecialised in some circumstances.
2 Likes
I'm thrilled to be discussing this proposal. It's an important missing feature, and I think this is going in the right direction. I'd love to get to write some code against it.
My primary concern is that we have different models of using parameter packs, one at the type level and one at the value level. At the type level, we have an explicit expansion syntax: parameter packs are placeholders for a single pack element that occur within some arbitrary pattern, and that pattern is evaluated for element of the parameter pack to produce a list. This is a powerful model that is similar to C++ variadic templates. It allows for two kinds of transformations:
- An arbitrary map operation. Given a parameter pack
Ts, one can apply a type mapping operation to each element in the pack. For example, we could create a tuple of arrays of all of the element types of a pack of collections like this:typealias ElementsTuple<Ts: Collection...> = ([Ts.Element]...) typealias Example1 = ElementsTuple<[Int], Set<String>, [String: Int]> // produces ([Int], [String], [(key: String, value: Int)]) - An implicit zip operation. Given two parameter packs
TsandUs, one can perform a pairwise expansion that zips them together, like this:struct ZippedPairs<Ts...> { typealias Inner<Us...> = ((Ts, Us)...) } typealias Example2 = ZipperPairs<Int, UInt>.Inner<Float, Double> // produces ((Int, Float), (UInt, Double))
These are both useful transformations made possible by the explicit pack expansion approach used at the type level in this proposal.
At the value level, the model in this proposal is different. Function parameter packs are heterogeneous tuples, which implicitly expand when forwarded. For example, this is from the proposal:
func forward<T...>(_ xs: T...) { return onwards(xs) }
func onwards<T...>(_ xs: T...) { /**/ }
xs is a tuple, but it forwards as if it were expanded into separate elements that are then collected back into the parameter pack. If you wanted to forward xs as a single tuple value, you would need to bounce through a non-variadic parameter:
func forwardAsSingle<T...>(_ xs: T...) { return onwardsAsSingle(xs) }
func onwardsAsSingle<T>(_ x: T) { return onwards(x) }
Because of the implicit expansion, the proposal has a "magic map" operation using a closure-like syntax. Here's an example from the proposal:
func flattenAll<Ts...>(_ seqs: [Ts?]...) -> ([Ts]...) {
return seqs.map { $0.compactMap { $0 } }
}
This is where the difference between the type-level and value-level approaches shows up, because we've used pack-expansion syntax in the type system (return type is ([Ts]...)) and a completely different "magic map" syntax with a pseudo-closure (seqs.map { /* body of closure-ish with $0 }) to produces values of that type. This bothers me for two main reasons:
- The same conceptual operation is expressed in two very different ways even though the common case is that you want those things to match, to so have to learn two different things and rectify them.
- The capabilities at the type and value level differ, because there is no magic "zip" for values. That means you can express things easily in the type level (like
ZippedPairs) that cannot be expressed at the type level.
So, I think we have to pick one model and use it for both types and values, which means choosing one of these options:
- At the type level we say that a type pack is really a tuple type that gets implicitly expanded. We extend/change the "magic map" syntax so it works for both types and values, and (if we care about the capability) add a "magic zip" syntax that works for both.
- At the value level we use explicit pack expansions. A value parameter pack must always be expanded (e.g., with
...). We have to either figure out how to deal with the ambiguities of...in an expression context or pick a different syntax, but again: I think it needs to be consistent between types and values either way.
I have a really hard time seeing #1 work out. With #1, the magic-map closure syntax is a little bit of a stretch for values, because it's not a real closure---more like a generic closure, which isn't otherwise expressible in the language. It also doesn't extend will to types (where we don't have closures!), so we'd need to rethink it. And personally, I really struggle with the dual nature of parameter packs that are tuples in some places but expand into multiple elements in different places. This feels like a place where we want to be explicit.
#2 is one conceptual notion (expansion) that covers map and zip naturally, with very little syntax (postfix ...) that has direct parallels for both types and values. For example, here's flattenAll with pack expansions for values as well:
func flattenAll<Ts...>(_ seqs: [Ts?]...) -> ([Ts]...) {
return (seqs.compactMap { $0 }...}
}
Going from [Ts?] to [Ts] involves the compactMap, so the structures line up. Map (which we're doing here) and zip are both easy. For example:
func zipValues<Ts..., Us...>(firsts: Ts..., seconds: Us...) -> ((Ts, Us)...) {
return ((firsts, seconds)...)
}
Note the parallel structure in the return type and return value of zipValues: one notion of pack expansions, same syntax, same capabilities.
Now, we cannot ignore the ambiguity between the postfix ... here that's used to expand values and the existing postfix ... that's used to form ranges that go to the end of a collection. This trips us up if we were to have, say, a parameter pack of Comparable values:
func ranges<Ts: Comparable...>(values: Ts...) {
let a = (values...) // this can be either (a) a tuple contains the values expanded out, or
// (b) an error, because the ... forms a ClosedRange but Ts is not expanded anywhere
}
In this case, only (a) type-checks, but we can do some sneaky things to make a true ambiguity:
func acceptAnything<Ts...>(_: Ts...) { }
func ranges<Ts: Comparable..., Us...>(values: Ts..., otherValues: Us...) /* where Us and Ts have the same arty */ {
acceptAnything((values..., otherValues)...)
// either (a) values is expanded into each tuple passed to acceptAnything so arguments have types (Ts..., Us)..., or
// (b) values and otherValues are expanded pairwise and the arguments have types (ClosedRange<Ts>, Us)...
}
I propose a rule: if any of the values one refers to within the pattern of a ... expression are parameter packs, they are expanded by the .... This corresponds to interpretation (a) in both examples above, and is not source-breaking because existing Swift code does not have parameter packs.
This does make it hard to have parameter packs that are members of something, because then answering the question "is this a parameter pack?" might be tricky. For example:
struct StoresAPack<Ts...> {
var tuple: (Ts...)
}
A value of tuple type is not a pack, so while we can access the tuple property, we can't splat it, e.g.,
extension StoresAPack {
func printMe() {
print('[', tuple..., ']') // error: no operator `...` on value of type `(Ts...)`.
}
}
There are a couple of potential solutions here, none of which are wonderful but all of which are doable:
-
We could have a third overloaded meaning of postfix
...when applied to an expression that doesn't have a parameter pack in it, which is that it performs some explicit "expansion" operation. TheprintMeabove would be well-formed and just work when the argument to postfix...is a tuple. -
We could add a special operator (let's call it
*for kicks) to do the operation described in (1), so the print operation would look something like:print('[', *tuple..., ']') -
We could allow the introduction of a local value pack whose initializer must be a tuple, where that tuple will be implicitly expanded:
let values... = tuple
print('[', values..., ']')
Of these, (1) seems cleanest for users, (3) is safest because it's easiest, and (2) will likely have us in an endless bike shed.
Anyway, that's my Grand Proposal. What do y'all think?
Doug
21 Likes
The definition of ElementsTuple should be ([Ts.Element]...), right? Otherwise I don't understand how this example should work.
If we went this way, the model for how to work with variadic generics in Swift would become quite similar to how parameter packs are handled in C++, which could be an upside, as it is probably the language where this feature is currently used the most.
One downside of this approach is that we can do a pack expansion only on a single expression containing a parameter pack value. Consider e.g. the zip example from the original pitch. We have the following map operation there:
To be able to express this without such a map function but just with an explicit pack expansion, we'd still need some kind of generic closure. It could e.g. look like this:
let values = try? ({
guard let nexts = iterators.next() else { throw Stop.iterating }
return nexts
}())...
This would be very similar to the C++ way of tackling such a problem and I would probably be fine with it in Swift, but for more complicated expressions involving closures it could definitely be quite challenging to find out, how exactly they are getting expanded. For such a task, I could definitely see a point that the magic map or something similar could be a bit clearer...
Case (a) was probably meant to be something else?
Would that mean that we could expand tuples into function parameters in general? Like e.g. would something like this work:
let tuple = (5, "Hello")
print(tuple...) // prints '5 Hello'
?
Or this:
func foo(_ bar: Int, _ baz: String) {
// ...
}
foo(tuple...)
?
If these worked, I could imagine some problems with using the ... syntax again, as soon as we have SE-0283.
I wonder how the proposal would read if it would use pack keyword instead of the ... suffix. The whole thread is really hard to parse, at least for me. I also don't think that ... syntax will be extendable in the future, it anything it will probably become even harder to read.
5 Likes
Yes, you're right! Fixed, thanks.
Right. I'd probably be tempted to put that closure into a file-private extension on IteratorProtocol to make it simpler.
.., yeah it was meant to be a tuple with the values from values. I've fixed the comment in my original post. (Thanks again)
Yes, this form should be able to expand any value of tuple type into its separate elements. (And yes to your second example)
Are you thinking about the problem of a user-defined ... operator being defined on a tuple type, or something else?
Doug
1 Like
I’m super tripping over the constraint that parameter packs are saturating when passed to other methods. It really feels like you should be able to at least prepend, but ideally also append known elements to a given parameter pack.
Additionally I’d love to have a type level dropFirst/Last operation that shortens a parameter pack by 1. You’d also need the ability to detect an empty parameter pack.
What’s the logic behind this constraint, and can we work around it?
5 Likes
No, I'm thinking about that as soon as tuples become Comparable the syntax tuple... would become a valid partial range expression, wouldn't it?
2 Likes
I kind of agree, but also sort of disagree.
I think what you're saying about #1 echoes what I've been saying about iterating packs using for loops, and the idea that it would introduce a local generic parameter. If we replace "for loop" with "forEach function", we arrive at what you're saying about generic closures.
So perhaps we need to add generic closures? Perhaps some aspects of variadic generics are not really viable until we plug some of these long-standing gaps in the generics model?
As for the disconnect with the type system, while it doesn't support closures, it also doesn't need to express imperative code - only types. But I'm not sure we couldn't express things like (T...) -> ([T]...) using something resembling a map function: T.map([T]). It looks a bit strange compared to our existing syntax, but all of this new syntax looks strange and very delicate.
Your solution seems to revolve around making the imperative language syntax more like the type system, so we use magic, extremely terse syntax rather than magic functions. Personally, I find syntax like this rather opaque:
((firsts, seconds)...)
But what would it look like if we went the other way? And introduced some sort of function syntax in the type system?
struct Foo<T> {}
func zipAndWrap<Ts..., Us...>(firsts: Ts..., seconds: Us...) -> ((Foo<Ts>, Foo<Us>)...) {
return ((Foo(firsts), Foo(seconds))...)
}
func zipAndWrap<Ts..., Us...>(firsts: Ts..., seconds: Us...) -> zip(Ts.map(Foo<Ts>), Us.map(Foo<Us>)) {
return zip(Ts.map { Foo($0) }, Us.map { Foo($0) })
}
The benefit of something like zip(T.map(Foo<T>), U.map(Foo<U>)) is that it is relatively easy to read, without learning new rules or grappling with the idea that Ts refers to n different types each time you use it. With the function syntax, it is quite clear that each T becomes a Foo<T>, each U becomes a Foo<U>, and the result contains each of these, zipped.
1 Like
A quick note that this is how Python solves the problem with *args, but only in function forwarding situations.
Thanks! This (and @Douglas_Gregor’s post) help shed some light on why such a design might be undesirable.
Doug, I like the focus on aligning the value- and type-level syntax, but I come away more convinced than ever that the reuse of the ... syntax should probably just be abandoned. If Swift’s history and existing features were different it might fit well, but IMO the potential overlap with existing uses of ... in both type and value position are really good reasons to prefer a different spelling.
Besides the grammatical ambiguities that can (maybe?) be worked around with some clever rule design, I find the confounding of local reasoning that I mentioned in the previous thread to still be highly problematic, doubly so if we’re considering introducing such an issue to value-level reasoning as well. With Doug’s rule (paraphrased) “... means a parameter pack expansion if any mentioned value in the pattern is a parameter pack,” a user reading the code would have to inspect each name in the (arbitrarily complex) pattern expression to determine if it’s a parameter pack (or conversely, inspect all in-scope parameter packs and check for any usages) before they could confidently understand to what language mechanism a mention of ... refers to.
If we consider solution (1) to the tuple... problem above, then the inspection process may need to consider in-scope names which may not be in the same file, or even the same module. I am not so confident that the vast majority of future uses of ... with variadic generics will be so unambiguous on their face that these local reasoning issues will be exceedingly rare.
I’m sympathetic to the desire to avoid an “endless bikeshed,” but I really feel this is a situation where almost any alternative would be preferable. The proposal states that ... was chosen for analogy with normal variadic parameters, but variadic generics are fundamentally very different beasts, to the point that I think the analogous syntax might be actively undesirable even without the issues that have already been raised. I don’t think we should resign ourselves to an (IMO fairly significant) issue just because bikeshedding could potentially get out of hand. (Deliberately biting my tongue regarding my personal preferred alternate syntax as a demonstration of good avoid-the-bikeshedding faith. 
)
Another option would be to settle the model for variadic generics without fully settling on a terse syntax just yet. We could introduce the necessary expected-to-be-deprecated attributes/compiler primitives (e.g. @variadicGenericParam(T), @variadicExpansion(T), @variadicExpansion(ts.map { ... }), etc.) and leave the question of the right terse syntax to be resolved in a future proposal. This might not result in the prettiest code, but it wouldn’t cause issues for clients of variadic generic APIs. One downside is that this would prevent some future disambiguation strategies such as Doug’s since we would have in-the-wild uses of variadic generics once the new syntax is settled, but I think we could still disambiguate based on whether the variadic generic parameter was declared using the ‘new’ syntax or not. And IMO, a syntax that requires such disambiguation rules is undesirable anyway.
1 Like
I have no such compunction 
Given that we are considering, almost in parallel, the addition of regex literals to the language, I think it'd be quite nice to consider seriously the notation T* for zero or more Ts. Unpacking/splatting could then adopt the corresponding prefix syntax *T à la Python.
The result would be familiar to users of other languages and be usable in both type and value positions without stepping on any standard library operators currently in use.
It may be a good time to refresh our collective memory that implicit tuple splatting was explicitly removed from the language in SE-0029, with the core team stating (emphasis added):
We acknowledge that we're removing a useful feature without providing an equally expressive drop-in replacement. However, maintaining this behavior in the type checker is a severe source of implementation complexity, and actively interferes with our plans to solidify the type system. We feel that removing the existing behavior is a necessary step toward stabilizing the language and toward building a well-designed alternative feature for explicit argument forwarding in the future.
In other words, explicit splatting with an operator or some other spelling is the plan of record.
7 Likes
It would be familiar to users of other languages as the pointer type (postfix in C, prefix in Go and Rust) and dereference operator (prefix in C). :)
4 Likes
IMO we should not consider ourselves too constrained by conflicting/similar syntax in other languages, particularly not to the point that we are harming the usability/readability of the feature for Swift programmers who have not used the alternate language(s) in question.
Users coming from other languages already have to contend with bits of syntax that don’t match up (e.g. Rust’s use of postfix ?), and variadic generics are a pretty advanced feature. I think it’s fair to expect users encountering VG to do a little learning for themselves about the syntax Swift uses.
7 Likes
I’ve given this some thought, and it occurs to me that there are two common requests people have for variadics. If we are going to overhaul the current system to implement one, I think it is worth acknowledging both. I feel they are best summarized in terms of other ways to implement or approximate them.
-
func foo<E: Sequence>(vals: E)Many programmers (possibly just myself, but I think this is common) want shorthand for expressing an arbitrary sequence[1] of values, which is what all variadics in Swift currently do, without requiring the use of this shorthand. For instance, the ability to pass either a variable number of
Elementinstances or oneSequenceofElementinstances, possibly even lazy.Right now, this is impossible: a variadic function cannot be called with some
Sequenceinstance, even though many (I’d wager most) would work as written if given one. For obvious reasons, all of them would work as written if given anArray, though this is currently prohibited for various reasons[2]. -
func foo<E: TupleLike>(vals: E)The main goal of this pitch, with generic parameter packs, is to allow for arbitrary heterogeneous values to be passed to a function. This is very similar to accepting any tuple, as tuples are composed of a certain number of ordered values of certain types. Indeed, the main difference with parameter packs is simply the ability to add type constraints and otherwise work with them without specifying their exact composition.
To me, this is effectively an existential: nothing is required of the values except their conformance to certain protocols, and that means they must be treated in the body accordingly. In contrast to standard associated types, references to specific implementations can’t be easily made (you don’t even know if there are any implementations, let alone how many), and without requiring the implementations to be the same a lot of functionality is demonstrably impossible.
Both directions are useful, though not necessarily in the eyes of the same people. This pitch is about #2, but I think #1, where the variadic is mainly intended as shorthand for a sequence, should be kept in mind as a common use.
-
Certain uses may obviously warrant stricter requirements, particularly
Collectionor evenRandomAccessCollection, but a lot of things don’t. ↩︎ -
Many of these reasons are relevant to proposals here: potential ambiguity about nesting, optimizability at compile-time, etcetera.
I think the latter is only solvable by improving the generics model to have protocols with associated values (i.e. a fixed-size array), which is way beyond the scope of this pitch. It is telling that C’s fixed-size arrays are currently imported as tuples. ↩︎
2 Likes
Actually to me the most useful form of this would be for the set of variadic generic stuff to be an unordered set, allowing for something like:
struct Scope<T...> {
init(_ items: T...) {
self.store = items.keyed(by: T....self) // magic
}
let store: [T...Type: T] = [:]
func storing<X>(_ x: X) where X in T... -> some Scope {
let newStore = store + [X.self: x]
return Scope(store: newStore)
}
subscript<X>(stored: X) -> X {
store[X.self]
}
}
let scope = Scope(42, "help", 93.0)
let scope2 = Scope(42, "help")
typealias Foo = (Int, String, Float)
let foo: Foo = (scope[Int.self], scope[String.self], scope[Float.self]) // compiles
let foo2: Foo = (scope2[Int.self], scope2[String.self], scope2[Float.self]) // doesn't compile because scope2's variadic set of types doesn't contain a Float
Although, maybe you can think of a more elegant way to do this. I realize of course that all previous implementations of variadics are ordered, so I might need a new word besides "variadic" for a case where the collection is unordered. Hmm.