Another attempt at passing arrays as varargs (with implementation)

Hi all,

Passing arrays to variadic arguments has come up a lot on the mailing lists and forums in the past as a desirable feature, but as far as I can tell nothing has really made it beyond the pitch stage. I’ve been tinkering with the compiler recently and put together a prototype implementation based on some ideas from past discussions which I hope will allow us to explore this design space a little more easily. What follows is the rough draft of what a proposal based on it might look like. I’d be interested in hearing your feedback! I think that whatever form it ends up taking, an array splat feature would be a great addition to the language.

Swift Evolution PR: https://github.com/apple/swift-evolution/pull/1060

WIP Implementation link: https://github.com/apple/swift/pull/25997

as T... Syntax to Pass Array Elements as Varargs

Introduction

This proposal introduces a new use for the as type coercion operator, passing an Array's elements in lieu of variadic arguments. This allows for the manual forwarding of variadic arguments and enables cleaner API interfaces in many cases.

Swift-evolution thread: Thread

Past discussion threads:

Motivation

Currently, it is not possible to pass an Array's elements to a function as variadic arguments in Swift. As a result, it can be difficult to express some common patterns when working with variadic functions. Notably, when designing APIs with variadic arguments, many developers end up writing two versions, one which takes a variadic argument and another which takes an Array.

func foo(bar: Int...) {
  foo(bar: bar)
}

func foo(bar: [Int]) {
  // Actual Implementation
}

This redundancy is especially undesirable because in many cases, the variadic version of the function will be used far more often by clients. Additionally, arguments and default argument values must be kept in sync between the two function declarations, which can be tedious.

Another limitation of the language currently is the inability to forward variadic arguments. Because variadic arguments are made available as an Array in the function body, there is no way to forward them to another function which accepts variadic arguments (or the same function recursively). If the programmer has access to the source of the function being forwarded to, they can manually add an overload which accepts Arrays as described above. If they do not control the source, however, they may have to make nontrivial changes to their program logic in order to use the variadic interface.

Proposed solution

Introduce a new use case for the as operator, which can be used to pass the elements of an Array as variadic arguments. The new syntax is spelled as T..., where T is the element type of the array, and can be used as follows:

func foo(bar: Int...) { /* ... */ }

foo(bar: 1, 2, 3)
foo(bar: [1, 2, 3] as Int...) // Equivalent

The Array used with as T... need not be an array literal. The following examples would also be considered valid:

let x: [Int] = /* ... */
foo(bar: x as Int...)                        // Allowed
foo(bar: ([1] + x.map { $0 - 1 }) as Int...) // Also Allowed

as T... also enables flexible forwarding of variadic arguments. This makes it possible to express something like the following:

func log(_ args: Any..., isProduction: Bool) {
  if isProduction {
    // ...
  } else {
    print(args as Any...)
  }
}

Detailed design

Syntax Changes

This proposal uses the existing as syntax to coerce an array's elements to variadic arguments. Variadic types will now be valid when used on the RHS of the operator, where they were disallowed in the past.

Semantic Restrictions

as T... may only be used as the top-level expression of a function or subscript argument. Attempting to use it anywhere else will result in a compile time error. This restriction is equivalent to the one imposed on the use of & with inout arguments.

With this change, a variadic argument may accept 0 or more regular argument expressions, or a single as T... argument expression. Passing a combination of regular and as T... argument expressions to a single variadic argument is not allowed, nor is passing multiple as T... expressions. This restriction is put in place to avoid the need for implicit array copy operations at the call site, which would be easy for the programmer to overlook. This limitation can be worked-around by explicitly concatenating arrays within the as T... expression.

The as! and as? operators may not be used when coercing to a variadic type. This restriction is put in place because the coercion of an array to variadic arguments will never fail at runtime. Combining an array-to-varargs coercion with another coercion in a single use of the operator is not allowed. This is intended to prevent users from writing code like the following:

let x: Any = ...
foo(bar: (x as? Int...) ?? /* no way to provide a fallback value */)

Instead, users should write something like:

let x: Any = ...
let y = (x as? [Int]) ?? []
foo(bar: y as Int...)

New Diagnostics

A number of new diagnostics will be introduced to improve the experience of working with as T... and variadic arguments. These include:

  • New contextual conversion diagnostics for passing an array to a variadic argument which offer to add the as T....

  • A diagnostic for passing regular variadic arguments alongside coerced array elements with a fixit that performs array concatenation before coercion.

  • Diagnostics to enforce the restriction that as T... only appears as a function or subscript argument.

Alternate Spellings

The main alternative to the proposed as T... syntax is to use a pound-prefixed keyword when passing arrays as varargs. Some of the potential spellings include #splat, #variadic, #passVarargs, #asVarargs, and #arraySplat.

Source compatibility

This proposal is purely additive, so it has no impact on source compatibility.

Effect on ABI stability

This proposal does not change the ABI of any existing language features. Variadic arguments are already passed as Arrays in SIL, so no change is necessary to support the new as T... syntax.

Effect on API resilience

This proposal does not introduce any new features which could become part of a public API.

Alternatives considered

A number of alternatives to this proposal's approach have been pitched over the years:

Implicitly convert Arrays when passed to variadic arguments

One possibility which has been brought up in the past is abandoning an explicit 'splat' operator in favor of implicitly converting arrays to variadic arguments where possible. It is unclear how this could be implemented without breaking source compatibility. Consider the following example:

func f(x: Any...) {}
f(x: [1, 2, 3]) // Ambiguous!

In this case, it's ambiguous whether f(x:) will receive a single variadic argument of type [Int], or three variadic arguments of type Int. This ambiguity arises anytime both the element type T and Array itself are both convertible to the variadic argument type. It might be possible to introduce default behavior to resolve this ambiguity. However, it would come at the cost of additional complexity, as implicit conversions can be very difficult to reason about. Swift has previously removed similar implicit conversions which could have confusing behavior. It's also worth noting that the standard library's print function accepts an Any... argument, so this ambiguity would arise fairly often in practice.

It's also unclear how implicit conversions would affect overload ranking, another likely cause of source compatibility breakage.

Use a leading/trailing ... or * instead of as T...

Many past conversations around passing arrays as variadic arguments have pitched using * or ... as 'splat' operators instead of a heavier-weight expression like as T.... These operators read clearly at the call site, but they conflict with existing operators in the language. as T... has the advantage of avoiding these conflicts, and it's expected that this feature will not be used pervasively throughout a codebase, which helps justify a more verbose spelling.

Make T... its own type

Earlier discussions around variadic arguments pitched various approaches to introducing a new, non-Array type for variadic arguments, or introducing new attributes to control their behavior. Such changes can no longer be implemented without breaking binary compatibility. If a larger redesign of variadic arguments is desired, it might be possible to introduce it as a new feature while deprecating the old style. However, the array 'splat' feature doesn't justify such a large redesign on its own.

Acknowledgements

This proposal incorporates many ideas which have been pitched and discussed on the mailing lists and forums over the past several years. Thanks to all who participated in those discussions! I've tried to link as many of the past threads as I could find.

Thanks also to John McCall and Slava Pestov for laying the groundwork in the compiler recently which enables this feature!

16 Likes

Thank you for making this!

+1 from me.

1 Like

+1 from me on the general idea; I'll defer to others on syntax bikeshedding :)

4 Likes

Wouldn't #splat(array) be the better choice here?

Still, no matter how splatting is spelled, imho that is only workaround for an ugly wart inherited from C.
It's less terrible than the original, but it's still quite complex:

  • It adds a whole type to the language, which has arcane rules that only allow its use in method declarations (you can't have var Int…)
  • That special type is magically "transformed" into an array

Because of all that magic, you can't forward variadic parameters, which could be addressed by the proposal - but what's the result:

  1. We use splatting to turn [T] into T…
  2. In the variadic method, T… is magically transformed into [T]
  3. When the method wants to forward the parameter, it has to use #variadic once again…

So in common scenarios, we would have a repeated back and forth between [T] and T…, and that's just insane for me. It might already be too late for the alternative (getting rid of T… completely, and implement variadics as simple syntactic sugar), and I'm really sad about that, because it would not only make the language more simple, but also more powerful (you can only use variadics with arrays, although the feature would be nice to have with other Collections as well)

I remember when I learnt about variadics in C, I was somewhat enthusiastic for having discovered how the gurus wrote those magic functions which could deal with any number of parameters. Actually, such glorified memories are the only reason I can think of why people kept defending the ellipsis-syntax despite its downsides.

8 Likes

To be clear, the [T] -> T... -> T conversion doesn't have any performance impact, because T... does not actually exist as a type at runtime, and the argument is passed to the callee as an array. I realize that probably doesn't address your all of your concerns, but thought it was worth noting.

Because variadic arguments are always passed as arrays under the hood, I'm not sure it would ever be a good idea to allow arbitrary collections to be passed to variadic arguments, as it would require some kind of copy, or an ABI break

2 Likes

That would be a great feature. Thanks!

I think that we should maintain the same symmetry Python has with *args and **kwargs being the same in the caller definition and the calling point. For example:

// Spreading Arguments
func print1(_ args: Int...) {}
print1(1, 2)
print1([1, 2]...)

// Future Swift: Spreading Keyword Arguments
func print2(_ kwargs: @keyboardArguments Int...) {}
print2(a: 1, b: 2)
print2(@keyboardArguments ["a": 1, "b": 2]...)

I think that adding support for Collection could be considered a pretty big expansion of scope without considerable gain.

Supporting dictionaries and converting them seems closer to @dynamicCallable territory, besides.

2 Likes

The (technically) superfluous rituals are actually one reason why I dislike the whole situation ;-)

I guess there's a misunderstanding here:
It's not about passing arbitrary collections as arguments, but rather allow any type that can be expressed as an array literal to use variadic syntax.

func removeIDs(_ ids: @variadic Set<Int>)
…
// call as
object.removeIDs(1, 3, 5) // √
let set: Set<Int> = [7, 11]
object.removeIDs(set) // √
let anArrayOfInt = [16, 32]
object.removeIDs(anArrayOfInt) // compiler error

So there wouldn't be any magic types with implicit "conversion", but only the types that are actually used (maybe with some decoration - that's what the strawman @variadic stands for).

Arguments have an order, though. Restricting splat to Array conveys that. Passing in a Set as a splat seems like a recipe for confusion. I'm sure that an argument could be made that, because there is an order, that that is enough but it just doesn't seem worthwhile or reasonable to me.

3 Likes

Expanding variadic parameters beyond T... to other types is an orthogonal topic to @owenv's pitch here, I think.

Previous discussions have always collapsed as they expand to encompass everything about variadics; given that @owenv has created an implementation and a focused pitch, I think we would do best to focus on the design in front of us.


Personally, I like what we've got here.

The plan of record from the core team since removing implicit tuple splatting has included an explicit splatting feature; one syntax that was put forward (I think by the core team) was prefix *. This was always in the context of tuples; I think we have a tractable start here with arrays and T...

2 Likes

Thanks for working on this!

The proposed solution is not going to eliminate this in all cases. It introduces the #variadic syntax at the usage site which will sometimes be considered unacceptable. On the other hand, the proposed solution is probably the best we can do with the existing variadic syntax given the source compatibility constraint.

I would really like to see enhanced variadics happen eventually. Even if that pitch does move forward, this pitch fills an important gap, a gap that cannot be filled by enhanced variadics. Specifically, public variadic methods that do not provide an array-accepting overload are sometimes entirely unusable with array values as it is not always possible to write an array-accepting overload outside the module.

1 Like

I really don't like the #variadic syntax as the final syntax and I think that @owenv was right to set that aside in favor of getting everything else working.

9 Likes

In this case, what's the reason to not allow Array<Type> to implicitly convert to/from Type...? It is an array under the hood so why can't we just pass an array? It was/is my natural expectation. I believe I even tried it assuming it would work before having to look it up when it failed.

It seems unnecessary (maybe even misleading) to add syntax to something when _nothing's actually changing. I already shy away from using variadic functions due to this seemingly unnecessary restriction...expecting people to learn specific keywords for an API wouldn't help much.

If implicit conversion is a no go, could an Array version of variadic functions be synthesized?

e.g.

func foo(bar: Int...) {
  // Actual Implementation
}

could compile to:

func foo(bar: Int...) {
  foo(bar: bar)
}

func foo(bar: [Int]) {
  // Actual Implementation
}
2 Likes

Allowing implicit conversion would be a pretty big source breaking change unfortunately, as would be synthesizing an overload. Consider this case from the pitch, assuming we had implicit conversion:

func f(x: Any...) {}
f(x: [1, 2, 3]) // Ambiguous!

In this case, it's unclear whether f(x:) will receive a single variadic argument of type [Int] , or three variadic arguments of type Int .

Also, Swift has generally avoided implicit conversion rules, with good reason imo.

6 Likes

+1. I can't wait to have this feature (in whatever form it takes)! Thank you for your efforts.

That's Type inference ambiguity, so IMO it's fair to require an explicit Type in that type of rare situation.

//broken
f(x: [1, 2, 3]) // ERROR: The type of "x" is ambiguous. Please specify your desired Type.

//Fixed: 
f(x: [1, 2, 3] as Any) // passed as variadic
// or
f(x: [1, 2, 3] as [Any]) // passed as an array of parameters
// or (I'm guessing this would require extra work)
f(x: [1, 2, 3] as Any...) // passed as an array of parameters

Or it could have a similar behavior as printing Optionals:
Warning: Expression implicitly coerced from '[Int]' to 'Any'

If variadics are already being converted to an array...isn't an implicit conversion already happening? If anything it would be more accurate! :wink:

That said, the source breaking nature is certainly an issue. The Fixits could/should be pretty simple and from a (or at least my) usage perspective it seems like the expectation is it should "just work" without any additional thought/syntax from the user. But whether its worth it for what kind of feels like a bolted-on feature anyway...idk.

5 Likes

Just my two cents, but I'd much prefer a spread operator like in JavaScript than #variadic(x) which feels more cumbersome.

let xs = [1, 2, 3]
f(xs...)

This could also be used for flattening array literals which is cumbersome to do now:

let ys = [
 xs...,
 4, 5, 6
]
7 Likes

I'd love to see this and would much prefer it to #variadic(x). It provides a nice symmetry to the variadic parameter declaration.

5 Likes

I really like that this is being addressed again. It's a serious limitation that I've hit repeatedly (most recently with the inability to properly wrap os_log).

I'm not a fan of the #variadic syntax, but I don't have any good alternatives to propose. In particular, a trailing ... operator is rather appealing (hence why multiple people have already suggested it) but it doesn't work because f(xs...) already compiles as an argument of type PartialRangeFrom<_>.

Sticking with the #variadic syntax, my one suggestion there would be to make the parens optional, so it's just #variadic expr. You could still write the parens of course, since a parenthesized expression is still an expression, but the token immediately after expr is always going to be ) (or I suppose ] for a subscript call) so requiring the parens here just feels like noise.

3 Likes

Since we're bikeshedding, based on the above, I'd like to put an option out there that uses no additional syntax but simply uses the type coercion operator:

func foo(bar: Int...) { /* ... */ }

foo(bar: [1, 2, 3] as Int...)
19 Likes