Another attempt at passing arrays as varargs (with implementation)

Evolution Pitches
1

Hi all,

Passing arrays to variadic arguments has come up a lot on the mailing lists and forums in the past as a desirable feature, but as far as I can tell nothing has really made it beyond the pitch stage. I’ve been tinkering with the compiler recently and put together a prototype implementation based on some ideas from past discussions which I hope will allow us to explore this design space a little more easily. What follows is the rough draft of what a proposal based on it might look like. I’d be interested in hearing your feedback! I think that whatever form it ends up taking, an array splat feature would be a great addition to the language.

WIP Implementation link: [DNM][requires-evolution] Explicit varargs expansion expression by owenv · Pull Request #25997 · apple/swift · GitHub

Prefix * Varargs 'Splat' Operator for Arrays

Introduction

This proposal introduces a new prefix * operator which is used to pass an Array's elements in lieu of variadic arguments. This allows for the manual forwarding of variadic arguments and enables cleaner API interfaces in many cases.

Swift-evolution thread: Thread

Past discussion threads:

Motivation

Currently, it is not possible to pass an Array's elements to a function as variadic arguments in Swift. As a result, it can be difficult to express some common patterns when working with variadic functions. Notably, when designing APIs with variadic arguments, many developers end up writing two versions, one which takes a variadic argument and another which takes an Array.

func foo(bar: Int...) {
  foo(bar: bar)
}

func foo(bar: [Int]) {
  // Actual Implementation
}

This redundancy is especially undesirable because in many cases, the variadic version of the function will be used far more often by clients. Additionally, arguments and default argument values must be kept in sync between the two function declarations, which can be tedious.

Another serious limitation of the language currently is the inability to forward variadic arguments. Because variadic arguments are made available as an Array in the function body, there is no way to forward them to another function which accepts variadic arguments (or the same function recursively). If the programmer has access to the source of the function being forwarded to, they can manually add an overload which accepts Arrays as described above. If they do not control the source, however, they may have to make nontrivial changes to their program logic in order to use the variadic interface.

This problem also appears when overriding a method which accepts a variadic argument. Consider the following example:

class A {
  func foo(bar: Int...) { /* ... */ }
}

class B: A {
  override func foo(bar: Int...) {
    // No way to call super.foo(bar:) with bar
  }
}

Proposed solution

Introduce a new prefix * operator, which can be used to pass the elements of an Array as variadic arguments. The operator may be used as follows:

func foo(bar: Int...) { /* ... */ }

foo(bar: 1, 2, 3)
foo(bar: *[1, 2, 3]) // Equivalent

The Array passed as variadic arguments need not be an array literal. The following examples would also be considered valid:

let x: [Int] = /* ... */
foo(bar: *x)                        // Allowed
foo(bar: *([1] + x.map { $0 - 1 })) // Also Allowed

The new operator also enables flexible forwarding of variadic arguments. This makes it possible to express something like the following:

func log(_ args: Any..., isProduction: Bool) {
  if isProduction {
    // ...
  } else {
    print(*args)
  }
}

It also makes it possible to call super when overriding methods which accept variadic arguments:

class A {
  func foo(bar: Int...) { /* ... */ }
}

class B: A {
  override func foo(bar: Int...) {
    super.foo(bar: *bar)
    // Custom behavior
  }
}

Detailed design

Operator

This proposal introduces a new operator to the standard library, prefix *. The operator's implementation will be internal to the compiler, but it can be thought of as effectively having the following signature:

prefix operator *

prefix func *<T>(_ array: [T]) -> T...

In practice, the operator will be declared in the standard library with an unused implementation. It will be flagged with an @_semantics attribute which the compiler will recognize and use to generate the necessary code. However, type checking will otherwise work the same as for any other operator.

Semantic Restrictions

Array splatting using * may only be used as the top-level expression of a function or subscript argument. Attempting to use it anywhere else will result in a compile time error. This restriction is equivalent to the one imposed on the use of & with inout arguments.

With this change, a variadic argument may accept 0 or more regular argument expressions, or a single splatted Array. Passing a combination of regular and splatted argument expressions to a single variadic argument is not allowed, nor is passing multiple splatted expressions. This restriction is put in place to avoid the need for implicit array copy operations at the call site, which would be easy for the programmer to overlook. This limitation can be worked-around by explicitly concatenating arrays within the as T... expression.

Source compatibility

This proposal is purely additive and does not impact source compatibility. The future directions section discusses how this feature can be introduced without placing unnecessary source compatibility restrictions on future features like tuple splat and variadic generics.

Effect on ABI stability

This proposal does not change the ABI of any existing language features. Variadic arguments are already passed as Arrays in SIL, so no ABI change is necessary to support the new * operator.

Effect on API resilience

This proposal does not introduce any new features which could become part of a public API.

Alternatives considered

A number of alternatives to this proposal's approach have been pitched over the years:

Alternate Spellings

A number of alternate spellings for this feature have been suggested:

Prefix/Postfix ...

The existing ... operator has been brought up frequently as a candidate for a 'spread' operator due to its similarity to the existing T... syntax for declaring variadic arguments. However, overloading this operator would introduce a couple of issues.

First, ... can currently be used as either a prefix or postfix operator to construct a range given any Comparable argument. Array does not conform to Comparable, so it would be possible to disambiguate these two use cases today. However, it would mean Array could never gain a conditional conformance to Comparable in the future which performed lexicographic comparison. If the newly introduced operator was extended to also perform tuple splatting as part of a future proposal, overloading ... would also mean that tuples might not be able to gain Comparable conformances, an oft-requested feature.

If ... was reused as a splat operator, it would also be one of the only operators in the standard library with multiple distinct sets of semantics. Previous discussions have expressed regret that the infix + operator has two different meanings in Swift, addition and string concatenation. Even when operator usage is never ambiguous, this overloading of an operator's meaning and semantics is generally undesirable. Because there are a number of pitched alternatives to ... which do not have this issue, it's worth considering those operators first, all else being equal.

as ... as Shorthand for as T..., or as a Standalone Sigil

A previous draft of this proposal used the syntax as T... to expand an Array into variadic arguments. This approach was abandoned because it unnecessarily restates the array element type, and does not naturally extend to heterogenous collections like tuples and generic parameter packs which may eventually want to adopt similar splatting syntax.

One solution to this problem which was brought up is using as ... as a shorthand for as T.... However, this is inconsistent with existing uses for the as operator, which does not currently allow writing as ? as shorthand for as Optional, as [] as shorthand for as Array, or as [:] instead of as Dictionary. Additionally, it's unclear how it would extend to heterogenous collections in a natural way when considering future language features.

as ... syntax has also been proposed which treats ... as a sigil instead of a type. This usage clashes with the the existing role of as, which always has a type on the RHS today.

A Pound-Prefixed Keyword Like #variadic or #explode

Another alternative to *, or any other operator, is to introduce a new pound-prefixed keyword to perform array splatting, with syntax like the following:

f(#variadic([1,2,3]))

This syntax has many of the same advantages as introducing a new operator for splatting. However, the syntax is more verbose, to the point where it may harm readability and ergonomics in contexts where the feature is used frequently. Past discussions have been overwhelmingly in favor of using an operator instead.

Prefix * was ultimately chosen as the proposed spelling for a few reasons:

  • It has no existing meaning in Swift today, reducing source compatibility concerns if the operator is extended to apply to tuples or generic parameter packs in the future.
  • There is existing precedent for using * as a splat operator in languages like Python and Ruby.
  • * has been brought up in the past as a potential tuple splat operator. Using it for arrays and other splat operations as well would be consistent with those future designs.
  • ... can keep its existing, single meaning as a range operator in expression contexts.

The main downside of adopting prefix * is it may impede future efforts to use the operator for pointer manipulation, as described under Future Directions.

Implicitly convert Arrays when passed to variadic arguments

One possibility which has been brought up in the past is abandoning an explicit 'splat' operator in favor of implicitly converting arrays to variadic arguments where possible. It is unclear how this could be implemented without breaking source compatibility. Consider the following example:

func f(x: Any...) {}
f(x: [1, 2, 3]) // Ambiguous!

In this case, it's ambiguous whether f(x:) will receive a single variadic argument of type [Int], or three variadic arguments of type Int. This ambiguity arises anytime both the element type T and Array itself are both convertible to the variadic argument type. It might be possible to introduce default behavior to resolve this ambiguity. However, it would come at the cost of additional complexity, as implicit conversions can be very difficult to reason about. Swift has previously removed similar implicit conversions which could have confusing behavior (See SE-29 and SE-72). It's also worth noting that the standard library's print function accepts an Any... argument, so this ambiguity would arise fairly often in practice.

It's also unclear how implicit conversions would affect overload ranking, another likely cause of source compatibility breakage.

Future Directions

Pointer Manipulation Using *

In the past, it's been suggested that the prefix * operator might be used at some point in the future as a way of reducing boilerplate when writing pointer-manipulation code. This use case was noted in the alternatives considered of SE-29, which recommended that, "'prefix-star' should be left unused for now in case we want to use it to refer to memory-related operations in the future." If prefix * was adopted as the 'splat' operator, it would preclude some pointer-manipulation use cases.

Tuple Splat

Explicit tuple splat has been discussed as a desirable feature to have ever since SE-29 removed implicit tuple splat from the language. At the time, prefix * was put forward as a possible explicit tuple splat operator.

Using * as both an array splat operator and a tuple splat operator in the future would unify the syntax of two very similar operations. Because prefix * is being introduced as a new operator in this proposal, it could be applied to tuples in the future without raising any source compatibility concerns. Unlike ..., there would be no potential issues with allowing tuples to conform to Comparable at some point in the future.

Variadic Generics

Note : This section references terminology and ideas from both the Generics Manifesto and Andrea Tomarelli's Variadic Generics proposal draft.

Variadic generics are another future feature likely to interact with splatting syntax when working with generic parameter packs. At this time, it's unclear whether parameter packs will be implemented as 'variadic tuples' or some new kind of structural type in Swift's type system. Therefore, both scenarios should be considered to ensure the new * operator can be extended to support them if desired.

If variadic generics were implemented using variadic tuples, the * splat operator could be extended to work with them in the natural way as described above. Similarly, * could be applied to a new generic parameter pack structural type, if one existed, with few source compatibility concerns.

In the past, some variadic generics pitches have suggested adding an implicit forwarding mechanism when working with parameter packs. Such a feature would allow users to use dot syntax, subscripts, operators, etc. which are commonly applicable to elements of the pack on the pack itself. If implemented naively, this might lead to ambiguity if * was applied to a parameter pack which contained, for example, Arrays. There are a few ways to resolve the potential issue. One would be to require an explicit map operation when operating on a parameter pack instead of performing implicit forwarding. Alternatively, if implicit forwarding is desired, it would be possible to disambiguate uses of *. This is because *, as proposed for operating on arrays, is only valid as the top level expression of a function or subscript argument. Therefore * applied to a parameter pack would always apply to the parameter pack, and not its component parts. This solution may not work in all cases if generic parameter packs were allowed to contain nested parameter packs which could be flattened. However, that issue would likely arise no matter what syntax was chosen for the feature.

Acknowledgements

This proposal incorporates many ideas which have been pitched and discussed on the mailing lists and forums over the past several years. Thanks to all who participated in those discussions! I've tried to link as many of the past threads as I could find.

Thanks also to John McCall and Slava Pestov for laying the groundwork in the compiler recently which enables this feature!

30 Likes
2

Thank you for making this!

+1 from me.

2 Likes
3

+1 from me on the general idea; I'll defer to others on syntax bikeshedding :)

5 Likes
4

Wouldn't #splat(array) be the better choice here?

Still, no matter how splatting is spelled, imho that is only workaround for an ugly wart inherited from C.
It's less terrible than the original, but it's still quite complex:

  • It adds a whole type to the language, which has arcane rules that only allow its use in method declarations (you can't have var Int…)
  • That special type is magically "transformed" into an array

Because of all that magic, you can't forward variadic parameters, which could be addressed by the proposal - but what's the result:

  1. We use splatting to turn [T] into T…
  2. In the variadic method, T… is magically transformed into [T]
  3. When the method wants to forward the parameter, it has to use #variadic once again…

So in common scenarios, we would have a repeated back and forth between [T] and T…, and that's just insane for me. It might already be too late for the alternative (getting rid of T… completely, and implement variadics as simple syntactic sugar), and I'm really sad about that, because it would not only make the language more simple, but also more powerful (you can only use variadics with arrays, although the feature would be nice to have with other Collections as well)

I remember when I learnt about variadics in C, I was somewhat enthusiastic for having discovered how the gurus wrote those magic functions which could deal with any number of parameters. Actually, such glorified memories are the only reason I can think of why people kept defending the ellipsis-syntax despite its downsides.

9 Likes
5

To be clear, the [T] -> T... -> T conversion doesn't have any performance impact, because T... does not actually exist as a type at runtime, and the argument is passed to the callee as an array. I realize that probably doesn't address your all of your concerns, but thought it was worth noting.

Because variadic arguments are always passed as arrays under the hood, I'm not sure it would ever be a good idea to allow arbitrary collections to be passed to variadic arguments, as it would require some kind of copy, or an ABI break

3 Likes
6

That would be a great feature. Thanks!

I think that we should maintain the same symmetry Python has with *args and **kwargs being the same in the caller definition and the calling point. For example:

// Spreading Arguments
func print1(_ args: Int...) {}
print1(1, 2)
print1([1, 2]...)

// Future Swift: Spreading Keyword Arguments
func print2(_ kwargs: @keyboardArguments Int...) {}
print2(a: 1, b: 2)
print2(@keyboardArguments ["a": 1, "b": 2]...)
7

I think that adding support for Collection could be considered a pretty big expansion of scope without considerable gain.

Supporting dictionaries and converting them seems closer to @dynamicCallable territory, besides.

2 Likes
8

The (technically) superfluous rituals are actually one reason why I dislike the whole situation ;-)

I guess there's a misunderstanding here:
It's not about passing arbitrary collections as arguments, but rather allow any type that can be expressed as an array literal to use variadic syntax.

func removeIDs(_ ids: @variadic Set<Int>)
…
// call as
object.removeIDs(1, 3, 5) // √
let set: Set<Int> = [7, 11]
object.removeIDs(set) // √
let anArrayOfInt = [16, 32]
object.removeIDs(anArrayOfInt) // compiler error

So there wouldn't be any magic types with implicit "conversion", but only the types that are actually used (maybe with some decoration - that's what the strawman @variadic stands for).

9

Arguments have an order, though. Restricting splat to Array conveys that. Passing in a Set as a splat seems like a recipe for confusion. I'm sure that an argument could be made that, because there is an order, that that is enough but it just doesn't seem worthwhile or reasonable to me.

3 Likes
10

Expanding variadic parameters beyond T... to other types is an orthogonal topic to @owenv's pitch here, I think.

Previous discussions have always collapsed as they expand to encompass everything about variadics; given that @owenv has created an implementation and a focused pitch, I think we would do best to focus on the design in front of us.

Personally, I like what we've got here.

The plan of record from the core team since removing implicit tuple splatting has included an explicit splatting feature; one syntax that was put forward (I think by the core team) was prefix *. This was always in the context of tuples; I think we have a tractable start here with arrays and T...

2 Likes
11

Thanks for working on this!

The proposed solution is not going to eliminate this in all cases. It introduces the #variadic syntax at the usage site which will sometimes be considered unacceptable. On the other hand, the proposed solution is probably the best we can do with the existing variadic syntax given the source compatibility constraint.

I would really like to see enhanced variadics happen eventually. Even if that pitch does move forward, this pitch fills an important gap, a gap that cannot be filled by enhanced variadics. Specifically, public variadic methods that do not provide an array-accepting overload are sometimes entirely unusable with array values as it is not always possible to write an array-accepting overload outside the module.

1 Like
12

I really don't like the #variadic syntax as the final syntax and I think that @owenv was right to set that aside in favor of getting everything else working.

9 Likes
13

In this case, what's the reason to not allow Array<Type> to implicitly convert to/from Type...? It is an array under the hood so why can't we just pass an array? It was/is my natural expectation. I believe I even tried it assuming it would work before having to look it up when it failed.

It seems unnecessary (maybe even misleading) to add syntax to something when _nothing's actually changing. I already shy away from using variadic functions due to this seemingly unnecessary restriction...expecting people to learn specific keywords for an API wouldn't help much.

If implicit conversion is a no go, could an Array version of variadic functions be synthesized?

e.g.

func foo(bar: Int...) {
  // Actual Implementation
}

could compile to:

func foo(bar: Int...) {
  foo(bar: bar)
}

func foo(bar: [Int]) {
  // Actual Implementation
}
4 Likes
14

Allowing implicit conversion would be a pretty big source breaking change unfortunately, as would be synthesizing an overload. Consider this case from the pitch, assuming we had implicit conversion:

func f(x: Any...) {}
f(x: [1, 2, 3]) // Ambiguous!

In this case, it's unclear whether f(x:) will receive a single variadic argument of type [Int] , or three variadic arguments of type Int .

Also, Swift has generally avoided implicit conversion rules, with good reason imo.

7 Likes
15

+1. I can't wait to have this feature (in whatever form it takes)! Thank you for your efforts.

16

That's Type inference ambiguity, so IMO it's fair to require an explicit Type in that type of rare situation.

//broken
f(x: [1, 2, 3]) // ERROR: The type of "x" is ambiguous. Please specify your desired Type.

//Fixed: 
f(x: [1, 2, 3] as Any) // passed as variadic
// or
f(x: [1, 2, 3] as [Any]) // passed as an array of parameters
// or (I'm guessing this would require extra work)
f(x: [1, 2, 3] as Any...) // passed as an array of parameters

Or it could have a similar behavior as printing Optionals:
Warning: Expression implicitly coerced from '[Int]' to 'Any'

If variadics are already being converted to an array...isn't an implicit conversion already happening? If anything it would be more accurate! :wink:

That said, the source breaking nature is certainly an issue. The Fixits could/should be pretty simple and from a (or at least my) usage perspective it seems like the expectation is it should "just work" without any additional thought/syntax from the user. But whether its worth it for what kind of feels like a bolted-on feature anyway...idk.

6 Likes
17

Just my two cents, but I'd much prefer a spread operator like in JavaScript than #variadic(x) which feels more cumbersome.

let xs = [1, 2, 3]
f(xs...)

This could also be used for flattening array literals which is cumbersome to do now:

let ys = [
 xs...,
 4, 5, 6
]
9 Likes
18

I'd love to see this and would much prefer it to #variadic(x). It provides a nice symmetry to the variadic parameter declaration.

6 Likes
19

I really like that this is being addressed again. It's a serious limitation that I've hit repeatedly (most recently with the inability to properly wrap os_log).

I'm not a fan of the #variadic syntax, but I don't have any good alternatives to propose. In particular, a trailing ... operator is rather appealing (hence why multiple people have already suggested it) but it doesn't work because f(xs...) already compiles as an argument of type PartialRangeFrom<_>.

Sticking with the #variadic syntax, my one suggestion there would be to make the parens optional, so it's just #variadic expr. You could still write the parens of course, since a parenthesized expression is still an expression, but the token immediately after expr is always going to be ) (or I suppose ] for a subscript call) so requiring the parens here just feels like noise.

3 Likes
20

Since we're bikeshedding, based on the above, I'd like to put an option out there that uses no additional syntax but simply uses the type coercion operator:

func foo(bar: Int...) { /* ... */ }

foo(bar: [1, 2, 3] as Int...)
22 Likes