Amend SE-0216 Dynamic Callable: Reduce Overloads

Amend SE-0216 Dynamic Callable: Reduce Overloads

Introduction

SE-0216 introduced support for user-defined dynamic "callable" types. In this proposal, a type marked @dynamicCallable must implement one of the following methods:

func dynamicallyCall(withArguments: [T1]) -> T2
func dynamicallyCall(withKeywordArguments: [S : T3]) -> T4

The original proposal stated that when both these methods are provided in the primary declaration of the type, the type checker finds the most specific match.

This proposal demonstrates that allowing both methods to be provided for @dynamicCallable can lead to unclear, surprising behavior, and proposes that the user must choose one method from the two.

Motivation

The proposed amendment is motivated by three things:

1. Unclear which method is being called, when no keywords are given

In a language that supports both normal arguments and keyword arguments, both of which are supported by dynamicallyCall(withKeywordArguments:), in which case keywords will be empty strings when a normal argument list is provided.

@dynamicCallable
struct T {
    func dynamicallyCall(withKeywordArguments: KeyValuePairs<String, Int>) -> Float { ... }
}

let x: T = ...
x(a, b)
// desugared: x.foo.dynamicallyCall(withKeywordArguments: ["": a, "": b])
x(a: a, b: b)
// desugared: x.foo.dynamicallyCall(withKeywordArguments: ["a": a, "b": b])

But when dynamicallyCall(withArguments:) is also implemented, a dynamic call with no argument labels will be desugared into the array initializer instead of the dictionary literal initializer.

@dynamicCallable
struct T {
    func dynamicallyCall(withKeywordArguments: KeyValuePairs<String, Int>) -> Float { ... }
}

let x: T = ...
x(a, b)
// desugared: x.foo.dynamicallyCall(withArguments: [a, b])
x(a: a, b: b)
// desugared: x.foo.dynamicallyCall(withKeywordArguments: ["a": a, "b": b])

Now we have two different ways of representing the same case: the array form and the dictionary literal form, and the latter has the full representational power to express calls without keywords. In languages that support keyword arguments, calls with keyword arguments are usually handled the same way as non-keyword arguments, there's hardly any motivation for implementing a separate dynamicallyCall(withArguments:) when dynamicallyCall(withKeywordArguments:) is already implemented by a dynamic callable type.

2. Potentially surprising behavior when two methods have different return types

When dynamicallyCall(withArguments:) and dynamicallyCall(withKeywordArguments:) are both provided and have different return types, the behavior can be surprising: It is not obvious which method a dynamic call is being desugared to.

@dynamicCallable
struct T {
    func dynamicallyCall(withArguments: [Int]) -> String { ... }
    func dynamicallyCall(withKeywordArguments: KeyValuePairs<String, Int>) -> Float { ... }
}

let x: T = ...
let _: String = x()
// desugared: x.foo.dynamicallyCall(withArguments: [])
let _: String = x(a: a)
//              ^ error: expression returns Float
let _: Float = x()
// desugared: x.foo.dynamicallyCall(withKeywordArguments: [:])

One may argue that this is a customization point that libraries can leverage to get intended behavior. However, this customization point is not motivated by concrete use cases and can lead to unintended behavior, so it should be reassessed and/or disallowed.

3. Implementation is significantly harder than claimed in the original proposal

The implementation of this proposal is still in progress. @dan-zheng's implementation hit a blocker due to difficulty in resolving overloads described above in the constraint system. Dan has detailed explanations here.

Implementation difficulty is a minor argument in this proposal. This proposal is mainly motivated by #1 and #2.

Proposed solution

As explained above, there's no concrete use case or demonstrated merit in allowing the user to provide both methods and resolving dynamic calls based on "most specific match", and at times it makes the call sites unclear and behavior surprising. We should not allow such customization until there's a convincing use case that would form a new proposal to enable this customization point.

Dynamic callable types must choose between dynamicallyCall(withArguments:) and dynamicallyCall(withKeywordArguments:). When both methods are provided in the primary type declaration, the compiler emits an error.

@dynamicCallable
struct T {
    func dynamicallyCall(withArguments: [Int]) -> String { ... }
    func dynamicallyCall(withKeywordArguments: KeyValuePairs<String, Int>) -> Float { ... }
}
@dynamicCallable
^~~~~~~~~~~~~~~~
error: 'T' has both 'dynamicallyCall(withArguments:)' and 'dynamicallyCall(withKeywordArguments:)' methods, but only one of them should exist

func dynamicallyCall(withArguments: [Int]) -> String { ... }
     ^~~~~~~~~~~~~~~
note: found this method for '@dynamicCallable'

func dynamicallyCall(withKeywordArguments: KeyValuePairs<String, Int>) -> Float { ... }
     ^~~~~~~~~~~~~~~
note: found this method for '@dynamicCallable'

Alternatives considered

  1. Make T2 equal T4 so that all dynamic calls return the same type.
    This would resolve the second concern, but won't resolve other concerns.

  2. @George suggested a value type DynamicArguments which has both subscript(_: String) and subscript(_: Int). This author believes that this would make interoperability less safe when we interoperate with a language that does not support keyword arguments. In the existing solution, implementing dynamicallyCall(withArguments:) without implementing dynamicallyCall(withKeywordArguments:) can guarantee that a call like x(a: b) gets rejected at compile-time. But DynamicArguments would not provide that guarantee. Moreover, the existing approach maintains better simplicity in that it doesn't introduce or require the user to internalize any new stdlib type.

8 Likes

I supported a simpler overload set in the original review and I support it now.

1 Like

I didn't see this in "Alternatives Considered", but apologize if it is mentioned elsewhere:

It may feel less foreign to provide a value type that is DynamicArguments. This value can have both subscript(name: String) and subscript(_ position: Int) (or, alternatively, analogous throwing methods).

This simplifies the selection of which method to call (there is only one) and leaves more room to improve the implementation of dynamicCallable in the future.

I wonder, do you intend to define DynamicArguments as a protocol? I don't think this is possible because it's not possible to specify an accurate/useful return type for the subscript methods.

// A "concrete" protocol doesn't seem to work.
// The requirements for a "DynamicArguments" type would have to be implicit and ad-hoc.
protocol DynamicArguments {
  // `Any` as a return type is not accurate. Unsafe/inefficient casting is required.
  subscript(name: String) -> ???
  subscript(_ position: String) -> ???
}

In any case, I think adding abstractions like DynamicArguments complicates the already ad-hoc type checking rules for @dynamicCallable, which is undesirable. Making things less ad-hoc would be great, if possible.

I see your point. A concrete protocol would use an API similar to what the Decoder protocol uses (ie. argument<T>(name: String) throws -> T)

Alternatively, you could use also use a similar strategy to Fix ExpressibleByStringInterpolation

Indeed, now that I think about it, representing a dynamic method call has a good amount of overlap with representing (for instance) a SQL statement with ExpressibleByStringInterpolation. Both are somewhat free-form lists of a (possibly constrained) set of types.

Python interoperability can then vend a concrete PythonDefaultArguments which only accepts arguments representable in Python.

Sorry, to expand on my previous point: I think it's good to try to keep @dynamicCallable as close to a syntactic sugar as possible, to make dynamic call behavior easy to understand. I would say DynamicArguments is quite harder to understand.

As I explain in this comment, the current @dynamicCallable design is already not an exact sugar, which leads to (yet unsolved) implementation challenges.

1 Like

Thanks. Added to alternatives. Please take a look!

I think there are three different stages here, and IMO they should be addressed by staging the implementation, landing patches incrementally - they don't need to be addressed by changing the proposal:

  1. Support dynamicCallable for types that implement exactly one of these methods. This should be straight-forward and fits with your current implementation approach.

  2. Extend the implementation to support implementation of both methods, but require they have the same result type.

  3. Extend the implementation to support multiple result types.

Each of these are clearly distinct and value-adding steps, and you can stop at any logical point given your implementation bandwidth. I don't think that downscoping the proposal itself is particularly useful, because we want at least #1 and #2. I agree that supporting #3 is more questionable.

4 Likes

I fully agree the implementation should take different stages, enabling features incrementally.

However, this pitch is not about down-scaling the proposal or motivated by the fact that implementation is hard. It's motivated by the fact that the semantics proposed in the original proposal can lead to confusion via overly-free customization points (explained in motivations #1 and #2 in this pitch).

In particular, I'm not sure the language behavior that this would enable is a useful or positive thing for Swift.

1 Like

Thanks for suggestions on staging the implementation! I agree that incremental work is good.
However, the blocker I encountered prevents step #1 from being implemented (visit the link for the full explanation):

I need some guidance on working around this blocker. Perhaps I should ask for help on a new thread.

Yeah, this proposal for amendment is separate from (though slightly motivated by) implementation. The main issue is potentially unexpected/unclear behaviors. It's better to review this before the implementation is finished so that we don't break code.

1 Like

I don't find the first two motivations that convincing, because neither seems surprising or confusing at all to me. The implementation difficulty is the most compelling part to me, particularly after reading the associated pull request discussion, but you say that is a minor argument.

Here's a different way to characterize the first two motivations: if a @dynamicCallable type defines both the withArguments and withKeywordArguments methods, how does overload resolution work?

Having read the proposal, the answer may seem obvious (do ambiguity resolution based on the most specific match). I'd argue it's not obvious though, for people who are encountering @dynamicCallable for the first time and are trying to piece things together.

A simple solution to the problem (as proposed) is to disallow withArguments and withKeywordArguments methods from both being defined. This is justified for the use case of dynamic language interoperability because languages either support keyword arguments or don't (and languages that support keyword arguments also tend to support non-keyword arguments, etc).

2 Likes

Sure, a lot of things about overload resolution in Swift generally might not be immediately obvious to people encountering them for the first (or hundredth) time. I don't find this case particularly complicated though, it seems a lot conceptually simpler than many of the other subtle overload ranking questions.

I suppose that's fair. Can you think of cases where defining both withArguments and withKeywordArguments is useful?

It's debatable whether features that may cause confusion and that have few use cases should be supported. "Generality" is a supporting argument. "Confusion" and "unusefulness" are counterarguments.

I haven't thought much about it, sorry, maybe you could track down some people who participated in the pitch/proposal threads who had concrete use cases for dynamic callable and mention them here. Besides generality, I presume there would be an efficiency argument for avoiding constructing a keyword “dictionary” if it isn't required. I don't know to what extent that is important or can be mitigated by the compiler, though.

This code example wouldn't work. If your keyword type is [String:Int], you can't have multiple values for the same key (""). Also, even if you could, dictionaries don't have a defined order. There would be no way to know if a or b is the first argument. For this code to work the keyword type would have to be KeyValuePairs<String,Int> or some other type that is ExpressibleByDictionaryLiteral.

As written, I think your code examples are actually an argument against what you're proposing. The type people will reach for to represent keywords is indeed a dictionary. And a dictionary doesn't support identical keys or ordering. So it can't support positional arguments. So you need another method.

Thank you for pointing this out. I actually meant to use DictionaryLiteral (or KeyValuePairs) but typed out a dictionary type because I copied [S : T3] from the original proposal and swapped the types (right, the original proposal also shouldn't have used this type in its example). I'm aware that it wouldn't work if the argument type was really a dictionary. I'll fix the code example.

As explained, this is my typo that will be fixed. In all my code examples I meant to use KeyValuePairs<String, Int>

2 Likes

I like this idea, and my impression is that there was no use case that would prove this generality to be useful in practice.

Yes, that would form an argument, but it's trivial especially when the efficiency of dynamic calls is not in question. The representational efficiency of [a, b, c] is not much lower than that of ["" : a, "" : b, "" : c]. Empty strings are cheap to represent, and both are just ordered lists that do not need much runtime complexity to form.

@rxwei The use case was for languages such as Lua, with:

  • unlabelled arguments by default, e.g. f(1, 2);

  • syntactic sugar for a table of labelled arguments, e.g. f{a=1, b=2}.

Lua Reference Manual: Function Calls

This could probably be implemented with only dynamicallyCall(withKeywordArguments:) for both calls.