SE-0216 @dynamicCallable implementation blockers

[dan-zheng]

Hi all,

I ran into some blockers implementing SE-0216 for user-defined dynamic "callable" types and need help.

• Sorry for the very long post, click here to jump to a description of the blockers. The key points are in bold.
• Here is the WIP PR.

Introduction

In this proposal, types marked with @dynamicCallable must implement one of the following methods:

func dynamicallyCall(withArguments: [T1]) -> T2
func dynamicallyCall(withKeywordArguments: T3) -> T4
// `T1` can be any type that conforms to `ExpressibleByArrayLiteral`.
// `T3` can be any type that conforms to `ExpressibleByDictionaryLiteral` where
// `T3.Key` conforms to `ExpressibleByStringLiteral`.

Such types then become callable:

@dynamicCallable
struct Callable {
  func dynamicallyCall(withArguments: [Int]) -> Int
  func dynamicallyCall(withKeywordArguments: KeyValuePairs<String, Int>) -> Int
}

func foo(x: Callable) {
  x(1, 2, 3) // sugar for `x.dynamicallyCall(withArguments: [1, 2, 3])`
  x(a: 1, 2, b: 3) // sugar for `x.dynamicallyCall(withKeywordArguments: ["a": 1, "": 2, "b": 3])`
}

As proposed, @dynamicCallable can be applied to any nominal type: structs, classes, enums, and protocols.
I believe @dynamicCallable method resolution should behave just like protocol requirement resolution: it should be possible to have overloaded dynamicallyCall methods and dynamic calls should behave exactly like direct calls (e.g. x(a: 1, 2) should resolve in the exact same way as x.dynamicallyCall(withKeywordArguments: ["a": 1, "": 2]).

Implementation approach

Originally, I tried to implement @dynamicCallable using a lookup-based approach outside of the constraint system. This was problematic for many reasons, the biggest one was that overloaded method resolution didn't work well.

I've now moved on to a constraint system based approach for resolving dynamic calls. The general algorithm is this:

1. In CSSimplify.cpp, check if the "called" nominal type has or somehow "inherits" the @dynamicCallable attribute. Then, look up and cache all dynamicallyCall(withArguments:) and dynamicallyCall(withKeywordArguments:) methods within the nominal type.
2. Then, attempt to resolve the correct method by creating a disjunction constraint over all candidate methods (this is done via CS.addOverloadSet(tv, candidates, ...) where tv is a type variable). I then add various constraints on tv.

Let's work through an example program:

@dynamicCallable
struct Callable {
  func dynamicallyCall(withArguments: [Int]) {}
  func dynamicallyCall(withArguments: [Float]) {}
}
func test(x: Callable) {
  x(1, 2) // We're handling this in `simplifyApplicableFnConstraint` in CSSimplify now.
}

Here are the type variables:

Type Variables:
  # The constraint system created the first 5 type variables.
  $T0 [lvalue allowed] as Callable @ locator@0x7ff6ac037800 [DeclRef@test.swift:7:3]
  $T1 fully_bound literal=3 involves_type_vars bindings={(subtypes of) (default from ExpressibleByIntegerLiteral) Int} @ locator@0x7ff6ac037880 [IntegerLiteral@test.swift:7:5]
  $T2 fully_bound literal=3 involves_type_vars bindings={(subtypes of) (default from ExpressibleByIntegerLiteral) Int} @ locator@0x7ff6ac037920 [IntegerLiteral@test.swift:7:8]
  $T3 fully_bound subtype_of_existential involves_type_vars bindings={} @ locator@0x7ff6ac037a08 [Call@test.swift:7:3 -> function result]
  $T4 [lvalue allowed] subtype_of_existential involves_type_vars bindings={} @ locator@0x7ff6ac037aa0 [Call@test.swift:7:3 -> apply function]

  # I create $T5 to represent the type of the argument to the
  # `dynamicallyCall(withArguments:)` method.
  $T5 fully_bound subtype_of_existential involves_type_vars bindings={} @ locator@0x7ff6ac037aa0 [Call@test.swift:7:3 -> apply function] # `apply function` is probably wrong here

And here are the constraints:

Inactive Constraints:
  # These first two constraints were generated by the constraint system.
  $T1 literal conforms to ExpressibleByIntegerLiteral [[locator@0x7ff6ac037880 [IntegerLiteral@test.swift:7:5]]];
  $T2 literal conforms to ExpressibleByIntegerLiteral [[locator@0x7ff6ac037920 [IntegerLiteral@test.swift:7:8]]];

  # The rest below are mine.
  disjunction [[locator@0x7ff6ac037aa0 [Call@test.swift:7:3 -> apply function]]]:$T4 bound to decl test.(file).Callable.dynamicallyCall(withArguments:)@test.swift:3:8 : (Callable) -> ([Int]) -> () at test.swift:3:8 [[locator@0x7ff6ac037aa0 [Call@test.swift:7:3 -> apply function]]]; or $T4 bound to decl test.(file).Callable.dynamicallyCall(withArguments:)@test.swift:4:8 : (Callable) -> ([Float]) -> () at test.swift:4:8 [[locator@0x7ff6ac037aa0 [Call@test.swift:7:3 -> apply function]]];
  $T5 conforms to ExpressibleByArrayLiteral [[locator@0x7ff6ac037aa0 [Call@test.swift:7:3 -> apply function]]];
  $T5.ArrayLiteralElement can default to Any [[locator@0x7ff6ac037aa0 [Call@test.swift:7:3 -> apply function]]];
  $T1 arg conv $T5.ArrayLiteralElement [[locator@0x7ff6ac037d48 [Call@test.swift:7:3 -> apply function -> tuple element #0]]];
  $T2 arg conv $T5.ArrayLiteralElement [[locator@0x7ff6ac037dd8 [Call@test.swift:7:3 -> apply function -> tuple element #1]]];
  ($T5) -> $T3 applicable fn $T4 [[locator@0x7ff6ac037aa0 [Call@test.swift:7:3 -> apply function]]];
Resolved overloads:
  selected overload set choice x: $T0 == Callable

The disjunction constraint binds $T4 to each of the candidate dynamicallyCall methods.
I create $T5 representing the type of the argument to $T5. I add conversion constraints from $T1 and $T2 to $T5.ArrayLiteralElement.
I add an applicable function constraint ($T5) -> $T3 applicable fn $T4.

These constraints seem fine to me, please reply if anything looks wrong at this point.

3. The hard work is done. If the disjunction constraint is unsolved, a diagnostic will naturally be emitted. If it is solved, access the resolved dynamicallyCall method in CSApply.cpp and do expression rewriting.

Implementation blockers

Dynamic calls aren't an exact sugar

A tricky detail about dynamic calls are that they're not an exact syntactic sugar for direct calls to dynamicallyCall.

Dynamic calls like x(1, 2) have multiple arguments in the AST (and thus multiple type variables for each argument) while direct calls only have one array-like argument: x.dynamicallyCall(withArguments: [1, 2]).

This causes simplification of the ($T5) -> $T3 applicable fn $T4 constraint to fail, because it's expected that the number of arguments is consistent:

Assertion failed: (params.size() == labels.size()), function relabelParams, file /Users/dan/swift-build/swift/lib/AST/ASTContext.cpp, line 3623.
Stack dump:
...
3.	While type-checking expression at [test.swift:7:3 - line:7:9] RangeText="x(1, 2"
...
swift::AnyFunctionType::relabelParams(llvm::MutableArrayRef<swift::AnyFunctionType::Param>, llvm::ArrayRef<swift::Identifier>) + 119
swift::constraints::matchCallArguments(swift::constraints::ConstraintSystem&, bool, llvm::ArrayRef<swift::AnyFunctionType::Param>, llvm::ArrayRef<swift::AnyFunctionType::Param>, swift::constraints::ConstraintLocatorBuilder) + 1616
swift::constraints::ConstraintSystem::simplifyApplicableFnConstraint(swift::Type, swift::Type, swift::OptionSet<swift::constraints::ConstraintSystem::TypeMatchFlags, unsigned int>, swift::constraints::ConstraintLocatorBuilder) + 1574
swift::constraints::ConstraintSystem::simplifyConstraint(swift::constraints::Constraint const&) + 1150
swift::constraints::ConstraintSystem::simplify(bool) + 210
swift::constraints::ConstraintSystem::solveRec(llvm::SmallVectorImpl<swift::constraints::Solution>&) + 83
swift::constraints::DisjunctionChoice::solve(llvm::SmallVectorImpl<swift::constraints::Solution>&) + 113
swift::constraints::ConstraintSystem::solveForDisjunctionChoices(swift::constraints::Disjunction&, llvm::SmallVectorImpl<swift::constraints::Solution>&) + 1338

I need help solving this problem.

An idea that probably won't work is adding a flag to matchCallArguments to specifically handle @dynamicCallable calls. However, I see no way to actually set that flag because matchCallArguments is not being invoked directly. This is super ad-hoc anyways.

An idea that may work is adding a new DynamicCallableApplicableFn constraint, and using that instead: ($T5) -> $T3 dynamic callable applicable fn $T4. With a custom constraint, I can do whatever simplification logic I want, including calling matchCallArguments with a flag. Is it reasonable to add a new constraint kind just for @dynamicCallable? One unknown is how to detect whether the argument type ($T5 here) has a constraint conforming it to ExpressibleByArrayLiteral or ExpressibleByDictionaryLiteral. If detection isn't possible, then two variants of the DynamicCallableApplicableFn constraint may be necessary.

Label-less dynamic calls can desugar to either withArguments or withKeywordArguments method calls

Consider the following program:

@dynamicCallable
struct S {
  func dynamicallyCall(withArguments: [String]) -> Int { return 1 }
  func dynamicallyCall(withKeywordArguments: [String : String]) -> Float { return 3.14 }
}

let s = S()
let _: Float = s("hi")
// Should resolve to `s.dynamicallyCall(withKeywordArguments: ["": "hi"]

With the current proposal, the above program is valid. This means, when resolving label-less dynamic calls, I need to create disjunction constraints over both withArguments and withKeywordArguments methods.

However, the set of constraints associated with the withArguments disjunction constraint and withKeywordArgument disjunction constraint are different. I don't see how to reconcile this. Conceptually, I need something like a union constraint: (disjunctionOverArgsMethods($T1) /\ isValidArgsMethod($T1)) \/ (disjunctionOverKwargsMethods($T1) /\ isValidKwargsMethod($T1)). I'm not sure it's possible to express this using constraints.

However, this amendment to SE-0216 makes the problem go away. If @dynamicCallable types can only define either the withArguments method or the withKeywordArguments one, then I never need to check both sets of constraints. I strongly support the amendment (rationale here) and it would simplify the implementation effort if accepted.

[dan-zheng]

Proactively reaching out for help: @rudkx @Douglas_Gregor @Slava_Pestov

Any ideas or advice would be greatly appreciated.

[rudkx (Mark Lacey)]

Hi Dan,

It has been difficult to find time to really dig into this and provide useful advice, but here are some thoughts.

What I think you want to end up generating for the constraint system for an example like this:

@dynamicallyCallable
struct Callable {
  func dynamicallyCall(withArguments: [Int]) {}
  func dynamicallyCall(withKeywordArguments: [String : Int]) {}
}

let c = Callable()
c(1, 2, 3)

is a single ApplicableFunction constraint with a single argument of type Array<$T1> that resolves to the withArguments overload of dynamicallyCall. However the example you gave with:

let _: Float = s("hi")
// Should resolve to `s.dynamicallyCall(withKeywordArguments: ["": "hi"]

obviously conflicts with that idea, so I'm wondering you really intended to say that this is the overload that it would resolve to?

More generally (and sorry I haven't had a chance to read the SE proposal in detail again), if you're allowing multiple overloads of withArguments: with different types then you would generate a disjunction of those overloads and allow that to be resolved through the constraint solver attempting options in the disjunction.

There is another interesting wrinkle here that I don't think has been considered.

In Swift we can have a property and method with the same name:

@dynamicallyCallable
struct Callable {
  func dynamicallyCall(withArguments: [Int]) {}
}

struct S {
  func c(a: Int, b: Int) {}
  var c: Callable = Callable()
}

let s = S()
s.c(1,2) // are we calling the method in S or the dynamicallyCall method in Callable?

I think this could end up being really problematic to deal with, so you may need to explicitly disallow it (meaning having a property of a @dynamicallyCallable type with the same name as a method) to get this proposal implemented.

Let me know if you think what I've written thus far in regards to what the constraint system should look like at a high level seems reasonable.

We can then consider how we might be able to get to that point with the WIP implementation you have started.

[dan-zheng]

Hi Mark, thanks for the reply! Sorry for the late response. I have a specific question for you in bold, near the bottom of this post.

@rudkx

However the example you gave with:

let _: Float = s("hi")
// Should resolve to `s.dynamicallyCall(withKeywordArguments: ["": "hi"]

obviously conflicts with that idea, so I'm wondering you really intended to say that this is the overload that it would resolve to?

The example here is an advanced case, and with this amendment we don't have to handle it. I would like to solve the simple case first, which is illustrated in your example below and in my first example.

@rudkx

What I think you want to end up generating for the constraint system for an example like this:

@dynamicallyCallable
struct Callable {
  func dynamicallyCall(withArguments: [Int]) {}
  func dynamicallyCall(withKeywordArguments: [String : Int]) {}
}

let c = Callable()
c(1, 2, 3)

is a single ApplicableFunction constraint with a single argument of type Array<$T1> that resolves to the withArguments overload of dynamicallyCall .

This is the "simple" case. I do generate the constraints you described, actually. In the example I shared above, $T4 binds to the withArguments method overloads, $T5 is the type of the single argument conforming to ExpressibleByArrayLiteral, and a ($T5) -> $T3 applicable fn $T4 constraint is generated.

The problem is the crash in matchCallArguments above. I believe normally, each type variable corresponds to an expression in the AST, so the "shape" of the ApplicableFn constraint matches the shape of the expression/type variables (e.g. foo(1, [2, 3]) produces ($T4, $T5) -> $T6 applicable fn $T1).

However, dynamic calls break this precedent. The type variables in the CallExpr look like ($T4, $T5, ..., $Tn) -> $Result but the constraint always looks like ($ExpressibleByArrayLiteral) -> $Result. I think this is the reason for the matchCallArguments crash.

I'm not sure about how to best fix this crash: my best idea now to add a new "DynamicCallableApplicableFn" constraint kind whose simplification logic looks similar to ApplicableFn except it calls a modified version of matchCallArguments. This seems like it'll lead to a lot of code dupe though - please let me know if this approach is acceptable or if you have other ideas.

@rudkx

In Swift we can have a property and method with the same name:

...

I think this could end up being really problematic to deal with, so you may need to explicitly disallow it (meaning having a property of a @dynamicallyCallable type with the same name as a method) to get this proposal implemented.

Thank you for pointing out this tricky case. Explicitly disallowing it for now makes sense, I'll certainly add a test case like the example you provided to exercise that logic.

[rudkx (Mark Lacey)]

@dan-zheng

I'm not sure about how to best fix this crash: my best idea now to add a new "DynamicCallableApplicableFn" constraint kind whose simplification logic looks similar to ApplicableFn except it calls a modified version of matchCallArguments . This seems like it'll lead to a lot of code dupe though - please let me know if this approach is acceptable or if you have other ideas.

I am not sure that this is actually workable, since you won't always know that you have something that is dynamically callable when you go to generate the function application constraint.

For example, for:

@dynamicallyCallable
struct Callable {
  func dynamicallyCall(withArguments: [Int]) {}
  func dynamicallyCall(withKeywordArguments: [String : Int]) {}
}

class C {
  let c = Callable()
}

func foo() -> C { return C() }
func foo() -> Int { return 1 }

foo().c(1, 2, 3)

The initial constraints we generate include:

  disjunction [[locator@0x10b8fc218 [OverloadedDeclRef@/tmp/dyncall.swift:14:1]]]:$T0 bound to decl dyncall.(file).foo()@/tmp/dyncall.swift:11:6 : () -> C at /tmp/dyncall.swift:11:6 [[locator@0x10b8fc218 [OverloadedDeclRef@/tmp/dyncall.swift:14:1]]]; or $T0 bound to decl dyncall.(file).foo()@/tmp/dyncall.swift:12:6 : () -> Int at /tmp/dyncall.swift:12:6 [[locator@0x10b8fc218 [OverloadedDeclRef@/tmp/dyncall.swift:14:1]]];
  () -> $T1 applicable fn $T0 [[locator@0x10b8fc400 [Call@/tmp/dyncall.swift:14:1 -> apply function]]];
  $T1[.c: value] == $T2 [[locator@0x10b8fc488 [UnresolvedDot@/tmp/dyncall.swift:14:7 -> member]]];
  ($T3, $T4, $T5) -> $T6 applicable fn $T2 [[locator@0x10b8fc828 [Call@/tmp/dyncall.swift:14:7 -> apply function]]];

So when we go to generate the ApplicableFunction constraint, we don't even know what the type of c is to know that it is dynamically callable.

[dan-zheng]

Thanks for bringing up that point.

I tested your snippet on my current @dynamicCallable development branch and actually foo() in foo().c(1, 2, 3) resolves to Callable correctly, probably because foo().c is referenced and Int doesn't define a c member.

I also tested the following code:

@dynamicCallable
struct Callable {
  func dynamicallyCall(withArguments: [Int]) {}
}

class C {
  let callable = Callable()
}

class D {
  func callable(_ x: Int, _ y: Int, _ z: Int) {}
}

func foo() -> D { return D() }
func foo() -> C { return C() }

// What does `foo()` resolve to?
foo().callable(1, 2, 3)

This seems like a trickier case because the results of both foo calls defines a callable property.

However, for both of these snippets, the constraint solver explores a disjunction choice where foo returns C:

Type Variables:
  $T0 [lvalue allowed] as () -> C @ locator@0x7fd7fc84de18 [OverloadedDeclRef@overload2.swift:17:1]
  $T1 as C @ locator@0x7fd7fc84df98 [Call@overload2.swift:17:1 -> function result]
  $T2 [lvalue allowed] as Callable @ locator@0x7fd7fc84e088 [UnresolvedDot@overload2.swift:17:7 -> member]
  $T3 fully_bound literal=3 involves_type_vars bindings={(subtypes of) (default from ExpressibleByIntegerLiteral) Int} @ locator@0x7fd7fc84e138 [IntegerLiteral@overload2.swift:17:16]
  $T4 fully_bound literal=3 involves_type_vars bindings={(subtypes of) (default from ExpressibleByIntegerLiteral) Int} @ locator@0x7fd7fc84e1d8 [IntegerLiteral@overload2.swift:17:19]
  $T5 fully_bound literal=3 involves_type_vars bindings={(subtypes of) (default from ExpressibleByIntegerLiteral) Int} @ locator@0x7fd7fc84e278 [IntegerLiteral@overload2.swift:17:22]
  $T6 fully_bound subtype_of_existential involves_type_vars bindings={} @ locator@0x7fd7fc84e378 [Call@overload2.swift:17:7 -> function result]
  $T7 [lvalue allowed] as ([Int]) -> () @ locator@0x7fd7fc84e428 [Call@overload2.swift:17:7 -> apply function]

Active Constraints:
  $T3 literal conforms to ExpressibleByIntegerLiteral [[locator@0x7fd7fc84e138 [IntegerLiteral@overload2.swift:17:16]]];
  $T4 literal conforms to ExpressibleByIntegerLiteral [[locator@0x7fd7fc84e1d8 [IntegerLiteral@overload2.swift:17:19]]];
  $T5 literal conforms to ExpressibleByIntegerLiteral [[locator@0x7fd7fc84e278 [IntegerLiteral@overload2.swift:17:22]]];

Inactive Constraints:
Resolved overloads:
  selected overload set choice Callable.dynamicallyCall: $T7 == ([Int]) -> ()
  selected overload set choice C.callable: $T2 == Callable
  selected overload set choice foo: $T0 == () -> C

And for both snippets, the same crash is the ultimate problem:

Assertion failed: (params.size() == labels.size()), function relabelParams, file /Users/dan/swift-build/swift/lib/AST/ASTContext.cpp, line 3623.

This leads me to believe generating a DynamicCallableApplicableFn constraint may still work. At the time such a constraint is generated, the original OverloadedDeclRef disjunction is gone (because a branch of it is being explored where there is indeed a @dynamicCallable application).

Please let me know if this doesn't make sense. I'm a bit flooded lately but I'll try to implement a DynamicCallableApplicableFn constraint ASAP to see if it fixes the matchCallArguments crash.