Structural opaque result types

Evolution Pitches
,
1

EDIT [8/13/21]: Added sections 'Detailed Design - Constraint Inference' and 'Alternatives Considered - Constraint Inference'.

EDIT [8/13/21]: Clarified language in 'Detailed Design - Higher Order Functions' and 'Alternatives Considered - Higher Order Functions' thanks to some comments by @ensan-hcl.

Introduction

An opaque result type may be used as the result type of a function, the type of a variable, or the result type of a subscript. In all cases, the opaque result type must be the entire type. This proposal recommends lifting that restriction and allowing opaque result types in "structural" positions.

Motivation

The current restriction on opaque result types prevents them from being used in many common API patterns. Some examples are as follows:

// we cannot express a function that might fail to produce an opaque result type
func f0() -> (some P)? { /* ... */ }

// we cannot use an opaque result type as one of several return values
func f1() -> (some P, some Q) { /* ... */ }

// we cannot return a lazily computed opaque result type
func f2() -> () -> some P { /* ... */ }

// more generally, we cannot embed an opaque result type into a larger structure
func f3() -> S<some P> { /* ... */ }

Proposed Solution

We should allow opaque result types in structural positions in the result type of a function, the type of a variable, or the result type of a subscript.

Detailed Design

Syntax for Optionals

The some keyword binds more loosely than ? or !. An optional of an opaque result type must be written (some P)?, and an optional of an unwrapped opaque result type must be written (some P)!.

some P? gets interpreted as some Optional<P> and therefore produces an error because an opaque type must be constrained to Any, AnyObject, a protocol composition, and/or a base class. The analogous thing is true of some P!.

Higher Order Functions

If the result type of a function, the type of a variable, or the result type of a subscript is a function type, that function type can contain arbitrary structural opaque result types. For example, func f() -> () -> some P and func g() -> (some P) -> () are valid function definitions.

Constraint Inference

When a generic parameter type is used in a structural position in the signature of a function, the compiler implicitly constrains the generic parameter based on the context is which it is used. E.g.,

struct H<T: Hashable> { init(_ t: T) {} }
struct S<T>{ init(_ t: T) {} }

// same as 'f<T: Hashable>' because 'H<T>' implies 'T: Hashable'
func f<T>(_ t: T) -> H<T> {
    var h = Hasher()
    h.combine(t) // OK - we know 'T: Hashable'
    let _ = h.finalize()
    return H(0)
}

// 'S<T>' doesn't imply anything about 'T'
func g<T>(_ t: T) -> S<T> {
    var h = Hasher()
    h.combine(t) // ERROR - instance method 'combine' requires that 'T' conform to 'Hashable'
    let _ = h.finalize()
    return S(0)
}

Opaque result types do not feature such inference. E.g.,

// ERROR - type 'some P' does not conform to protocol 'Hashable'
func f<T>(_ t: T) -> H<some P> { /* ... */ }

Source Compatibility

This change is purely additive so has no source compatibility consequences.

As discussed in SE-0244:

If opaque result types are retroactively adopted in a library, it would initially break source compatibility [...] but could provide longer-term benefits for both source and ABI stability because fewer details would be exposed to clients. There are some mitigations for source compatibility, e.g., a longer deprecation cycle for the types or overloading the old signature (that returns the named types) with the new signature (that returns an opaque result type).

Effect on ABI Stability

This change is purely additive so has no ABI stability consequences.

As discussed in SE-0244:

[C]hanging an existing API to make use of opaque result types instead of returning concrete types would be an ABI-breaking change, so one of the source compatibility mitigations mentioned above would also need to be deployed to maintain ABI compatibility with existing binary clients.

Effect on API Resilience

This change is purely additive so has no API resilience consequences.

Rust's impl Trait

As discussed in SE-0244, Swift's opaque result types were inspired by impl Trait in Rust, which is described in RFC-1522 and extended in RFC-1951.

Though SE-0244 lists several differences between some and impl Trait, one difference it does not make explicit is that impl Trait is allowed in structural positions, in similar to the manner to that suggested by this proposal. One difference between this proposal and impl Trait is that impl Trait may not appear in the return type of closure traits or function pointers.

Alternatives Considered

Syntax for Optionals

This proposal recommends that an optional of an opaque result type with a conformance constraint to a protocol P be notated (some P)?. However, a user's first instinct might be to write some P?. This latter syntax is moderately less verbose, and is, in fact, unambiguous since Optional<P> is not a valid opaque result type constraint. It would be possible to add a special case that expands some P? into (some P)?. The analogous thing would be done with some P! and (some P)!.

However, this is inconsistent with other parts of the language, e.g. the interpretation of () -> P? as () -> Optional<P> or the fact that P & Q? is an invalid construction which is properly written as (P & Q)?. Adding special cases to the language can decrease its learnability.

Furthermore, since P? is never a correct constraint, it would be possible to (and in fact this proposal's implementation does) provide a "fix it" to the user which suggests that they change some P? to (some P)?.

Higher Order Functions

Consider the function func f() -> (some P) -> (). The closure value produced by calling f has type (some P) -> (), meaning it takes an opaque result type as an argument. That argument has some concrete type, T, determined by the body of the closure. Assuming no special structure on P, such as ExpressibleByIntegerLiteral, the user cannot call the closure. If they were able to, then they would be depending at the source level on the concrete type of T to remain fixed, which is one of the things opaque result types are designed to prevent.

We could try to stop the user from defining a higher order function which returns a function any of whose arguments are opaque result types. However, is already possible to define uncallable functions in Swift, for instance a function taking an uninhabited protocol composition, and, in fact, the returned closure might be callable, something it is non-trivial to determine in general.

Another reason one might want to disallow returning functions that take opaque result types as arguments is that top level functions can never have opaque result types as arguments.

This decision should be considered in the context of generalized some syntax, which we are likely to implement in the future. The approach to higher order functions which is more consistent with this generalized syntax is debatable.

Constraint Inference

We could infer additional constraints on opaque result types based on context, but this would likely be confusing for the user. Whereas the syntax for generic parameters draws the user's attention to the underlying type itself, i.e. the T in f<T>, opaque result type syntax draws the user's attention to an explicit list of the protocols which the underlying type satisfies, i.e. the P in some P. At least one constraining protocol must be specified, unlike with generic parameters. The closest thing to <T> one can write is some Any.

The decision about what to do in this case seems pretty clear, which is why this section was not included in the original version of this proposal. The main utility of this discussion is in teasing out why opaque result types should function differently than generic parameters because of the implications this has for named opaque result types, which will likely be proposed in the future.

Though this is outside the scope of this proposal, named opaque result types have a similar syntactic quality to generic parameters, and therefore should probably be subject to constraint inference in the result of a function, as well as the type of variable or the result type of a subscript.

Future Directions

This proposal is a natural stepping stone to fully generalized reverse generics, as demonstrated by the following code snippet from the generics UI design document:

func groupedValues<C: Collection>(in collection: C) -> <Output: Collection> (even: Output, odd: Output)
  where C.Element == Int, Output.Element == Int
{
  return (even: collection.lazy.filter { $0 % 2 == 0 },
          odd: collection.lazy.filter { $0 % 2 != 0 })
}
29 Likes
2

It's great!

But I doubt this point.

[Edit]
I've misread. This is only an example which closure with opaque result type argument can be called.

IIUC, your argument is that, there is no way to call g, and therefore should be explicitly disallowed. But it's not true. I previously posted about 'reverse generic argument types'. Consider following closure:

var closure: (some Numeric) -> () = { (value: Double) in print(value) }

some Numeric is actually Double, but it doesn't matter. We have information about some Numeric: it conforms to ExpressibleByIntegerLiteral and it has static property zero. Therefore, even though we cannot call this closure with Double, we can still use this closure like this:

closure(42)       // 42
closure(21)       // 21
closure(.zero)    // 0

Actually similar behavior has been in the current Swift already. You can try how it works:

protocol Printer {
    func print(value: Self)
}
extension Double: Printer {
    func print(value: Double) { Swift.print(value) }
}
let d: some (Numeric & Printer) = 42.0
let closure = d.print
// you can still call this closure!
closure(42.0)    // error (since type is opaque)
closure(42)      // OK (from ExpressibleByIntegerLiteral)
closure(.zero)   // OK (from AdditiveArithmetic)

Like this, having g doesn't cause any problems. You can still call the closure, and use it. There is no reason to explicitly disallow it.

7 Likes
3

No, I don't think it's being suggested these be banned – Ben's just pointing out that you could create function return types that can't be called. In your example, you are adding more information, and that then does allow it to be callable.

4 Likes
4

Oh, I'm sorry! I've misunderstood. Then I have nothing to object :laughing:

5

What Ben C said is exactly right, I just intended to point out that such a thing is possible, not advocate that we ban it.

I like your example however because it illustrates that even determining whether something is callable is non-trivial in the general case.

5 Likes
6

Thumbs up for raising the problem, but I think suggested syntax lacks generality - it is not possible to express dependencies between different parts of the type. For example:

  • A tuple of two values of the same opaque type
  • A tuple of the value and a “setter” function for that value
  • A dictionary where key and value are wrappers around some type

Earlier I was suggesting a syntax that solves this for existentials, I guess it can be adapted to opaque types as well:

func f4() -> some<T> (T, T)
func f5() -> some<T: Hashable> (T, (T) -> Void)
func f6() -> (some <T> Dictionary<KeyWrapper<T>, ValueWrapper<T>> where T: P>)

In the last example, type needs to be taken into parentheses, otherwise the where clause would belong to the entire function declaration

EDIT: Forgot to mention, in f5() example, you have function type with an opaque argument type which is nevertheless callable:

let (oldValue, oldSetter) = f5()
oldSetter(oldValue)
let (newValue, newSetter) = f5()
if newValue != oldValue {
    oldSetter(newValue)
}
3 Likes
7

Would this change require additional changes to the demangler and thereby preclude back-deployment again?

(I don’t think this should affect the evaluation of the pitch, but people tend to object when that sort of thing comes adds as a surprise.)

8

The initial implementation was done with an eye toward allowing for generalization to multiple opaque type variables in a declaration's return type, so there would hopefully not be back deployment constraints. It could be possible that a complete implementation ends up revealing bugs in older runtimes' handling nonetheless.

4 Likes
9

While you right that the anonymous structural opaque result types I am proposing are not general enough to address every use case on their own, they are intended as a stepping stoned to named opaque result types, as mentioned in the "future directions" section. It seems like named opaque result types are actually exactly what you're suggesting, just with a moderately different syntax from that described in the post written by Joe which I linked.

In my opinion, the syntax for named opaque result types is necessarily rather cumbersome, and the Rust ecosystem suggests that there is a large space of practical API designs where structural opaque result type are needed, but where a fully general syntax is not. I also think that implementing named opaque result types without anonymous structural opaque result types would feel like a strange hole in the language design.

Furthermore, this proposal creates a nice symmetry to generalized some syntax, another likely future direction, just as named opaque result types create a nice symmetry with generic parameter lists. You have a verbose form for when you need detailed constraints, and a friendly, terse form for the simple cases.

From a compiler implementation standpoint, the type checker work which I did to facilitate structural opaque result types will also serve to support named opaque result types.

13 Likes
10

It's just a question, not an objection, but how do you explain this behavior? Considering generalized some syntax in function argument position, the code would be as following.

// it takes **generic** argument and returns opaque result (reverse generic result) 
func double(value: some Numeric) -> some Numeric {
    return value * 2
}
// it takes **reverse generic** argument and returns reverse generic result
let double: (some Numeric) -> some Numeric = { (value: Int) in value * 2 }

I definitely agree that the behavior of closure is right, but I want to know how can we justify this confusing difference.

11

The proposal notes that function return types with some in a contravariant position are uncallable. Would it be possible for these to be treated as if they are ‘regular’ generics, not reverse generics? Eg

func f(a: some A) -> (some B) -> (some C)

would be equivalent to

func f<T1 : T, T2 : B>(a: T1) -> <T3 : C> (T2) -> (T3)

I think this would resolve the inconsistency you show above. Maybe there is some particular use for returning an uncallable function though?

3 Likes
12

For me it seems more confusing.
I think what you are saying is that some in the return position of a function can be interpreted differently in different cases.

[EDIT] It's wrong
~~Using `typealias`, it is obviously strange. You cannot determine how `some` works with type signature.~~ 
typealias Function<T, U> = (T) -> U
// here some A and some B are generic, and some C is reverse generic
func f(a: some A) -> Function<some B, some C>
func f<T1 : T, T2 : B>(a: T1) -> <T3 : C> (T2) -> (T3)

typealias Function2<U, T> = (T) -> U
// here some A and some C are generic, and some B is reverse generic
func f(a: some A) -> Function2<some B, some C>
func f<T1 : T, T3 : C>(a: T1) -> <T2 : B> (T3) -> (T2)

typealias Function3<T> = (T) -> T
// here some A is generic, but some B is ...?
func f(a: some A) -> Function3<some B>

I don't think it can work well.
[EDIT] add more confusing examples. It is really hard to determine which some is generic and which some is reverse generic. They should have only one interpretation, but what rule decide how to interpret these signatures? (of course it would be really rare to see such signatures, though)

func f(value: some A) -> ((some B) -> (some C)) -> (some D)
func f(value: some A) -> ((some B) -> (some C), (some D) -> (some E))
func f(closure: (some A) -> (some B)) -> (() -> (some C)) -> ((some D) -> ())

Maybe typealias can be workaround for this?

typealias T4<T> = (T, T)
func f4() -> T4<some Any>

typealias T5<T: Hashable> = (T, (T) -> Void)
func f5() -> T5<some Hashable>

typealias T6<T: P> = Dictionary<KeyWrapper<T>, ValueWrapper<T>>
func f6() -> T6<some P>
13

I agree with everything @ensan-hcl mentioned about how typealias would interact with this suggestion.

In terms of teaching the difference between func f(_ x: some P) -> some P and let x: (some P) -> some P, I think the rule governing this case is simple, if a bit strange at first glance. I expect it will be quite rare in practice for someone to try to use some in the arguments of a returned closure--especially a beginner--, so it is probably acceptable, if undesirable, that they may have to do some searching to figure out what is going on. The Swift docs should definitely address this distinction.

2 Likes
14

Firstly, I've been blocked by the lack of a feature like this in the past and will be very happy to see something along these lines come to fruition.

Since the eventual full reverse-generic syntax has been referenced here a number of times I wanted to reference this post A new idea about generics that I made earlier this summer. The reason I think it may be relevant for some who are interested is because the syntax I propose here is at odds with the hypothetical reverse-generic syntax that you use in the post @bdriscoll, which as far as I've seen is the most commonly referenced potential future-syntax for that feature. I know that this syntax extension that you're proposing here does not necessarily step on the toes of that bigger decision (about the design of the full reverse-generics syntax), but maybe it does, so I thought that this discussion that we had at the post linked above might be of interest to someone.

15

Does this pitch enable (some P).Type?

func foo() -> (some Numeric).Type {
    return Int.self
}
let someType = foo()
print(someType.zero) // 0
5 Likes
16

In lieu of this getting added, I'm curious to know how people are working around it right now. Similar to @jeremyabannister I'm also blocked on a project I'm working on due to the opaque types limited usage. Or at least, my code is getting incredibly horribly ugly have to work around this constraint. Is everyone just falling back to Generics?