Opaque result types

(David Hart) #46

Add an extra reason: it could greatly improve error messages which would use the typealias name instead of "the result type of makeMeACollection".

(Michael Ilseman) #47

This looks really neat!

What are the library evolution implications here?

E.g., the actual type can change in a new library version without causing an ABI break, just like un-annotated structs.

  • public func foo() -> opaque Collection can change the return type version-to-version without breaking binary compatibility.

However, does adding @inlinable force the concrete return type to be ABI? I.e. is it possible to have the choice of keeping the return type opaque at compile time for inlinable functions, leaving the following options:

  1. @inlinable public func foo() -> opaque Collection
  2. @inlinable public func bar() -> @fixedLayout opaque Collection

But, I don't see how it's possible to support #1 and enforce same-type at runtime, unless inits/calls are automagically virtualized requiring any new version's return type to also supports those inits/calls.

This seems like quite a bit of complexity, and I suppose a library author can always declare a resilient struct type to return for this purpose. However, the same-type runtime constraint would mean that @inlinable requires both semantic equivalence in new versions and the same return type, which may not be obvious.

Have you considered ways to forward an opaque return type from a helper function call, or otherwise constrain multiple function's opaque return types to be the same at runtime? Fine-tuning a library's ABI often involves hand-outlining just the right parts, and without this the library author reverts to concrete return types for all outlined helper functions. This isn't a big deal, just wondering if this was considered.

(Paul Cantrell) #48

This answers some of my immediate questions. My first reaction was that it’s putting the cart before the horse to create a specially constrained version of generalized existentials for optimization purposes before generalized existentials exist in their more general form. The rationale makes a bit more sense now.

I do wonder about the opaque keyword, and the mental model it implies viewed from the far (hopefully not too far) future perspective where we have generalized existentials, and this might look less like a separate feature and more like a special constraint.

My own use cases for generalized existentials have all been either:

  1. heterogenous collections (i.e. some Foo<T> grouped as [Foo] where the T varies between elements), or
  2. less commonly, methods that need to accept an input whose type params are unknown to the caller (i.e. process(foo: Foo); usually comes up because of #1).

This proposal doesn’t address either of those concerns, correct?

3 Likes
(Joe Groff) #49

If it weren't too late to break source compatibility, I would wager that opaque concrete types are the more commonly useful feature than existentials, and in hindsight, maybe we should have saved the protocol-in-type-position syntax for an opaque type rather than using that syntax for existentials, and decorated existentials in some way instead. Rust is making that pivot even despite having source compatibility.

8 Likes
Associated types constrained by `AnyObject` can't be matched?
(Ian Partridge) #50

This occurred to me too. Then maybe an existential could be any Foo which reads better to me.

2 Likes
(Dave Abrahams) #51

As @Douglas_Gregor already knows, I'm a big fan of this feature: LGTM!

A few minor points:

I'm a little concerned about overloading “_” in the way proposed; everywhere else it means a kind of nothingness (no label, no pattern constraint, no name) but here it's being used to refer to a specific thing. It's hard to see that on a continuum with its other uses.

I'm also concerned about the arguable overloading of “->” and lack of symmetry in this example:

To me something like this would be more logical:

extension BidirectionalCollection {
  public func reversed()
      -> opaque BidirectionalCollection where _.Element == Element,
      -> opaque RandomAccessCollection where Self : RandomAccessCollection,
      -> opaque MutableCollection where Self : MutableCollection 
  {
    return ReversedCollection<Self>(...)
  }
}

I know what you mean, but I don't think subscript has an “element type” does it? Can we just say “return type” since a subscript always has a getter? The next sentence doesn't sound like a “however” to me; it's just a restatement, so “however” is confusing.

It's not clear what you mean here. If I read it literally it seems to be saying that this proposal introduces contextual named types to the language by allowing them in opaque result type constraints. If so, IMO you should say so more directly. If not, maybe you could clarify.

Is it possible to work around that with something like this as long as you were willing to return a nominal type instead of a tuple?

protocol SameElement {
    associatedtype C0 : Collection
    associatedtype C1 : Collection where C1.Element == C0.Element
    var c0: C0 { get }
    var c1: C1 { get }
}
func g() -> opaque SameElement { }
1 Like
(Brent Royal-Gordon) #52

I wonder if we might be better off thinking of this feature as an annotation on a type declaration, not on a return type. That is, maybe we want LazyCompactMapCollection to be opaque in all public uses, not to have its use in compactMap(_:) be opaque.

3 Likes
#53

Excuse me for thinking aloud, if I may…

The proposal introduces a new keyword. It has often been mentioned that scattering the language up with too many keywords is not a good idea. What if we introduce something like a placeholder type, which from the outside would more look like a 'concrete' type.

func foo() -> opaque Collection where _.Element: Hashable {}

could become something like

func foo() -> Opaque<Collection where _.Element: Hashable> {}

I think this would also immediately solve the question raised upthread whether opaque P & opaque Q could be allowed. No.

Thinking further. Would Opaque<> or Qpaque<_> then be allowed? Basically it would mean nothing more than Any, any type without any constraints. If so, why not call the opaque placeholder type Any in the first place?

The above could become

func foo() -> Any<Collection where _.Element: Hashable> {}

There was concern about using the underscore to refering to the type itself, instead of the omission of something. This probably could be solved with what we're doing elswhere in a generic context:

func foo() -> Any<T: Collection where T.Element: Hashable> {}

or

func foo() -> Any<This: Collection where This.Element: Hashable> {}

Lots of room for bikeshedding left… ;-)

I could imagine that this approach could also simplify the question what type(of:) should give back.

Excuse me for an amateur thinking aloud. I hope I'm not completely off the track…

3 Likes
(David Hart) #54

This syntax worries me because the where can be very confusing: the first one prefixes the list of truths about the opaque type, whereas the following ones prefix conditions.

1 Like
#55

Perhaps "$" would be a better fit. We use $0, $1 and so forth as a placeholder/ substitution for values passed in into closures. Why not reusing it as a placeholder sign in this case?

func foo() -> opaque BidirectionalCollection where _.Element == Element

would be

func foo() -> opaque BidirectionalCollection where $.Element == Element
2 Likes
(Adrian Zubarev) #56

I think you forgot the zero there, should be $0.Element no?

I'd like to extend that idea a little because I kind of like it. I still think that personally I'd prefer normal where clauses, that do not require any knowledge about the parent type where the returned opaque type is declared, that it should be moved into a typealias (especially when used on constants and variables).

Accumulated some syntax forms from this thread:

// Generalized `$0` shorthand syntax
public opaque typealias ReversedCollection<T> : BidirectionalCollection where T : BidirectionalCollection

extension BidirectionalCollection {
  public func reversed() -> ReversedCollection
    where 
    $0.Element == Element,
    // The following constraints are crystal clear conditional conformances
    $0 : RandomAccessCollection where Self : RandomAccessCollection,
    $0 : MutableCollection where Self : MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

// Or allowing extension on opaque types which would extend the concrete
// type hidden by the opaque type
extension opaque Collection ... {
  ...
}
extension ReversedCollection ... {
  ...
}

The generalized $ shorthand syntax could potentially solve some issues where we want to return multiple opaque types at once.

// Only the elements of the second opaque collection required to conform to `Equatable`.
// Without further constraints the elements of both collections are considered different.
func g() -> (opaque Collection, opaque Collection) where $1.Element : Equatable {
  ...
}
(Howard Lovatt) #57

At WWDC Tim Cook said "We created Swift to make it easy — to make it so easy to learn to code that is easy as our products are to use." I think this message is totally lost with a rash of proposals that require to say the least 'extreme' annotations.

This opaque protocols proposal is unfortunately another example of this trend.

There is a much simpler alternative make protocols generic, then we wouldn't need associated types, existentials, generalised existentials, nor opaque protocols. A lot simpler.

There is also an 'elephant in the room' (for none english natives this phrase means that there is something no one is saying); competition from other languages like Kotlin and Python. These language are gaining traction on both mobile (Kotlin) and server side (both). Swift has to compete with these if it wants world-domination and they are both a lot easier to use already without adding yet more annotation.

Sorry to be downer.

4 Likes
(Douglas Gregor) #58

Correct.

Right. This is described in the updated version of the rendered document. @inlinable exposes the function body, which includes the concrete type that underlying the opaque result type. Once you've exposed that, you can't change it resiliently.

It's been discussed a little bit here as an "opaque typealias". I think that would allow for fine-tuning.

Doug

1 Like
(Douglas Gregor) #59

Yeah, I agree that the use of _ here has the wrong connotations. Offline, I've had the suggestion to use opaque, since it's the opaque type we're talking about.

Ah, interesting! I like this better than what's in the proposal now.

Yes, this works, but if you end up having to invent a protocol for this... it doesn't seem like it's all that much better than defining a new resilient struct or two.

(I'll try to clarify the proposal w.r.t. your other points)

Doug

(Chéyo Jiménez) #60

I love this.
Some concerns:

I agree with others that this should be some form of typealias or declaration.

I think a better name for this features is something along the lines of proxy for a type.

I don’t think opaque is a good name user facing name because as far as I can tell there is a lot the compiler will know about this type.

This feature reminds me of unknown from typescript whixh is like Swift’s any.

(Douglas Gregor) #61

I'd state this a bit differently. I think we shouldn't be afraid to introduce a keyword when we are introducing a new kind of "thing". But we should be wary of introducing any new "thing", especially if it's very similar to something that already exists in the language.

I think your Opaque<...> and Any<...> suggestions fit nicely together. I have two concerns. First, the Any<...> was discussed extensively as part of SE-0095 a while back, and at this point I don't think we should go back and change it. Second, Opaque<...> wouldn't be a general-purpose type you could use anywhere: it is syntactically limited to result types, which makes opaque feel (to me) more like a modifier on the result type than a full-fledged type in the type system.

type(of:) returns the underlying concrete type, at run time.

Doug

2 Likes
(Douglas Gregor) #62

I've been thinking about this for a bit, and the second bullet is really what gets to me. That syntax I conjured up to describe conditional conformance is rather awful. It's completely non obvious and quite verbose. @dabrahams improved on it a bit, but it's verbose by necessity.

I think the right course forward for this proposal is to introduce opaque type aliases, which give a name to the opaque type so that it can be reused/tweaked. That provides a natural way to describe the conditional conformance, so we can eliminate the awful conditional-conformance-in-the-result-type syntax from the proposal.

Doug

5 Likes
(Douglas Gregor) #63

I'll ignore all of off-topic stuff from your post, but I want to respond to this part here. Generic protocols can replace associated types (some things would get worse, some things would get better), don't change much about existentials (the same problems of type identity remain), or generalized existentials (some trivial cases get simpler; the general case remains about the same), and don't solve the problems opaque result types solve (which are primarily about type identity).

I've seen this same line about generic protocols being "all we need" to simplify the language a number of times, enough that I think a significant number of people believe it despite the complete lack of a testable design that would achieve the stated wins. A handful of small examples doesn't cut it---we're talking about something that could radically reshape the way we work with generics, as well as the standard library. If you want to talk design of generic protocols, that's great... for another thread, but please refrain from claiming that "feature X will fix everything" unless you're prepared to provide sufficient detail to evaluate said claim.

Doug

19 Likes
(Anthony Latsis) #64

@Douglas_Gregor Do I understand correctly that an opaque type can be expressed as follows?

func foo<Opaque: Collection>() -> Opaque where Opaque.Element == Int {...}

If so, do we really need a new modifier and additional specific underscore syntax to refer to ATs?

1 Like
(Brent Royal-Gordon) #65

The signature you list would allow the caller to choose a type for Opaque, and it could be any Collection where Element == Int. This feature has the callee choose one type; it’s just that its choice is hidden.

1 Like