Opaque result types

You may want to remove the semicolon in the following example from your proposal:

func f2(i: Int) -> opaque P {   // okay: both returns produce Int
  if i > 10 { return i }
  return 0; // <-- here 
}

In the following example there is a missing }:

func f7(_ i: Int) -> opaque P {
  if i == 0 {
    return f7(1)                 // okay: returning the opaque result type of f(), similar to f5()
  /* here */ else if i < 0 {
    let result: Int = f7(-i)     // error: opaque result type of f() is not convertible to Int
    return result
  } else {
    return 0
  }
}

@Douglas_Gregor I understand that the top recursion is invalid because we want to store the opaque result into a constant and by the rules of opaque types the static type system does not know the concrete type at that point, even if it's inside the same function from where we'll return the concrete type. However I wonder if the following would be valid.

func f7(_ i: Int) -> opaque P {
  if i == 0 {
    return f7(1)
  } else if i < 0 {
    return f7(-i) as Int // Is this valid or not?
  } else {
    return 0
  }
}

Are there , missing between these opaque clauses?

extension BidirectionalCollection {
  public func reversed() 
      -> opaque BidirectionalCollection where _.Element == Element
         opaque RandomAccessCollection where Self: RandomAccessCollection
         opaque MutableCollection where Self: MutableCollection {
    return ReversedCollection<Self>(...)
  }
}

Have you forgot to update the grammar? Why is there a -> after the where clause?

opaque-conditional-requirement ::= where-clause '->' requirement-list

I'm not sure I follow this reasoning:

public typealias Reversed<Base : BidirectionalCollection> : opaque BidirectionalCollection 
  where _.Element == Base.Element = ReversedCollection<Base>

Then, we can describe conditional conformances on Reversed :

extension Reversed: RandomAccessCollection where Element == Base.Element { }

The conditional conformance must be satisfied by the underlying concrete type (here, ReversedCollection ), and the extension must be empty: Reversed is the same as ReversedCollection at runtime, so one cannot add any API to Reversed beyond what ReversedCollection supports.

If the concrete type is public then I can retroactively extend it with custom protocols, even add custom conditional conformances or simply add more custom API to the type. When such a type is hidden by an opaque type then we don't know which conditional conformances could exist on the hidden concrete type and therefore a library user can no longer add custom conditional conformances, retroactively conform to custom protocols or simply extend the API of the hidden concrete type. Why is this restriction necessary? I think there is enough information exposed by the opaque type to allow extending all subsets of opaque types at once.

// module A
public protocol P {
  func value() -> Int
}

extension Int : P {
  public func value() -> Int {
    return self
  }
}

extension Double : P {
  public func value() -> Int {
    return Int(self)
  }
}

public struct T {
   func opaqueP() -> opaque P {
     return 42 // Int
   }
}

// module B
// extends the concrete type that conforms to `P`
extension opaque P {
  func multipliedByTwo() -> Int {
    return self.value() * 2
  }
}

let t = T()
let p = t.opaqueP()
let value = p.value()
let new = p.multipliedByTwo() // this should be valid, because we indirectly extended `Int` in this case

Loosing the ability to extend these types would be simply a shoot in the own foot.

Hi @Douglas_Gregor,

Could you expand on the reasons for "I don't want to write the type"? In the document it just says that the type can be rather ugly, and that the author doesn't want to have to reason about it. I could perhaps see such an argument for end user code, but I would think reasoning through the types is vital for avoiding errors in library creation (especially the standard library). I feel like I must be missing something here...

The main reason I ask is that I feel like we are giving up both power and understandability in the long term by not explicitly writing the type. Specifically, I would like to gain the ABI protective power of opaque types for more than just return types. For example, instead of needing a custom wrapper, you could just use an Int for a type's index, while only exposing the fact that it is Comparable:

typealias Index = Int as Comparable  //Feel free to use a term besides "as"

Having both the internal and external components defined makes it easier to explain as a concept, especially when used with a setter. Using the magic return types, you have to know to look in the getter to understand the behavior of the setter:

With a subscript or a computed property, the type of the value provided to the setting (e.g., newValue ) is determined by the return statements in the getter, so the type is consistent and known only to the implementation of the property or subscript.

But with the type explicitly defined, you can just look for the type in the usual place:

var values: [Int] as P {
    get {...} //Dealing with [Int] here
    set {...} //Dealing with [Int] here too... only external sees it as P
}

Explicitly defining the internal type makes the concept of Opaque types much more concrete (at least to me) because everything is all in one place.

All of that said, if having to write the type really is a dealbreaker, would you be willing to consider a syntax that defaults to explicitly spelling things out, but lets you omit it when the compiler can figure it out for itself? For example:

func foo() -> _ as B:BidirectionalCollection where B.Element == Element

(This would honestly make me feel slightly better about the _.Element spelling, since the _ would match, though I would still strongly prefer $0 or explicit naming.)

3 Likes

As for supporting conditional conformances, we should, but I think it should be a separate proposal to update the syntax of where clauses in general to handle such things.

As a straw man proposal, add the keyword when within a where statement to indicate that that part of the clause is conditional on the part following when:

... where A == B, C == D when C:Comparable, E:Equatable
//A==B and E:Equatable are unconditional
//C==D is only applied as constraint when C is Comparable

If we define a syntax for conditional conformances in general, then it just naturally applies to opaque types:

extension BidirectionalCollection {
    public func reversed() -> ReversedCollection<Self> as 
                              C: BidirectionalCollection where C.Element == Element,
                              C: RandomAccessCollection when Self: RandomAccessCollection,
                              C: MutableCollection when Self: MutableCollection
}

This is a nice feature - it could help with some cases where we'd like to use existentials.

  • I very much like the idea of obscuring some of the standard library's transformed Collections. Not just because they have long, inelegant names, but also because it would allow us to drastically reduce the ABI-locked interface for the standard library.
  • Existentials are crucial for protocol-oriented programming. The more we can all make use of it, the more people will understand why, for example, associated-types and constrained existentials are a superior model to generic protocols.
  • Having the ability to tell the compiler the axes along which an existential box might be occupied by different types is very convenient for certain use-cases.

On the other hand...

  • Not a fan of the opaque keyword. All existentials are 'opaque'. These are actually less opaque than regular existentials (i.e. without the keyword), since the underlying identity is parameterised in some defined way (e.g. the function it came from). Also, this is a case where I'd support (gasp!) an @attribute.
  • The proposal talks about existentials conforming to their own protocols like it will never happen (even with generalised existentials), yet that is contradicted by the linked SO post and SR-55, which makes clear that these limitations are due to static requirements in the given protocols. I don't see how it's related to this discussion.
  • I'm fine with not requiring the result-type to be named, but I'm not-so-fine with not being able to explicitly refer to the result-type by any means and relying on inference to fill some magic, un-spellable thing in.
    Most of the obscure error messages and poor compilation performance I have seen are ultimately traced back to type-inference, so I've come to value the ability to sometimes explicitly state my intention. I also make heavy use of deferred-initialisation in my code, which requires a type specifier; boxing functions in PATs for this purpose will defeat the goal of reducing API surface area (if exposed), or drastically increase boilerplate (if not exposed).
  • I think there should be some way to explicitly communicate which (if any) generic parameters the opaque result depends on. For example:
// Result type does not depend on T
func makeACollection<T>(of items: T...) -> opaque Collection where Element == String {
  let strings = items.lazy.map(String.init(describing:))
  return MyImplementationDetail(copyingFrom: strings)
}

Also, a question: when discussing generalised existentials, path-dependent types come up quite often. If we did that, could we retrofit this feature with PDTs later-on?

I have more questions/comments but I'll leave it there for now.

Still a cool feature. Thanks for this!

5 Likes

I like the proposed feature, and it's great if it helps to make stdlib even better in terms of exposed types.

However, I do not think there's reasonable argument for not naming the opaque types. I would very much want to see the opaque types named always, because:

  • It's easier to reason what the code is doing, when trying to understand somebody else's code
  • There's less accidental errors, because we are not relying on the compiler to infer things correctly
  • We can use the same approach, if at some point there can be multiple opaque types in same statement
  • Swift is supposed to have ease of use and progressive disclosure as priorities. Trying to understand "_.Element" or ".Element" and what they refer to is much harder than understanding named type, such as "O.Element", because you can find the other places where that type is mentioned.
  • If the main motivation for opaque types is to be used in libraries to hide concrete types, it's not a vast amount of code (compared to all end user app code). So spending couple of seconds to write a type is ok.
5 Likes

No, that is ill-formed because the result of calling f7 cannot be coerced to Int. You could use as! Int and it would always succeed.

There is no missing ','; it would have been parsed as part of the where clause's list. The opaque keyword is enough here to denote a new clause.

Yes, I forgot. Updated now (along with typo fixes), thanks!

One reason is that we now need to be able to name the opaque result type from the outside. You can't extend opaque P: you need to extend "the opaque result type returned by T.opaqueP()".

If you're going to allow extensions of opaque types, the next logical step is to allow those extensions to declare protocol conformances. That's where we get ourselves into trouble, because:

(1) It becomes very easy to end up with multiple conformances of the same type to the same protocol, because it's easy for a user to extend "the opaque result type returned by T.opaqueP()" to conform to some protocol---even one exposed by the module A---and then have that conflict at runtime because the underlying concrete type already conformed to that protocol. Note that this problem could manifest differently over time: maybe T.opaqueP() returns Int today and there is no conflict, but a newer version of the library starts returning Double and now your conformance conflicts.

(2) The Swift runtime implementation currently wouldn't have any way to find the conformance at run time, because there is no runtime representation of opaque result types (intentionally!). Adding this additional capability significantly increases the complexity of the implementation.

Doug

1 Like

FWIW, an opaque typealias that allows conformances distinct from the underlying concrete type to be declared would be very much like a newtype (aside from also having distinct type identity). I have been wondering if there would be any implementation overlap between opaque types and newtype and whether newtype might make sense as a future direction.

4 Likes

Having a function with a result type that looks like this:

LazyMapSequence<
  LazyFilterSequence<
    LazyMapSequence<Elements, ElementOfResult?>
  >,
  ElementOfResult
>

is not helping anyone reason about the code. It is flooding them with irrelevant details. Better to say "it's a Collection whose Element type is ElementOfResult".

There really isn't any interesting inference here. The return statement can spell out the type explicitly, or leave it inferred, but callers are unaffected either way.

Personally, I don't consider multiple opaque types to be important enough to base other design decisions on it. It's left as a discussion (not a specific part of the proposal) because I suspect it would become more of an anti-pattern.

Maybe, but having a name that's used in the declaration but not available elsewhere is also confusing. Frankly, I wouldn't want the name in the declaration (e.g., in the generated interface or documentation) unless clients can use the name.

The motivation is two-fold: don't have to write these complicated composed types, and can hide them from clients. @gregtitus made a good argument that the former is the more important motivation, and I tend to agree.

Doug

I answered a similar question in Opaque result types - #169 by Douglas_Gregor; let's continue based on that one.

I think what you're asking for is opaque type aliases.

Doug

All I'm advocating for is using opaque-type-aliases instead of spelling out the whole thing, as well as (requirement for) named syntax for

func compactMap<...>(...) -> opaque O: Collection where O.Element == ElementOfResult { .. }

instead of using "_.Element". I think you might have misunderstood my intention.

I'm not sure what you expect these conditional conformances to do elsewhere. One could add conditional conformance clauses to generalized existentials, I guess, but since existentials don't conform to protocols it's unlikely that you'd be able to use them. For other cases (e.g., where clauses for extensions), you really want separate extensions because the where clause affects everything inside the extension and there's no way to make use of the conditional clauses.

Doug

I'm not a fan of the latter because I don't like introducing the name only for the scope of the declaration:

Doug

I understand this preference. However, you are trading this benefit at the cost of having to learn a new unique syntax. On the other hand "opaque O: Collection where O.Element" is immediately understandable to anyone who has used where clauses in protocols or elsewhere. But I'll let the core team to decide where the priorities are.

13 Likes

The limitations with static requirements/initialize requirements are fundamental, and are exacerbated by the way in which existentials are represented in the run-time implementation. So, while we could make existentials of protocols that don't involve static/initializer requirements conform to their own protocols, there would still be a "cliff" where that steps being true. I'll try to word this more accurate, but I think the point stands.

As I noted in another reply, there's no interesting type inference going on here: the return statement is type-checked without any contextual type information, and then we make sure it satisfies the requirements of the opaque result type.

It's a plausible extension; one would have to spell that somehow in the signature (since it affects clients' understanding of type identity), but it's an easy addition.

For opaque result types, you could certainly allow that syntax:

let foo = getSomethingOpaque()
let bar: foo.Element = foo.first!

However, it wouldn't be "path-dependent", because we have an understanding of static type identity for opaque result types. You could change let to var and do some mutation and it would still work with opaque result types (but would be a compile-time error with generalized existentials):

var foo = getSomethingOpaque()
let bar: foo.Element
if Bool.random() {
  foo = getSomethingOpaque()
}
bar = foo.first!   // okay with opaque result types + associated type references, compiler error with generalized existentials

Doug

Alternatively, we could say that taking a path-dependent type pins the dynamic type of a var (effectively narrowing its type from the existential type to its opened Self type at the point of opening), which would move the error here:

var foo = getSomethingOpaque()
let bar: foo.Element
if Bool.random() {
  // okay with opaque result types + associated type references, compiler error with generalized existentials
  // because you can't change the dynamic type of foo once it's opened
  foo = getSomethingOpaque() 
}

I think that the notation of path-dependent types could still be useful for non-existential opaque types, since it gives you a way to indirectly name a value's type without naming the type itself. In diagnostics, "type of foo" might also be a more immediately understandable way of describing the type than "opaque return type of getSomethingOpaque() <...environmental generic arguments...>".

1 Like

newtype is most closely related to opaque type aliases, because both introduce a new named type that is primarily described by its capabilities but has the same storage as some underlying type. One can think of new type as sugar for a struct containing a single stored property of the underlying type, so something like:

newtype Meters = Int

would desugar to something like:

struct Meters {
  var underlying: Int
}

Generally, one would expect some kind of opt-in forwarding of APIs from the underlying type to the newtype. For example:

newtype Meters: Numeric = Int

Now, Meters should conform to Numeric, using the implementations from the conformance of Int to Numeric. But this starts to fall apart when Self is used in the the protocols. For example:

protocol RandomValueSource {
  func getRandomValues(_ numValues: Int) -> [Self]
}

extension Int: RandomValueSource {
  func getRandomValues(_ numValues: Int) -> [Int] {
    return Array((0..<numValues).map(_ in Int.random(in: 0..<100)))
  }
}

How do we forward that conformance of RandomValueSource for Meters? It returns an [Int], and now we need a [Meters], even though it is a completely different type.

Because of this, I've mostly lost interest in newtype. Maybe Swift should get some kind of forwarding syntax to make it easy to write something like Meters as a wrapper type for Int, with opt-in synthesis for all of the members where it does work. That would be a nice way to eliminate some boilerplate from the system without introducing a new concept.

Opaque result types don't have these issues because there is no new runtime type involved.

Doug

5 Likes

This makes sense. It is the direction I was exploring in this draft: [Proposal Draft] automatic protocol forwarding. I had a second draft in progress but abandoned it when it became clear it wasn't going to be in scope in the near-term. I had suggested a future direction of newtype as syntactic sugar for the underlying forwarding mechanism. Obviously this direction is orthogonal to opaque type aliases despite the superficial similarity.

There's another case of protocol forwarding that might be interesting in composition with opaque return types. It's not uncommon for an API to return one of a handful of different types depending on conditions, e.g.:

func elements() -> AnyCollection<Element> {
  return _elementCount = 1 ? AnyCollection(CollectionOfOne(_element)) : AnyCollection([_elementA, _elementB])
}

Even with generalized existentials that self-conformed to their own protocol, a fully general type-erased container is overkill for the finite set of result types here. It'd be nice to be able to define an enum that automatically forwards conformance from its payloads and use it as an opaque return type here:

func elements() -> opaque Collection where _.Element == Element {
  enum ElementsCollection: Collection {
    case one(CollectionOfOne<Element>)
    case many(Array<Element>)
  }

  return _elementCount = 1 ? .one([_element]) : .many([_elementA, _elementB]) as ElementsCollection
}

The opaque type would still give the implementer freedom to change the exact set of types their implementation produces, without the overhead or complexity of a fully generalized existential or type-erased container.

2 Likes

It would be very cool to be able to have a forwarding mechanism powerful enough to do this! I would love to revisit the forwarding proposal someday but don't want to invest time without having a collaborator who is able to work on implementation.

On another forum I was discussing ways to try to clean up long where clauses, and came upon this sort of construction, which works great in Xcode 10 betas (though not in Swift 4.1):

extension Collection {
    public typealias OtherCollection<C: Collection> = C where C.Element == Self.Element
}

extension Collection where Element : Hashable {
    func sameElements<T>(with: OtherCollection<T>) -> Bool {
        return Set(self) == Set(with)
    }
}

That is, we can avoid repeating common sets of requirements by offloading them onto a named generic typealias and the resulting code looks pretty nice.

Personally, I would love to see the stdlib interface someday look like:

extension Collection {
    public typealias OtherCollection<C: Collection> = C where C.Element == Self.Element

    var lazy: opaque OtherCollection<_>
    func filter((Element) -> Bool) -> opaque OtherCollection<_>
    func reversed() -> opaque OtherCollection<_>
    /* etc. */
}

That is, a "collection with the same element type as this collection" is a very common thing to want to name and refer to, and I'd love to be able to do so with a concise and common shared spelling like this one.

However! This use of type aliases to offload requirement clauses would be broken by the proposed opaque type alias future direction in this proposal, because it would require all of those OtherCollections to be the same concrete type, which they obviously are not.

So I just wanted to throw that out as a possible bump in the road with regard to that section of the proposal.

2 Likes