Opaque result types

[ahti (Lukas Stabe 🙃)]

@anthonylatsis

How is this true? The following should be perfectly fine once we fully support opened existentials.

func foo<T: Collection>() -> T where T.Element == Int { return Array<Int>() }

If such a function really had to be able to return, or coerce to, whatever type satisfies the constraints , the choice would indeed be after the user.

Otherwise, it isn't really a choice – rather, it's a guess through coercing, as it would be for a function returning an opaque type. If you want to coerce to some concrete type, that is.

The function is declared as returning an instance of the generic parameter T. The caller chooses the generic parameters. Writing foo() as [Int] or foo() as Set<Int> isn't "a guess through coercing", it's simply telling the type checker what type you expect as a result of the call to foo(), and thereby indirectly setting the generic parameter.

The function body is incorrect, because it tries to return [Int] where T is needed, and thus won't compile. Opening existentials is unrelated as far as I can tell.

[hartbit (David Hart)]

@Douglas_Gregor

I think the right course forward for this proposal is to introduce opaque type aliases, which give a name to the opaque type so that it can be reused/tweaked. That provides a natural way to describe the conditional conformance, so we can eliminate the awful conditional-conformance-in-the-result-type syntax from the proposal.

That's exactly my opinion, and here's the syntax I was thinking of:

opaquetype SubBidirectionalCollection<C: BidirectionalCollection> = BidirectionalCollection where Element == C.Element
extension SubBidirectionalCollection: RandomAccessCollection where C: RandomAccessCollection {}
extension SubBidirectionalCollection: MutableCollection where C: MutableCollection {}

extension BidirectionalCollection {
    func reversed() -> SubBidirectionalCollection<Self> {
        return ReversedCollection<Self>(...)
    }
}

[anthonylatsis]

Ah, apologies. @xwu, the question is resolved. I left the actual meaning of that generic parameter completely unnoticed.

@ahti

The function body is incorrect, because it tries to return [Int] where T is needed, and thus won't compile. Opening existentials is unrelated as far as I can tell.

Yes, that's true. Thanks.

[hlovatt (Howard Lovatt)]

@Torust

With an opaque type, you know that the particular type returned from a particular method will always have the same associated types

In Swift you have a closed world compilation (except for OS upgrades) that allow you to know the type, this is unlike Java etc. that allow parts of the program to be loaded dynamically that the original compiler has never seen. Consider the makeIterator example:

protocol IteratorProtocol<Element> {
    func next() -> Element
}
protocol Sequence<Element> {
    func makeIterator() -> IteratorProtocol<Element>
    ...
}
struct Array<Element>: Sequence<Element> {
    func makeIterator() -> IteratorProtocol<Element> { return ArrayIterator(self) }
    ...
}
private struct ArrayIterator<Element>: IteratorProtocol<Element> { ... }

Although makeIterator returns IteratorProtocol the compiler knows that it really returns the private struct ArrayIterator and hence it annotates makeIterator in Array as returning ArrayIterator. This information , ArrayIterator, can then be used by the optimizer.

The key difference between Swift and Java is that Swift is a closed world compilation and therefore the compiler can annotate with actual types.

[Torust (Thomas Roughton)]

@hlovatt

The key difference between Swift and Java is that Swift is a closed world compilation and therefore the compiler can annotate with actual types.

This is only true within a module, and not across resilience boundaries. You’re completely correct that the compiler should be able to optimise that within a module, but what about the scenario where the calling module can’t be allowed to know ArrayIterator exists or depend upon the returned type always being ArrayIterator?

The idea of ABI stability is that dependencies should be able to change their implementation (including what types they return) without the dependant code having to recompile. That doesn’t work if the ABI is locked to specific types.

[hlovatt (Howard Lovatt)]

Except for OS upgrades (see below) you do know everything at compile time. Third party libraries are shipped with your code and you are compiling against these. IE the whole of the binary, except for the OS bit, comes from your compiler and therefore you can optimise across boundaries.

This leaves the OS upgrade, either:

1. An Apple only annotation would be required that promised that the return type wouldn't change when the OS changes would be needed; or
2. Your code would have to be re-linked and re-optimised by Apple when they upgraded the OS.

PS I have said Apple, but same would be true if another vendor shipped Swift Foundation as part of their OS. Simpler just to say Apple, since another vendor is some way off .

[anthonylatsis]

@Douglas_Gregor

I've been thinking about this for a bit, and the second bullet is really what gets to me. That syntax I conjured up to describe conditional conformance is rather awful. It's completely non obvious and quite verbose. @dabrahams improved on it a bit, but it's verbose by necessity.

A couple of ideas on this matter.

//#1 Tackling the syntax

// For in-place definition, implies that the opaque type can be named, and a 'when' keyword. 
-> opaque T where T: BidirectionalCollection,
                  T: RandomAccessCollection when Self: RandomAccessCollection,
                  T: MutableCollection when Self: MutableCollection,
                  T.Element == Element

// Through type aliases
typealias T = opaque BidirectionalCollection where 
                     T: RandomAccessCollection when Self: RandomAccessCollection,
                     T: MutableCollection when Self: MutableCollection,
                     T.Element == Element

// #2 Tackling verboseness

// If the compiler could infer a single return type and then auto-generate the necessary
// constraints for T, it's opaque version (the 'existential skeleton'),
// that would be fantastic. T.Element == Element is an example of an additional external constraint.
-> opaque T where T.Element == Element

@Douglas_Gregor

I think the right course forward for this proposal is to introduce opaque type aliases, which give a name to the opaque type so that it can be reused/tweaked. That provides a natural way to describe the conditional conformance, so we can eliminate the awful conditional-conformance-in-the-result-type syntax from the proposal.

What we really want is to be able to express merely the existence of a conditional conformance, right? I wouldn't say the current syntax for declaring conditional conformances is suitable for that, especially if we want to express a conditional conformance in a signature.

[Torust (Thomas Roughton)]

Aside from the fact that Apple recompiling your code when the OS is upgraded would require them having access to your source (or at least all private and internal implementation details of your modules), I think you’re ignoring other cases for 3rd party libraries. I can think of a number of C++ libraries which are binaries + headers (and since they don’t have ABI stability it usually means a separate version for Clang or GCC, MSVC, and libc++ or libstdc++); Autodesk’s FBX SDK and the FMOD audio library are the first two that spring to mind.

Let’s say you, as a Swift developer, wanted to build a non-trivial closed-source library that others could license to use (or that you can’t open-source for legal reasons). Because it’s closed-source, you want to keep the implementation details private so they can’t be trivially decompiled or reverse engineered - that means no private types leaking out to the public interface. Isn’t that sort of product as equally valid as end-user application development?

I think it might be worth splitting this out into a separate thread, as well - it’s fairly off-topic for opaque types.

[beccadax (Becca Royal-Gordon)]

Okay, but the use case described is specifically the standard library, which will be part of the OS starting in Swift 5. So basically, we’re designing the “Apple-only” annotation that allows the type to change.

[hlovatt (Howard Lovatt)]

But the point is; the third party libraries are shipped by you, not by Apple. Hence, when you ship the new version of your App you also ship the new version of the third party library and you have optimised your App against that new library. Therefore; you do not require ABI stability for third party libraries, only for Apple supplied libraries.

This could be as simple as an Apple only annotation, e.g.:

struct Array<Element>: Sequence<Element> {
    func makeIterator() -> @Allways(ArrayIterator<Element>) IteratorProtocol<Element> { ... }
    ...
}

@Allways(ArrayIterator<Element>) would imply or require @usableFromInline be an annotation on ArrayIterator.

[ahti (Lukas Stabe 🙃)]

I'm pretty sure I mentioned this to you before, but there are many use-cases for dynamic libraries that are neither built by Apple nor shipped bundled with some app. Those two are the ones possible for iOS apps, but every other platform has uses for dynamically linked libraries built by everyone from OS maintainers to end users.

Discussions relating to the performance and behavior of dynamically linked code are therefore important and can't be solved with "well that's an Apple only thing", because that is simply untrue.

[zwaldowski (Zachary Waldowski)]

I'm a huge fan of the feature and want it immediately. Always a good first step for a pitch. Plus, I'm here for an easily-digestible versions of Rust features any day.

I'm unclear where this would actually leave us relative to generalized existentials; it seems like everything I'd want out of that feature. I'd like the proposal to describe more clearly where it ends and the author's concept of generalized existentials begins, and where this feature's limitations are, because the as-is text assumes I have that context already.

I am a huge -1 on using a mystery (indeed, opaque) keyword to accomplish such an important syntax. If what actually ends up being implemented is early/limited generalized existentials (as it seems to me, though again I don't know quite how wrong I am), we should go ahead and burn the right syntax on this feature and incrementally lift the limitations like we have conditional conformances.

[hlovatt (Howard Lovatt)]

Currently there is only Apple dynamically linked libraries and as I said in a PS I was using Apple as a shorthand for OS Vendor. For separately installed, not part of App and not part of OS, third party libraries there is a long way to go. You need a version numbering scheme, the equivalent of @available, conditional compilation on version number, etc. It is however likely that like @available, @allways (or whatever it is called) can be extended once dynamically linked, non-OS, libraries are supported.

Also for non-Apple, the only 'foreign' language supported is C. Therefore, at this stage protocols, don't come into it.

[Jean-Daniel]

The fact the a library is embedded in an App does not means that the app vendor has the library sources.
While this is true today because there is not stable ABI, it would be perfectly possible to distribute binary framework to embed in app in the future.

Such library would not have to bother with @availability as updating it will be done at the same time that the app, but it may still be a resilience boundary, unless we introduced a 'embedded library' compiler mode.

[rintaro (Rintaro Ishizaki)]

@Douglas_Gregor

Opaque result types can also be used in stored properties that have an initializer, in which case the concrete type is the type of the initializer:

var strings: opaque Collection where _.Element == String = ["hello", "world"]

What's the motivation behind allowing this?

It falls out of the model and I see no reason to restrict it.

For example:

func generateCollection() -> opaque Collection where _.Element == String {
  return ["foo", "bar"]
}
struct Foo {
  var strings = generateCollection()
}

I assume:

var foo = Foo()
foo.strings = Foo().strings // okay.
foo.strings = generateCollection() // error: because the type of `foo.strings` is "the opaque type of `Foo.string`" instead of "the opaque result type of `generateCollection()`"

But you said:

@Douglas_Gregor

//---- Module B
var baz = foo()

If so, what gives type(of: baz) ?

It's "the opaque result type of A.foo() ". It's going to be weird when it shows up in diagnostics, and I hope we can come up with a good way to describe it.

I'm confused. What about:

//------ Module A
public protocol P {}
private func generateP() -> opaque P { ... }

public var globalValue1 = generateP()
public var globalValue2 = generateP()

//------ module main
import A
globalValue1 = globalValue2

Is this legal because the types of globalValue1 and globalValue2 are both "the opaque return type of generateP()"? If so, how the compiler knows those are the same types, from the .moduleinterface?

[dabrahams (Dave Abrahams)]

@hlovatt

In Swift you have a closed world compilation (except for OS upgrades) that allow you to know the type, this is unlike Java etc. that allow parts of the program to be loaded dynamically that the original compiler has never seen.

It is inaccurate to say that Java is different from Swift in this regard. The main point of the resilience work is to allow that dynamic relationship to keep working even when different revisions of the compiler were used to build the different parts (i.e., what @Torust said and I only just discovered after posting).

@hlovatt

The key difference between Swift and Java is that Swift is a closed world compilation and therefore the compiler can annotate with actual types.

Swift's generics system is fundamentally more expressive than Java's. That's why, for example, Java's Comparable interface needs to resort to runtime type-checking, even allowing the call to sort arrays of non-comparable items without a compiler error.

[ahti (Lukas Stabe 🙃)]

@hlovatt

Currently there is only Apple dynamically linked libraries

$ echo "public func answer() -> Any { return 41 }" | \
    swiftc -emit-module -emit-library -module-name DeepThought -
$ echo "import DeepThought; print(answer())" | \
    swiftc -L. -I. -lDeepThought -Xlinker -rpath -Xlinker $PWD -
$ ./main
41
# whoops, wrong answer
$ echo "public func answer() -> Any { return 42 }" | \
    swiftc -emit-module -emit-library -module-name DeepThought -
$ ./main
42
# much better, and no recompiling ./main

Sure, @available and more utilities supporting them would be nice, but you, me, and everyone else can produce and use dynamically linked libraries right now.

[hlovatt (Howard Lovatt)]

Yes you are right, I should have said on iOS (which you already pointed out to me but I didn't take in when I read your previous post).

[Douglas_Gregor]

@rintaro

I assume:

var foo = Foo()
foo.strings = Foo().strings // okay.
foo.strings = generateCollection() // error: because the type of `foo.strings` is "the opaque type of `Foo.string`" instead of "the opaque result type of `generateCollection()`"

Correct; that last line is an error.

@rintaro

But you said:

Douglas_Gregor:

//---- Module B

var baz = foo()

If so, what gives type(of: baz) ?

It's "the opaque result type of A.foo() ". It's going to be weird when it shows up in diagnostics, and I hope we can come up with a good way to describe it.

I'm confused.

type(of:) returns a runtime type that you can see when the program executes. It will be the underlying concrete type produced by the method. Within the static type system, we only know the type as "the opaque result type of (some declaration)".

@rintaro

//------ Module A
public protocol P {}
private func generateP() -> opaque P { ... }

public var globalValue1 = generateP()
public var globalValue2 = generateP()

//------ module main
import A
globalValue1 = globalValue2

Is this legal because the types of globalValue1 and globalValue2 are both "the opaque return type of generateP() "? If so, how the compiler knows those are the same types, from the .moduleinterface ?

Both globalValue1 and globalValue2 have inferred the type "opaque result type of generateP()", so they are known to have the same type.

Doug

[Douglas_Gregor]

@zwaldowski

I'm unclear where this would actually leave us relative to generalized existentials; it seems like everything I'd want out of that feature. I'd like the proposal to describe more clearly where it ends and the author's concept of generalized existentials begins, and where this feature's limitations are, because the as-is text assumes I have that context already.

Hmm. This section was meant to describe the differences. Can you say a little more about how/why that section misses the mark? I don't have a good sense of how to improve it.

Doug