Scoped Conformances

Authors: Dave Abrahams (@dabrahams) and Dmitri Gribenko (@gribozavr), with support and feedback from the Swift for TensorFlow Team.

This pitch is related to and in support of the recently posted pitch, Automatic Requirement Satisfaction in plain Swift.

Introduction

In Swift, protocol extensions provide a beautiful way to share API implementations across value types. For example, an extension on the Equatable protocol provides a public implementation of the != operator to all public conforming types. The power of this feature is limited, however, because the conformance can never be less visible than APIs provided by extensions depending on that conformance.

For example, suppose we often find ourselves creating types of this form:

public struct X: Comparable, CustomStringConvertible {
  internal var value: SomeType

  public static func <(a: Self, b: Self) -> Bool {
    a.value < b.value
  }

  public var description: String { String(describing: value) }
  ...
}

It’s tempting to use protocol extensions to provide the implementations of < and description; in theory we could simply write:

public struct X: Wrapper, Comparable, CustomStringConvertible {
  internal var value: SomeType
}

and let extensions that depend on Wrapper do all the hard work. However, when we try to define these extensions, we’ll find that Swift won’t let us keep Wrapper internal.

internal protocol Wrapper {
  associatedtype Value
  var value: Value { get }
}

extension Equatable where Self: Wrapper, Value: Comparable {
  // error: cannot declare a public operator function in an
  // extension with internal requirements
  public static func <(a: Self, b: Self) -> Bool {
    a.value < b.value
  }
}
// similar extension of CustomStringConvertible...

The only way to achieve the code reuse we’re after is to make Wrapper a public protocol, which in turn makes X’s conformance to Wrapper and everything needed to satisfy it (value and Value) part of Xs public interface.

Thus, we are forced to choose between boilerplate and exposing more API than intended. Even if we use underscore-prefixed APIs to hide unwanted public names, code reuse comes at a hidden cost: the compiler is forced to generate public type metadata and witness tables for all the requirements of Wrapper, bloating binary size with information that will never be used.

Because this limitation compromises a large class of otherwise-useful API building blocks, we propose to lift it.

Scoped Conformances

To begin with the minimal pitch serving the purposes of the authors, we need to modify the problem slightly; the exact motivating example is covered in “Future Directions.” In the meantime, imagine we wanted to expose Wrapper and X publicly, but not X’s conformance to Wrapper or the API that X depends on to satisfy Wrapper’s requirements (value and Value).

To keep the Wrapper conformance out of X’s public API, we propose an idea discussed briefly in the Generics Manifesto: scoped conformances. With a scoped conformance, the declaration of X could be written this way:

public struct X: fileprivate Wrapper,
  Comparable, CustomStringConvertible
{
  internal var value: SomeType
}

The conformance to Wrapper, being visible only in the file where X is defined, is available to allow the requirements of Comparable and CustomStringConvertible to be satisfied, but does not leak into X’s public API or force its value or Value to become public. The explicit fileprivate label cues the compiler to try to eliminate associated metadata, which—because there can be no existential Wrapper—is trivially possible.

Note: the original title of this pitch was "Non-public conformances implementing public API," to emphasize that while X: Wrapper is not visible outside the file, the func < that depends on X: Wrapper is visible publicly.

Dynamic Casts

The manifesto identifies the handling of dynamic casts as the major problem with scoped conformances, using this example:

// File1.swift
public protocol P { }
public struct X { }
extension X : internal P { ... }

// File2.swift
func foo(value: Any) {
  if let x = value as? P { print("P") }
}
foo(X())

The question posed is, “Under what circumstances should [the program] print ‘P’?” Our answer is that X’s conformance (or not) to P is captured in the scope where the X value is first converted to an existential type. If the cast occurred in the scope where a conformance of X to P is visible (in the same module as File1.swift, in this case), foo will print “P”. So if File1.swift and File2.swift are in the same module, the internal conformance to P is visible at the call to foo, where the X instance is converted to Any, and “P” is printed. If File1.swift and File2.swift are in separate modules, nothing is printed.

Future directions

Thinking of the problem in terms of a public protocol and a fileprivate conformance satisfies the authors’ immediate needs, and is enough to support the vending of useful API building blocks, like Wrapper, from libraries. It also simplifies the discussion by only using a public protocol to create public API.

But what about the original example, where we want to keep Wrapper internal? We believe this is a mostly-straightforward extension of the same idea that, because the conformance does not need to escape the module scope, boils down to the same arrangement, except with Wrapper entirely hidden from X’s clients. There are, however, a few details that need to be worked out (see “Open Questions,” below).

We think it should also be possible to extend the system to support conformances scoped more narrowly—to a type or to a function body, for example—but haven’t considered the question deeply.

Beyond these examples, we believe this design provides a framework for thinking about the answers to unsolved or unspecified problems in Swift’s generics system, such as how conformances in multiple modules should work, and how algorithm specialization can work with conditional conformances of generic types, because it enshrines the idea that the visibility of conformances in the scope where a generic parameter is bound to a concrete type, or an existential is formed, determines how implementations are fulfilled.

Open Questions

We believe these issues are not difficult to address, but as the work still needs to be done, they bear mentioning:

  • If we allow public requirements to be satisfied by extensions depending on non-public names (as in the opening example), must the satisfying definitions (< and description) be explicitly declared public?
  • Should we allow the new public API that doesn’t satisfy any existing requirements to be generated by extensions that depend on non-public names? If so, presumably only when the new API is explicitly declared public?
  • In a scoped conformance to a refined protocol, what scope is implied for the conformances to the protocol’s less-refined ancestors?
  • How is scope resolved when two scoped conformances imply different scopes for a common ancestor protocol?

Earlier Related Discussion

In 2018, Jordan Rose raised the possibility of allowing non-public methods to satisfy public protocol requirements, a similar feature that achieves some of the same goals.

29 Likes

Is there an implementation plan for this? Right now, the representation of an Any value does not include where it was formed. The representation of an AnyObject value that's actually a class cannot include where it was formed.

I'll probably have further thoughts on this approach in general later, but this bit seems important to clear up sooner rather than later. (I think my preferred approach would be "non-public conformances never satisfy dynamic casts".)

1 Like

Thanks for raising this, Jordan! There's currently no implementation plan for it, and we'll have to look into the points you've raised. Naturally if it turns out that these semantics are unimplementable, we'll have to accept an alternative semantics, and in that case I would prefer the same approach as you.

1 Like

Can you elaborate on why this choice was made rather than choosing an approach that is driven by visibility of the conformance at the site of the cast?

How would you describe a public conformance that relies on non-public names? i.e. in your example Wrapper is internal. How would users of X know when it conforms to Comparable? What if X was generic and the conformance to Wrapper was conditional?

In the case where the conformance to a less-refined ancestor is explicit the answer is clear: use the scope of that conformance declaration. When the conformance is implied I think the scope should also match the declaration, but haven't thought too deeply about it.

It seems like this would have to be an ambiguity error. An explicit conformance to the ancestor would be required to disambiguate.

I think I would find this to be surprising behavior. Would it be possible to have (at least in some cases) a compile-time warning about this?

I've wanted something like this for a long time. In particular, it could make defining COW storage types easier:

protocol Copyable {
  func makeCopy() -> Self
}
protocol HasCOWStorage {
  associatedtype StorageType: Copyable
  var storage: StorageType { get set }
}
extension HasCOWStorage {
  public subscript<T>(dynamicMember: ReferenceWritableKeyPath<StorageType, T>) -> T {
    get { 
      return storage[keyPath: dynamicMember]
    }
    set {
      if !isKnownUniquelyReferenced(&storage) { storage = storage.makeCopy() } 
      storage[keyPath: dynamicMember] = newValue
    }
  }
}

If you could have a fileprivate conformance satisfied by a fileprivate member, you could do this:

public struct MyLargeValue: fileprivate HasCOWStorage {
  public class StorageType: Copyable {
    var x: Int64
    var y: Int64
    var z: Int64
    // ... lots of ivars.
    func makeCopy() -> Self { /* ... */ }
  }

  fileprivate var storage = StorageType()

  // export the keypath magic subscript somehow.
}

Even though you expose your storage type (in order for the keypath magic to work), you don't actually expose the .storage member, so clients wouldn't be able to bypass your COW checks.

EDIT: I guess nowadays you'd do some kind of fancy thing with property wrappers, but this has stuck with me for years as something I'm annoyed Swift can't do .

1 Like
I mis-read the proposal. Post is collapsed for posterity.

+1 to the feature. I have also been wishing for something like this. As an API author, there are times when I don't want to expose aspects of functionality as part of my public API. For example, if I want to expose some sort of opaque token type, I sometimes don't want to also expose that it's Comparable or RawRepresentable or whatever, because that's not how I'm intending clients to use it.

I'm not a fan of this syntax. In an ideal world, I would be in favor of a rule like:

Conformances inherit the visibility of the declaration.

So:

// conformances to Comparable and CSC are public because this is a public declaration
public struct X: Comparable, CustomStringConvertible { ... }

// conformance to Wrapper is internal because this is an internal declaration
internal extension X: Wrapper { ... }

This avoids introducing any new syntax and avoids confusion around doing something like this (which would be syntactically allowed but semantically rejected):

internal struct X: public Comparable { ... }
2 Likes

At the level below the language level, it does not matter that the public conformance uses non-public names in it. Consider the following, which is obviously OK:

internal func compareHelper(lhs: X, rhs: X) -> Bool { ... }

public struct Y: Comparable {
  public static func <(a: Self, b: Self) -> Bool {
    return compareHelper(lhs: a, rhs: b)
  }
}

The conformance of X to Wrapper is internal, but its conformance to Comparable is public, and would be treated everywhere (in the compiler, in the module interface etc.) as such.

Then the conformance of X to Comparable would be conditional with the same conditions.

But if one of those conditions is the Wrapper conformance how do users know when that condition is met and when it is not met?

We could extend the value witness table representation with a list of private conformances. Dynamic casts would first look into the VWT. Existing public VWT symbols would point to a VWT with an empty list of private conformances. When we need a VWT for a given scope where the existential is formed, we can use the default VWT if there are no private conformances. If there are private conformances visible in that scope, we create a new VWT dynamically by copying the default one and attaching the list of visible private conformances.

Sorry, could you provide an example?

Sure! One big reason is that it's consistent with the intended behavior of generic parameters as described by @Douglas_Gregor in this post, which is the only sensible model, AFAICT, for dealing with conflicting conformances in multiple modules. IIUC it is also consistent with implicits in Scala and other precedented language features.

How would you describe a public conformance that relies on non-public names? i.e. in your example Wrapper is internal. How would users of X know when it conforms to Comparable ?

Oh, do you mean, “how would such a conformance be described to users of X's public API?” In the example given, the conformance of X to Comparable is unconditional, so users would see it as a regular conformance to Comparable. If it were a conditional conformance, you still wouldn't be allowed to use private names in the where clause of its conditional conformance, so again, it would be documented in the ordinary way.

Yeah, that's the easy case ;-)

My instinct has been telling me that we could simply take the most-public implied conformance of the ancestor—why force users to disambiguate if we can avoid it?—but I'd like to have a discussion to analyze the implications of a choice like that more carefully. Certainly, forcing explicit disambiguation is a conservative choice; we can always take a less-strict approach later.

1 Like
@davedelong mis-read the proposal. Post is collapsed for posterity.

That is the rule today, and unfortunately it doesn't cover our intended use case. We have a public protocol, and we need our clients to be able to conform their public types to that public protocol without exposing the conformances. Thus the explicit fileprivate qualifier on the conformance.

I mis-read the proposal. Post is collapsed for posterity.

This is exactly the situation I'm talking about, but I'm sure I didn't state the "rule" well enough because I'm not well-versed with compiler terminology.

I'm saying we should make this syntax work:

public protocol Fooable {
    func doFoo()
}

public struct Bar {
    let baz: String
}

internal extension Bar: Fooable {
    func doFoo() { print(baz) }
}

Right now this dies with several errors:

error: method 'doFoo()' must be declared public because it matches a requirement in public protocol 'Fooable'
    func doFoo() {
         ^

note: move the instance method to another extension where it can be declared 'public' to satisfy the requirement
    func doFoo() {
         ^

error: 'internal' modifier cannot be used with extensions that declare protocol conformances
internal extension Bar: Fooable {
^~~~~~~~~

If we change the compiler to allow this, then we don't need to invent new syntax at all.

@davedelong mis-read the proposal. Post is collapsed for posterity.

Thanks for explaining. I'm afraid that for your suggestion to cover our use case, it would require changing the meaning of existing language constructs. Today, the access specifier preceding the extension sets the default access for the definitions within the extension, so what you wrote is equivalent to:

extension Bar: Fooable {
    internal func doFoo() { print(baz) }
}

IIUC you're now saying it should mean something like (in my proposed syntax):

extension Bar: internal Fooable {
    public func doFoo() { print(baz) }
}

But I'm pretty sure that is not a backward-compatible change for existing code.

I mis-read the proposal. Post is collapsed for posterity.

Putting a visibility modifier on an extension is shorthand for putting the same visibility modifier on every otherwise-unspecified thing within the extension.

Thus:

internal extension Bar: Fooable {
  func doFoo() { print(baz) }
}

should be equivalent to this (using your proposed syntax):

extension Bar: internal Foo {
  internal fun doFoo() { print(baz) }
}

So my question is:

Can we not just use the visibility of the extension to also apply to the visibility of the conformance?

It is not a backwards-compatible change. Today, internal in internal extension does not apply to protocol conformances. So implementing your suggestion would change meaning of existing code.

I'm not sure I follow. The syntax I'd like to see isn't allowed by the compiler, which means there is no existing code that has it.

Oh -- yes you're right, it is a special-case error today. I guess we could allow it, I don't see a problem technically, except for not being able to continue teaching internal extension as a syntax sugar -- because there is no syntax that it desugars to.

@davedelong mis-read the proposal. Post is collapsed for posterity.

But that wouldn't be legal in today's language, as you'd be satisfying a conformance to a public protocol (Foo) with an internal definition. That, as I said at the end of the pitch, is a different feature, and it raises some issues that our proposal does not. We're not prepared to try to work out those issues, nor are we sure that if they were worked out, our use case would be covered.

1 Like