Sealed protocols

It should also be noted that making a protocol sealed as proposed is not by itself enough to have exhaustive knowledge of the types conforming without significant implementation complexity and compile-time cost, so this proposal probably should not promise optimizations or language features that depend on this knowledge. Private types and local types within a module are not normally visible outside of their scope, and if we were going to base language features like exhaustive pattern matching or optimizations on this knowledge, it would require that every translation unit exhaustively scan the source of the program to find any private or local types that might be conforming, which would not normally be necessary.

Furthermore, layout optimizations on existentials are not as trivial as they may seem, since existentials are structural types, and significant amounts of runtime code expects existentials to have certain layouts. Existential-specific layout optimizations could also end up pessimizing conversions between related existential types, since their optimized forms could have significantly different layouts. An optimization pass could conceivably replace an existential by a synthesized enum in places where it's a concrete type, but we would likely have to reabstract to the generic existential representation any time we dynamically manipulate the existential type. It would also be impossible to lay out existentials for sealed public protocols differently without breaking ABI if we want it to be resilient to remove sealed-ness.

There are other benefits to this proposal for sure, but if exhaustive knowledge of conforming types is an important goal of this proposal, it would need some adjustment to achieve that goal, such as possibly restricting the conformance of private or local types. Otherwise, the proposal text should not take this for granted and should probably not promise anything about layout optimizations or exhaustive pattern matching.

6 Likes

I didn't mean to imply that this proposal would promise any of that. @Karl has already made it clear that it is out of scope for this proposal. sealed protocols have plenty of merit without those features. But I would like to see these directions explored in the future as they would reduce the number of tradeoffs we face when making design decisions. It would be fine if they were subject to limitations (such as all conformances must be visible in order to switch exhaustively).

I don't know too much about layout optimizations but it would be nice to be able to choose the logical semantics of protocols and existentials without having to pay for boxing. Conversions between related existentials are not always necessary so that cost isn't always a factor. It would sometimes be acceptable as a tradeoff if we had the ability to influence the choice of layout. re: resilience, I think it would be fine to require @frozen sealed for layout optimization of public protocols.

More specifically, the proposal draft at sealed protocols by karwa · Pull Request #972 · apple/swift-evolution · GitHub says:

Similarly, when the compiler has knowledge about the conforming types, it can use optimised operations to handle existentials.
Currently, we advise to make protocols which are only conformed-to by classes inherit AnyObject, but this then becomes part of the protocol's ABI
and clients may depend on it. sealed protocols have the possibility to lower this to an 'informal' optimisation within the declaring module,
and support more patterns between conforming types.

and makes passing reference to optimizability in other places, which is not really possible as proposed.

I see. Definitely makes sense to remove this text then. I think the feature has sufficient motivation without discussing optimizations.

I agree with Joe. I fear it will bring up many resilience and access control discussions which are not really the important problems to be facing at this point in Swift's evolution. I'm personally very -1 on this feature, from a "it adds a bunch of complexity for very little gain" perspective.

Edit: The complexity I'm concerned about here is "language and conceptual complexity", not compiler complexity.

-Chris

2 Likes

Can you elaborate on this? Just the basic proposal without any of the enhancements that have been discussed would be enough to eliminate the need for this awful hack that prevents conformances from being added outside a module. I have needed to use that hack in a few places and would very much like to get rid of it.

The fact is that there are important library design techniques that don’t work if conformances can be added outside the library. I think enabling them is an important goal and reduces complexity for both libraries and users by allowing the library to more clearly state its intent within the language itself.

6 Likes

Yes, please elaborate. There has been plenty of discussion on this topic over the years, and every time, people who write Swift libraries and applications every day share stories about the hacks and tricks they use to emulate this feature. You can't just drop a bomb like that and walk off ;)

The only access control discussion I can see is whether or not protocols should be sealed by default and made open instead of the reverse. It's worth having that discussion as soon as possible, but I think everybody understands that it's unlikely. I also don't see any "resilience" impact (in the @_fixed_contents/@_frozen sense); it's valuable even for source libraries to declare protocols as sealed. Again, writing good protocols is hard.

1 Like

The alternative to that hack is to document that this protocol is not to be conformed to, and leave it at that. Protocols aren't just bags of syntax. You must always read and understand the documented requirements of a protocol when you implement it – or else you haven't actually implemented the protocol.

In this case, if you read the requirements and they are "don't implement this", things are pretty clear. Proposing a new keyword is no small thing, and the case would need to be made that this proposal delivers significant benefit over this approach.

10 Likes

Yes. This is what the stdlib does today, and I don't hear much fuss about people trying to conform to StringProtocol.

I mean, the status quo is a very serious option, and nobody should feel bound to adopt the various hacks described in this thread.

This is especially true now that it has been clearly stated above that the pitch should not make any promise about possible compiler optimizations or extensions such as exhaustive switches.

1 Like

It's not in the standard library proper, but I would hope SwiftSyntax would seal the Syntax protocol instead of having an internal _SyntaxBase to serve as the "real" protocol for conformers.

1 Like

Some of us adopt a different philosophy. The hack is awful, but it doesn’t take that much code (which can even be generated making it almost as concise as using sealed). It provides an ironclad guarantee that the library is not abused. It is quite reasonable for a library to take relatively small measures like that to prevent abuse.

IMO, this is the most responsible approach to library development. Yes, protocols are also about semantics and users should read documentation and use them accordingly, but that doesn’t mean the language shouldn’t provide tools to prevent abuse.

Further, the reality is that not everyone bothers to read documentation. We shouldn’t modify our designs around that bad behavior, but we should still strive to offer the best experience possible for all Swift users. Every abuse a library is able to prevent by construction is a win in this regard. (I am not speaking of abuse with malicious intent here, primarily abuse by accident and / or naivety).

5 Likes

I don't know about the actual use cases for sealed, but I guess hiding a protocol might be suffient in some situations.
In this context, the meaning of "hiding" would be using an internal protocol in a public method, with the effect that the module would expose overloads of that method for each type that conforms to the protocol (while keeping the connecting protocol secret).
This woudn't need new keywords, and afaics has no impact on backwards compatibility.

Agree with this philosophy generally, but if this is to be the tentpole advantage of sealed over the status quo then we are modifying our designs--of Swift itself no less!--around bad behavior.

This is not how I see it. The tentpole advantage is that it allows us to express our design in the language itself. Preventing conformances outside the module is a relatively common design constraint and it is unfortunate that it cannot be expressed in the language.

Expressing our designs clearly in the language is an important goal IMO. This is one of the advantages of Swift-style protocols over the duck-typed generics that some languages have. If we want libraries to be able to define their public contracts as clearly as possible in the language then we need to support features like this and not just fall back on RTFM.

5 Likes

This would not meet any of the use cases I have. The protocol and conformances to it must be visible to users of the library. They are just not allowed to add new conformacnes.

The syntactic requirements of a protocol have always been expressible in the language itself, but the semantic requirements are not and largely cannot be. It is essential to a proper conformance to "RTFM"--that is not a fallback--and I would say that anything that gives a user the impression that they don't need to "RTFM" to understand the contract is not only a non-goal but an anti-goal.

1 Like

Another thing to consider though is what guarantee the "sealed" invariant gains you as a programmer developing your library. As proposed, "sealed" alone doesn't give the compiler enough information to statically verify that you've exhaustively switched over conforming types, so the only statically total way to interact with a protocol interface would still be through its requirements. Therefore, while you would be getting a guarantee that code outside your module doesn't add conformances, you'd still have to rely on dynamic assertions and/or auditing outside of what the compiler checks to ensure that you're exhaustively handling your own cases in switches or other non-protocol-requirement-dispatch mechanisms. OTOH, if you factor your code such that you always dispatch your type-specific logic through the protocol's requirements, then the "no outside conformances" constraint may not really buy you anything—it may not always be practical for external types to conform, but if they manage to, there's no direct harm in their doing so.

If part of the goal of this proposal is to allow exhaustive enumeration of conformances outside of protocol requirement dispatch, then an alternative design might need to be explored. However, having a requirement like "private and local classes are not allowed to conform if a protocol is sealed" would be really weird. In languages with "case classes" to represent closed sums as class hierarchies, for instance, all of the cases must be defined together with the parent class; maybe a sealed protocol could have a way to enumerate the conformances up front as part of the protocol definition?

There are a lot of semantics expressed in the type system and more semantics implied by the names we choose for our members. The fact that we can’t express all semantics directly in the language is not an excuse for supporting the ability to express the semantics that are reasonable to support in the language. I hope we can agree on this point and the debate is focused around what is “reasonable”.

IMO, there is very little burden on users who don’t use this feature, only benefits for users who need it, and it sounds like @Karl has an implementation. The staunch opposition that arose today is rather confusing, surprising and disappointing to me.

I think this would be acceptable. It’s inline with the spirit of most or all of the use cases I know of. One note: I think only the conformance declaration itself should be required to be stated with the protocol declaration while members implementing requirements could still be declared elsewhere. The same mechanism could also be used for internal and fileprivate protocols.

Sure, there are semantics expressed in the type system: that's precisely why protocols aren't just bags of syntax. Conforming to Error expresses semantics. And yes, names can imply semantics, and a name that implies unintended semantics would be rightfully judged a poor name. And yes, names are important. (But the compiler does not enforce the semantics of names. It's a pretty thin definition of "in the language" if the semantics implied in a method name is "in" but the semantics stated immediately above the declaration in a doc comment is "out.")

The opposition, I think, is borne out of the clarity that comes with sufficient discussion. We have learned that some of the proposed compiler benefits are not realizable. We are confronted with the challenge that there's an unclear division of labor between this feature and enums.

Circling back, the question is: what does supporting the expression of these specific semantics in this compiler-enforceable way, at the cost of a new addition to the language, gain users that isn't possible now? I would disagree that this is an end unto itself, and it's not clear to me anymore that there is much gained besides.