[Pitch] Rethrowing protocol conformances

Hey all,

Here's a pitch for a feature that @Philippe_Hausler, @Joe_Groff, @Tony_Parker and I have been thinking about to make rethrows also work for protocol conformances. It's motivated by some of the concurrency work, but is separable and useful on its own. Philippe has been prototyping it in the compiler so we have a fairly good sense that this is implementable and should make rethrows more useful. I'll keep an up-to-date version of this proposal here.

Rethrowing protocol conformances

Swift's rethrows feature allows a function to specify that it will throw only in narrow circumstances where it is calling a user-provided function argument. For example, the Sequence.map operation is rethrows:

extension Sequence {
  func map<Transformed>(transform: (Element) throws -> Transformed) rethrows -> [Transformed] {
    var result = [Transformed]()
    var iterator = self.makeIterator()
    while let element = iterator.next() {
      result.append(try transform(element))   // note: this is the only `try`!
    }
    return result
  }
}

When calling Sequence.map, the map is only considered to throw when the function argument passed to the transform parameter can throw:

_ = [1, 2, 3].map { String($0) }  // okay: map does not throw because the closure does not throw
_ = try ["1", "2", "3"].map { (string: String) -> Int in
  guard let result = Int(string) else { throw IntParseError(string) }
  return result
} // okay: map can throw because the closure can throw

Swift's rethrows is effective so long as the user-provided operations that potentially throw errors are passed via function parameters. However, many user-provided operations are provided indirectly through protocol conformances. This proposal extends the notion of rethrows to protocol conformances.

Motivation

Let's consider variant of map where getting the next element from an iterator could fail. This would allow us to (for example) better model input streams as sequences, because reading from an input stream can fail. The existing Sequence protocol doesn't support this, so we'll invent a new FailableSequence protocol and its corresponding iterator protocol:

protocol FailableIterator {
  associatedtype Element

  mutating func next() throws -> Element?
}

protocol FailableSequence {
  associatedtype Iterator: FailableIterator
  typealias Element = Iterator.Element

  func makeIterator() -> Iterator
}

Now, a sequence type that (say) reads lines from a terminal can conform to FailableSequence:

/// Read a line from standard input
func readline() throws -> String? { ... }

struct ReadLine: FailableSequence {
  struct Iterator: FailableIterator {
    typealias Element = String

    mutating func next() throws -> String? {
      return try readline()
    }
  }
  
  func makeIterator() -> Iterator { ... }
}

Note that types that conform to IteratorProtocol also satisfy the requirements of FailableIterator, and types that conform to Sequence also satisfy the requirements of FailableSequence, because a non-throwing method can satisfy a corresponding requirement that throws. For example, we can make Array conform to FailableSequence:

extension IndexingIterator: FailableIterator { } // Array.Iterator is an IndexingIterator
extension Array: FailableSequence { }

This makes FailableSequence more general than the existing Sequence. However, using arrays via FailableSequence is likely to be frustrating, because one has to assume that every operation that traverses a failable sequence can throw, even though traversing an array never throws. Historically, this is one of the reasons why Sequence doesn't support failable sequences.

Let's try to implement a form of map (call it map2) on FailableSequence:

extension FailableSequence {
  func map2<Transformed>(transform: (Element) throws -> Transformed) rethrows -> [Transformed] {
    var result = [Transformed]()
    var iterator = self.makeIterator()
    while let element = try iterator.next() { // error: call can throw, but the error is not handled; a function declared 'rethrows' may only throw if its parameter does
      result.append(try transform(element))
    }
    return result
  }
}

The error produced is correct: in a rethrows function, the only permissible throwing operations are calls to function parameters that are potentially-throwing functions. That guarantees, statically, that when the argument for transform is non-throwing, map2 will never throw. By having the call to next() be potentially throwing, we violate that guarantee because an error could be thrown from next() (and out through map2) even in cases where the transform argument is non-throwing.

This proposal seeks to make the above definition of map2 well-formed, and have a call to map2 be consider throwing when either the transform argument or the iterator's next operation is throwing. For example:

_ = try ReadLine().map2 { $0 + "!" } // okay: map2 can throw because ReadLine.Iterator's next() throws
_ = try ReadLine().map2 { (string: String) -> Int in
  guard let result = Int(string) else { throw IntParseError(string) }
  return result
} // okay: map2 can throw because the closure can throw and ReadLine.Iterator's next() throws

_ = [1, 2, 3].map2 { String($0) }  // okay: map2 does not throw because the closure does not throw and Array.Iterator's next() does not throw
_ = try ["1", "2", "3"].map2 { (string: String) -> Int in
  guard let result = Int(string) else { throw IntParseError(string) }
  return result
} // okay: map2 can throw because the closure can throw

Proposed solution

The proposed solution is to consider protocol conformances to be a source of throwing behavior for rethrows, allowing rethrows to reason about the throwing behavior of user operations provided via protocol conformances.

Rethrows protocols

Rethrowing behavior for protocol conformances begins within the protocols themselves. A protocol requirement will be able to be marked as rethrows as follows:

protocol FailableIterator {
  associatedtype Element

  mutating func next() rethrows -> Element?
}

Such a protocol is called a rethrows protocol. When a type conforms to a rethrows protocol, Swift records whether the method used to satisfy the next() requirement was throwing or not. For example, the conformance of ReadLine.Iterator to FailableIterator throws (because ReadLine.Iterator.next() is marked throws), but the conformance of IndexingIterator to Iterator does not (because IndexingIterator.next() is not marked throws).

Rethrows checking with protocol conformances

Any generic requirement that requires conformance to a rethrows protocol becomes part of rethrows checking. For example, consider a simple wrapper over FailableIterator's next():

func getNext<Iterator: FailableIterator>(_ iterator: inout Iterator) rethrows -> Iterator.Element? {
  return try iterator.next()
}

getNext(_:) is only using rethrows operations from FailableIterator, so it is well-formed: it only throws when the call to next() throws. Therefore, calls to getNext(_:) will throw only when the conformance provided for the Iterator: FailableIterator requirement throws. For example:

func testGetNext<C: Collection>(
    indexing: inout IndexingIterator<C>, readline: inout ReadLine.Iterator
) throws {
  getNext(&indexing).    // okay: conformance of IndexingIterator: FailableIterator does not throw, so call does not throw
  try getNext(&readline) // okay: conformance of ReadLine.Iterator: FailableIterator does throw, so call throws
}

The definition of a rethrows protocol is transitive: if any generic requirement within the definition of the protocol (e.g., via an inherited protocol or a requirement on an associated type) is a rethrows protocol, then at protocol is also a rethrows protocol. Therefore, FailableSequence is a rethrows protocol because its Iterator type must conform to the rethrows protocol FailableIterator:

protocol FailableSequence {  // implicitly a rethrows protocol
  associatedtype Iterator: FailableIterator   // generic requirement on the rethrows protocol FailableIterator
  typealias Element = Iterator.Element

  func makeIterator() -> Iterator
}

This definition is what permits our map2 example to become well-formed:

extension FailableSequence {
  func map2<Transformed>(transform: (Element) throws -> Transformed) rethrows -> [Transformed] {
    var result = [Transformed]()
    var iterator = self.makeIterator()
    while let element = try iterator.next() { // okay: FailableIterator.next() is rethrows
      result.append(try transform(element))   // okay: transform is throws
    }
    return result
  }
}

Defining a method in an extension of the FailableSequence protocol implies a requirement Self: FailableSequence, and FailableSequence is a rethrows protocol. In the body of map2, both calls to potentially-throwing operations are covered by rethrows checking: the call to iterator.next() throws when Self.Iterator's conformance to FailableIterator throws, and the call to transform throws when the transform argument throws.

Conditionally-rethrowing conformances

Both IndexingIterator and ReadLine.Iterator are simple in the sense that the next() either can't throw or can throw, and they don't depend on anything else. However, consider an adapter over another FailableIterator that does nothing but pass-through its next() to the underlying iterator:

struct FailableAdapter<Wrapped: FailableIterator>: FailableIterator {
  typealias Element = Wrapper.Element
  
  var wrapped: Wrapped
  
  mutating func next() rethrows -> Element? {
    return try wrapped.next()
  }
}

The FailableAdapter.next() function is permitted to be rethrows because it only calls into rethrows protocol requirements. Hence, calling next() on FailableAdapter<IndexingIterator<[Int]>> will not throw, but calling next() on FailableAdapter<ReadLine.Iterator> can throw.

This rethrows logic extends to the conformance of FailableAdapter to FailableIterator: because the next() method satisfying the rethrows requirement is itself rethrows, the conformance is throwing when the conformance of Wrapper: FailableIterator is throwing. This allows rethrowing-ness to compose, e.g.,

func adapted(arrayIterator: IndexingIterator<[Int]>, readLineIterator: ReadLine.Iterator) {
  var adaptedArrayIterator = FailableAdapter(wrapped: arrayIterator)
  getNext(&adaptedArrayIterator) // okay: FailableAdapter<IndexingIterator<[Int]>>: FailableIterator does not throw

  var adaptedReadLineIterator = FailableAdapter(wrapped: readLineIterator)
  try getNext(&adaptedReadLineIterator) // okay: FailableAdapter<ReadLine.Iterator>: FailableIterator can throw
}

Open questions

  • Q: This is so cool. Can we fix Sequence?

    • A: Probably not directly, because of ABI. In theory we might be able to retroactively add FailableSequence as a protocol that Sequence inherits (ditto for FailableIterator and IteratorProtocol), with an additional rule that allows Sequence and IteratorProtocol to not be rethrowing protocols because they've restated non-throwing versions of the protocols they inherit. This needs more thought, but would solve a longstanding weakness in the Sequence protocol.
  • Q: Do we need to mark protocol requirements as rethrows? Why won't throws work?

    • A: It's possible that we could use throws as the indicator on protocol requirements, similar to what we for function parameters. That makes this potentially a source-breaking change, since there are existing protocol conformances that would become non-throwing and that could change the behavior of some existing rethrows operations.

    Doug

17 Likes

I agree with the motivation, but disagree with the proposed solution.
The solution I would want would be to use typed throws.
e.g:

protocol FailableIterator {
  associatedtype Element
  associatedtype Error

  mutating func next() throws<Self.Error> -> Element?
}

In addition, how does your proposal deal with preexisting rethrows protocol requirements?
e.g:

protocol Foo {
  func bar(_: () throws -> Int) rethrows -> Int
}
1 Like

I will let Doug tackle the typed errors case (that is a pretty big can of worms). But in the existing cases we can evaluate each term to the rethrow-ness of the parameter. This means that the current rethrows concept will work using the existing code-paths in the compiler to generate it. The difference is that now there are additional descent into types to determine if participants (including self) are contributing to the throwyness of the function.

I like the idea, however the word rethrows is not quite accurate here. Such a protocol requirement does not rethrow, but rather it is rethrowable.

Perhaps throws? might convey the intent better, since the requirement may optionally throw, at the behest of the conforming type.

2 Likes

Functions passed directly to a rethrows function are extremely likely to be called in its implementation. That is less likely to be the case with protocol requirements. I have some concern that the proposed approach is too infectious.

For example, because Iterator sits at the foundation of the collection hierarchy, if we ignore the ABI issue and assume next should be rethrows, all of the collection protocols would become rethrows protocols. Would we want every rethrows API constrained to a collection protocol to throw when the iterator’s next throws?

Maybe I just need more time to think this through, but it seems like there are probably some unfortunate consequences lurking.

4 Likes

Hmm... I understand it as indirect rethrows for now.

If we can append rethrows to protocol method without " 'rethrows' function must take a throwing function argument" error, like below looks so weird..

does that mean there's NO 'rethrows' function must take a throwing function argument restriction any more, or in protocol only?

Or maybe

is the correct usage pattern.

I think what makes me uncomfortable is that the consequences of making a protocol rethrowing are nonlocal and far reaching. This behavior is much more subtle and therefore harder to reason about than the local behavior we have for rethrows today. We should consider this change carefully.

7 Likes

Fixing the diagnostic would of course fix the wording there, it must either take a throwing function argument or another rethrowing type. (the diagnostic verbiage should be carefully crafted to that end).

This is similar to the concept of the closures and granting protocol adoption the ability to participate just like closures can. So the restriction is that there must be some sort of type that exists in the parameter list that is either a closure or a type that adopts a protocol that has a requirement that is rethrowing.

I understand the motivation and agree with much of the analysis here. However, I do think this open question is very much worth considering. Two points to consider:

(1)

From a semantic standpoint, the fact that non-throwing implementations are already considered to fulfill throwing requirements means that Swift already recognizes that throwing protocol requirements only throw if the implementations throw. This pitch is about extending such recognition to rethrows checking. But if we think about a protocol definition as declaring what's semantically required of an implementation, there's not really anything there that distinguishes current throwing requirements from these "rethrowing" requirements.

The issues outlined here are of-a-kind with previous work (unfortunately never landed) to allow methods with non-optional return type to fulfill method requirements with optional return type. In both cases, there would be potential source breakage involved in introducing such features now. Still, I hope we'd never have to contemplate inventing a "reoptional" facility just to express that in the language.

(2)

Unless I'm mistaken, only inventing a new indicator word (in the same vein as "reoptional") can actually prevent source breakage on introducing this feature. Using rethrows risks source breakage just as throws does, because both currently mean something in declarations of protocol requirements. Consider:

protocol P {
  func frobnicate<T>(_ transform: (Self) throws -> T) rethrows -> [T]
}

The rethrows here only means rethrows in the traditional sense. Just as using throws as the indicator would potentially affect conformances to protocols that have throwing requirements, using rethrows as the indicator would potentially affect conformances to protocols that have rethrowing requirements. Indeed, there'd be the problem now that users could conform to P by providing a frobnicate that doesn't only rethrow but can throw willy-nilly. (And this is not some esoteric, made-up function, of course: frobnicate is just a thinly disguised map.)

It is true that there are practical considerations regarding source compatibility. However, there are also practical solutions. For instance, I think we could enable this feature with throws as the indicator while minimizing source breakage by keeping around both the old and new rethrows checking. Where the new rethrows checking causes source code no longer to compile, fall back to the old rethrows checking.

8 Likes

My mental model rethrows is that it is a convenience for the fact that throws is a subtype of non-throws functions. Reading the documentation of rethrows is straightforward when a rethrows function can throw [1]. Adding to the rethrows rules makes it harder to understand specially since adding a protocol can convert a non throwing function into throwing which can cause a cascading effect. Is rethrows the best tool for this? I feel that a new concept like @autothrow is warranted.

protocol FailableIterator {
  associatedtype Element

  mutating func next() -> @autothrows Element?
}

This would allow a type to implement next() as:
next() -> throws Element or
next() -> Element?

maybe even

next() -> Result<Element,Error>

[1] https://github.com/apple/swift/blob/main/docs/ErrorHandling.rst#rethrows

I’m with Xi on this. Supporting a notion of protocol requirements which may or may not throw, and being able to make functions using those requirements throw only if the conformance in question does, is a great feature I’d love to have—especially if we can adopt it without breaking ABI compatibility*. Reusing the rethrows keyword in protocol requirements to also specify conditionally-throwing requirements seems too clever by half.


As for IteratorProtocol, can we introduce a conditionally-throwing replacement for next(), probably under a different name, and then define default implementations for each that call the other? for can then use the old entry point for known-non-throwing conformances in backward deployment, and the new one otherwise.

This would have the same problem with mutually recursive default implementations as hashValue and hash(into:), but we really ought to productize a solution for that one day.

2 Likes

There might be an answer there, but you have to be explicit at every use site for it to work. Let's try to implement my map2 example on top of your FailableIterator. The same code does not work:

extension FailableSequence {
  func map2<Transformed>(transform: (Element) throws -> Transformed) rethrows -> [Transformed] {
    var result = [Transformed]()
    var iterator = self.makeIterator()
    while let element = try iterator.next() { // error: call can throw, but the error 'Self.Iterator.Error` is not handled; a function declared 'rethrows' may only throw if its parameter does
      result.append(try transform(element))
    }
    return result
  }
}

There is no direct solution for this in the current typed throws pitch. One can mimic the expected behavior with overloading:

extension FailableSequence {
  // Always throwing (not rethrows) because Self.Iterator.makeIterator can throw 
  func map2<Transformed>(transform: (Element) throws -> Transformed) throws -> [Transformed] {
    // implementation ..
        var result = [Transformed]()
    var iterator = self.makeIterator()
    while let element = try iterator.next() {
      result.append(try transform(element))
    }
    return result
  }
}

extension FailableSequence where Self.Iterator.Error == Never {
  // Throws when the function argument throws
  func map2<Transformed>(transform: (Element) throws -> Transformed) rethrows -> [Transformed] {
    // implementation ..
        var result = [Transformed]()
    var iterator = self.makeIterator()
    while let element = try iterator.next() {
      result.append(try transform(element))
    }
    return result
  }
}

One could perhaps extend the typed-throws proposal to have rethrows in addition to a throws clause, with a declaration like this:

extension FailableSequence {
  func map2<Transformed>(transform: (Element) throws -> Transformed) throws Self.Iterator.Error rethrows -> [Transformed] { ... }
}

Or extend the typed-throws proposal to put typed throws into function parameters as well (with inference from closure arguments, which is tricky to implement today) and allow functions to specify multiple typed throws:

extension FailableSequence {
  func map2<Transformed, TransformError: Error>(transform: (Element) throws(TransformError) -> Transformed) throws(Self.Iterator.Error, TransformError) -> [Transformed] { ... }
}

Note that you'll hit all of these same problems if you want to write zip2 with typed-throws.

That last one is arguably the best answer for a typed-throws solution because it subsumes rethrows entirely within a more-general framework. But here's the thing---the rethrows solution I wrote is more concise and far easier to get right. If we implement the proposed rethrows, it could later become syntactic sugar for typed throws. I think that should be an explicit goal of any typed-throws design.

Doug

Yes.

You're absolutely right. Using rethrows is not eliminating potential source breakage here.

I guess this is our enduring lesson from SE-0286, that we can implement both ways ;).

Yeah, I agree with you and Brent that throws is the better indicator here. It matches how function parameters work with rethrows, and applies the new rethrows checking behavior to the right places. I suspect that the source impact will be low in practice, because such a change can only turn throwing calls into non-throwing calls, and the diagnostic you get for having a try with no throwing calls in the expression is a warning:

warning: no calls to throwing functions occur within 'try' expression

Thank you!

Doug

3 Likes

Here's the design I'm thinking about:

protocol FailableIterator {
  associatedtype Element

  mutating func next() throws -> Element?
}

protocol IteratorProtocol: FailableIterator {
  mutating func next() -> Element?
}

We could consider IteratorProtocol.next() to override FailableIterator.next() (technically, the implementation already does this), and say that, because IteratorProtocol overrides all of the throwing requirements of FailableIterator with non-throwing versions, the "throwing-ness" of a conformance to FailableIterator isn't propagated to IteratorProtocol.

Sequence would be built on top of FailableSequence, e.g.,

protocol FailableSequence {
  associatedtype Element
  associatedtype Iterator: FailableIterator where Self.Element == Self.Iterator.Element

  func makeIterator() -> Iterator
}

protocol Sequence: FailableSequence where Self.Iterator: IteratorProtocol {
}

although presumably there are no throwing requirements in FailableSequence. The cool thing about this approach is that you don't break code built on top of IteratorProtocol and Sequence, but now you can define rethrows-capable functions on top of FailableIterator and FailableSequence.

I also believe we left enough wiggle room in the ABI to make it possible to stage this in, although the compiler doesn't necessarily know how to correctly generate such code.

Doug

Would you do this even if ABI and source compatibility were not an issue? This doesn't seem like something that should necessitate parallel hierarchies.

It looks like what you want is something like a constraint along the lines of T.Iterator: nothrow. This is a hypothetical constraint that allows a specific conformance to be constrained to be non-throwing. When all available conformances should be constrained to be non-throwing then T: nothrow could be used as a shorthand.

I would feel more comfortable about this proposal if it included a constraint like this as it gives generic code the ability to narrow the conformances that rethrows considers to the ones that are actually relevant. Most importantly, it could be used on an ad-hoc basis and in a way that doesn't impact conforming types (something that isn't possible with the refining protocol approach).

The constraint would have to mention both the protocol and conforming type, e.g., Self.Iterator: nothrow FailableIterator, but yes, that's a more explicit way of doing what I'm doing implicitly.

If we had that and were doing a Sequence hierarchy without backward-compatibility concerns, I'd probably want something like:

protocol IteratorProtocol {
  associatedtype Element
  mutating func next() throws -> Element?
}

protocol Sequence {
  associatedtype Iterator: IteratorProtocol
  typealias Element = Iterator.Element
  func makeIterator() -> Iterator
}

protocol Collection: Sequence where Self.Iterator: nothrow IteratorProtocol {
  // ...
}

So Sequences can fail, but once you get to Collection you always have non-throwing access to the elements.

Doug

[EDIT: Fixed a missing throws, the whole point of the example]

Right. I think I wasn't clear - I was giving an example based on a hypothetical:

protocol Iterator {
  associatedtype Element

  mutating func next() throws -> Element?
}

so the idea with T.Iterator: nothrow was that T's Iterator conformance is non-throwing. I think you're suggesting an alternative T: nothrow Iterator syntax which would also be fine (syntax is not my concern right now).

Did you intend to make that next a rethrows requirement?

The nothrow constraint addresses the concerns I have nicely. Is it something that's feasible to include in an updated proposal?

I'm not trying to nit-pick syntax, but trying to ensure that the right information is there: T.Iterator: nothrow syntax only mentions a type (T.Iterator). It does not mention the protocol whose conformance needs to be non-throwing. That's why I used Self.Iterator: nothrow IteratorProtocol to specify both type and protocol, so we know which specific conformance must be non-throwing.

Whoops, I went back and fixed it for posterity.

Yes.

Doug

Right, my mistake. I meant to write something like T.Iterator.IteratorProtocol: nothrow but your syntax was better anyway.

Awesome, thanks!

What would happen if I use the generic that are failable:

struct FailableJoinedSequence<S: FailableSequence> { ... }

Would/Should FJS.Iterator be able to keep track of the throwability of S.Iterator, and mark throwable accordingly?


From the github:

extension FailableSequence {
  func map2<Transformed>(transform: (Element) throws -> Transformed) rethrows -> [Transformed] {
    var result = [Transformed]()
    var iterator = self.makeIterator()
    while let element = try iterator.next() { // okay: FailableIterator.next() is rethrows
      result.append(try transform(element))   // okay: transform is throws
    }
    return result
  }
}

This is a very peculiar example. The throwability comes from Self.Iterator which is never mentioned in the function signature. This means that rethrows reaches far and deep into Self. I feel that it could be hard to keep track of, especially since there's no marking in FailableSequence that the throwability follows into Iterator.


What if instead of per-function marking, we do a per-protocol marking. It doesn't seem to affect much of the functionality. Further, I think it's better to have on-the-fly rethrowability by marking throws on the conformance.

// Normal protocol
protocol FailableIteratorProtocol {
  associatedtype Value
  func next() throws -> Value
}

protocol FailableSequence {
  // `FailableSequence` throws if `Iterator.next` throws
  associatedtype Iterator: throws FailableIteratorProtocol

  func makeIterator() -> Iterator
}

// Throws if the conformance of `S` to `FailableSequence` throws
func map2<S: throws FailableSequence>(_: S) rethrows {
  ...
}

which would make it much more explicit at function declaration which of the conformances contribute to the rethrow. It would also allow for future directions like nothrows:

// `S`'s conformance to `FS` can't have throwing function
func map3<S: nothrows FailableSequence>(_: S) { ... }

or rethrowable generics:

// S now contributes to `rethrows`
struct JoinedSequence<S: throws FailableSequence> {
  struct Iterator: FailableIterator {
    func next() rethrows -> S.Element { ... }
  }
}
Terms of Service

Privacy Policy

Cookie Policy