It would not be any different, nor would my opinion of it: in that hypothetical, I would be then be writing the same objection with respect to recoveryNextSpan() that we should not do that, because it'd end up with allowing iteration of an iterable in two different ways.
The salient difference here is: the behavior which I would object to then in that hypothetical, I have to object to now, because with the non-specification approach it is enabled here, by this protocol, in the form of nextSpan()—it's not some other same-name shadowing method but this specific protocol requirement that is doing the thing I'm concerned about.
I'm not sure I understand this point: that you'd agree if it's a different prefix but not (as here) with a different suffix? Consider the degenerate case of an Iterable which immediately throws but is recoverable: in the generic context, no iterations occur; but in the concrete case, the Iterable isn't empty. Isn't that...bad?
We are getting into the philosophical, but my view here is that under the proposed semantics, once an iterator has thrown, there is no protocol requirement nextSpan(). Any call to nextSpan() is, definitionally, outside the semantics of the protocol, and it is no different than if you were calling a differently-named method on the same value: you are allowed no expectations about what the behavior might be. Iteration quaIterable has ended and if you call nextSpan() you are doing something else.
My view is that, functionally, a concrete type saying "I can recover after throwing, and you can achieve this recovery by calling nextSpan()" has exactly the same bearing on the semantics of Iterable as does a concrete type saying "I can recover after throwing, and you can achieve this recovering by calling recoveryNextSpan()". So even a fully-specified throwing behavior for Iterable enables an (IMO) equivalent shape of misbehavior.
The Iterable is empty in both cases: for x in recoverableIterable as RecoverableIterable { ... } will produce the same values as for x in recoverableIterable as Iterable { ... }. The "elements of an Iterable are definitionally those that are produced by following the semantics of the Iterable protocol.
That one can perform additional operations on concrete values which are not specified in the semantics or syntax of a particular protocol to which that value conforms is not surprising. One does not say that Array produces "different values" in the concrete case compared to Sequence simply because you can call append(_:) before iteration in the Array case!
We are definitely getting into the philosophical, but I think it's clarifying. To me your example is inapposite because there's not a notion of "iterating" an Array that involves calling append(_:)—which, as you well know of course, notionally gives you a different (mutated) value.
My point in bringing up the example of iterating UTF8 versus UTF16, or equating NaNs, was to show that where there is a well understood operation represented by a protocol, our philosophy has been to restrict each type to performing that operation in one way. To my mind it falls under the umbrella of "conforming to a protocol in one way." We make it possible to perform variations on the operation by ergonomic conversions to other types, views, slices, etc. that then have their own conformance.
Put another way, our precedent says that given a type T: Fooable, we don't have users reason about "foo-ing" a T-qua-Fooable in contradistinction to "foo-ing" a T-qua-T. If T: Fooable, then we stipulate (by choice, as a policy) that they are always one and the same. If there are two ways to "foo," we offer T.AlterativeView: Fooable or Wrapper<T>: Fooable or something with the alternative behavior.
Right, and my position would be that the type should not do that and the stdlib should not be an enabler of it. Since we don't contort stdlib design around stopping end users from doing things with their types, the first part is irrelevant to the proposal. But the second part is under our control, in the form of "fully-specified throwing behavior."
Indeed, nor do I expect users to need to reason about this for Iterable. Adopting this proposal would be to say that iteration is the semantics specified by Iterable, and most users will interact with iteration via for-in loops (which behave identically in concrete or generic contexts) and abstractions built on top of Iterable (which behave identically when called in concrete or generic contexts). Manual juggling of IterableIterators is a much more niche use case and I think it is reasonable to expect users to be familiar with the semantics of IterableIteratorProtocol if they're delving into the details like that!
I don't find the examples of UTF8 or NaNs compelling because the purported equivalences you're drawing are talking about situations which would produce different values from the same sequence of operations. But for Iterable we are talking about different sequences of operations (namely, calling nextSpan() a greater number of times than is permitted by Iterable). It is not meaningful to ask whether nextSpan() from a concrete context produces the 'same value' as nextSpan() from a generic context after throwing because under the semantics of Iterable, "the result of nextSpan() after throwing" is not well-defined.
Sure: for a hypothetical value that throws on the third call, we can say that calling nextSpan() 0-3 times is the operation we regard as “iterating” while calling nextSpan() 4 times is “not-iterating.”
But I don’t think that’s a very friendly design—and, if we agree that this is “misbehavior” even if supported, for what useful behavior are we making this trade-off?
I think much more understandable to preserve the notion that calling nextSpan() (or in the degenerate case, not calling it at all) is “iterating” regardless of the state of the (borrowing) iterator.
Well, IMO the primary notion of iterating should be "what happens when you do a for-in loop", and I'd quibble that the the split ought to be characterized as "calling nextSpan() after throwing" vs. "not calling nextSpan() after throwing" rather than "3" vs. "4".
I was also adopting your framing of "misbehavior", I don't actually view the supposedly offending behavior here as poorly as you do, I think. It would certainly be a bit silly to deliberately design such a type, but I expect that most conformers will take this as license to say "Oh good, once I've thrown I don't need to do any special bookkeeping to ensure I enter a terminal state, I can just be done", and probably have their nextSpan() merely inherit the property of "don't call again after it's thrown".
If we can agree that this is what we want, then we ought to codify that as the semantics: after throwing, calling nextSpan has no specific behavior required of it, but you’re “done”—that is: it will never return a non-empty Span.
Even that potentially requires some bookkeeping, though—if your error was ephemeral you would have to 'remember' you threw an error so that your normal, happy path doesn't potentially return what would have been the next valid span but for the thrown error.
I think maybe the way to think about it is that throwing from nextSpaninvalidates the iterator, just as mutating a collection invalidates its indices. Maybe using an invalidated instance does something sensible, maybe it traps, maybe it does something weird. Maybe there are even fairly complicated conditions for what's permitted, like array indices still working if the mutation is after the index! The language does not require conforming types to proactively catch uses of an invalid instance, or indeed to do anything in particular at all (besides maintain safety).
That's fair. Two points: (a) is it unreasonable to demand that bookkeeping? it'd not be applicable to most conforming types; (b) even if unreasonable, can we do apply an approach as follows:
Most (all?) collections' indices, once mutated, can't be correctly used in any context and not just the generic one. For example, even if I'm using a zero-based Int-indexed type, 3 stops indexing into the same element if I insert or remove after the first element.
By contrast, in the unspecified-behavior-by-design approach here, we're explicitly leaving open the option of conformers for which nextSpan use after throw is valid in the concrete context: can we simply affirmatively foreclose that?
That is: actually state as a requirement that throwing from nextSpan invalidates the iterator--i.e., throw and you're done--such that no subsequent spans even if non-empty are valid elements? This would forbid semantically, though not by compiler enforcement, a resumable type conforming to Iterable.
I don’t think this example properly illustrates indices becoming invalid, at least not in the sense that, say String.Index indices may become invalid after a mutation.
The Collection docs say:
Saved indices may become invalid as a result of mutating operations. For more information about index invalidation in mutable collections, see the reference for […] the specific type you’re using.
(emphasis added)
This seems entirely analogous: an IterableIteratormay become invalid after nextSpan() throws, for more information see the reference for the specific type you’re using.
I do not think it makes sense to say that Array indices become invalid after an insert(_:at:) operation. In fact, all valid indices before remain valid indices after by nature of the documented behavior of Array. But one cannot assume this behavior from a generic context.
I’m also curious to see some examples. The only case I can think of is perhaps a lazy map or filter sequence, but Iterable seems like the wrong choice for this, because the lazy sequence’s iterator would need to have its own internal buffer instead of just returning values when read. So if Iterable is mostly intended to be used with collections which store their elements, under what circumstances would iteration throw an error?
"Indubitably, IterableIterator is an irritating, ignoble, insipid, idiosyncratic identity," icily intoned the irritable Internet iconoclasts, indignantly. "Ick!"
You could stash the rest of a partly-used span from the underlying iterator in a stored property, right? Because if so, you could at least implement a lazy filter with no element storage of its own, and it would be reasonable to want that to throw.
I want to be able to write this in non-allocating embedded swift:
// This function recursively maps in a MachO file and all its dependencies
func mapMachO(path: ByteString) throws(MachOError) {
let mapping = try MmapBuffer(path)
let macho: MachO(mapping.bytes)
// parsing every element of a dylib is expensive as it requires faulting in
// many pages to validate portions of the file we may not need, so we validate
// as we use it, which means anything we iterate can potentially fail
for try dependent in macho.linkedDylibs {
// In a real loader we would check if the library was already loaded, handle
// weak libs, and do a lot of other logic here, but ultimately call
try load(path: dependent.path)
}
}
In order for the above to work linkedDylibs needs to be Iterable, and we can hit parse errors as we iterate over it. We can use a result type, but it means we are going to be doing that in every single for loop we write.
I agree throwing is not generally useful for the spans inside of containers, but Iterable is currently the only viable path to getting for loops to work in non-allocating embedded mode, and when used in that context it seems quite reasonable to decode into an internal buffer and return it. I have implemented a fairly complete MachO parser using the proposal and it uses that pattern quite a bit.
As I mentioned earlier in the thread I think it is probably unfortunate that one proposal is trying to satisfy both the needs of high performance collection iteration AND making for loops work in a limited language mode. But I also suspect that (Sequence not withstanding) adding more distinct protocols that interact with the front end to desugar for is probably also not particularly desirable (there are many people on this thread who know far more about that than I do, so please tell me if I am incorrect). That's fine, and the protocol should bias towards its primary use...
I don't mind that I have to implement nextSpan() despite the fact that next() would be more convenient for me. I am happy to handle that extra complexity in the implementation of the parser, but I really want the interface to be ergonomic, and that means good error handling. Making every call site use return types feels like a lot of added ceremony. If we really are going to use the single protocol to satisfy both uses I think error handling is a fairly reasonable affordance for non-allocating embedded use case (but I admit I am likely biased here).
I've had a lot of success using Swift's ownership features to model SQLite's C API in Swift nearly 1:1 but statically safe. For example, stepping a SQLiteStatement returns a Row with an exclusive access lifetime dependency on the statement, which statically guarantees you can only have one per statement at a time that can't overlap with calls to API like reset or finalize. I would love for SQLiteStatement to conform to Iterable[1], not just for the ergonomic for-in syntax, but because generic algorithms like map/filter/reduce make sense for working with streams of rows. And being IO, naturally all of this can throw errors.
With Iterable having the iterator returned with a borrowed lifetime this would probably require some intermediate Rows cursor type that is returned with an exclusive lifetime dependency ↩︎
I’ve been thinking about the end-of-iteration and throwing rules, and about how much of iteration semantics still rely on convention.
Returning an empty Span signals the end of iteration, and the rule that all subsequent calls must also return an empty Span gives that signal predictable semantics. But this is still a convention that implementations have to uphold, rather than something the type system enforces.
I wonder whether Swift’s ownership features could make the iteration state more explicit. For example, very roughly:
protocol Iterator {
func startIteration() -> consuming Iteration
mutating func next(
for iteration: consuming Iteration
) -> consuming Result<(Span<Element>, Iteration), EndOfIteration>
consuming func finishIteration(_ token: consuming EndOfIteration)
}
struct Iteration: ~Copyable {
// private init; only the iterator can produce one
}
struct EndOfIteration: ~Copyable {
// private init; only `next` can produce one
}
The important part is not this exact spelling, but the shape:
next consumes the current iteration token, so the same logical iteration step cannot be reused.
If iteration can continue, next returns both the produced span and a fresh Iteration token.
If iteration is finished, next returns an EndOfIteration token instead of another Iteration, so the caller no longer has the capability required to call next again.
finishIteration consumes the end token and the iterator, making the “finished” state explicit, discarding the iterator as an effect.
This would make it impossible, by construction, to call next again after end-of-iteration, instead of relying on the convention that an empty span must keep being returned forever.
I realize this is a much more complicated model than the proposal’s current API, and it may be too heavy for the common case or for for-in lowering. But I thought it was worth mentioning because the ownership system seems capable of expressing some of these iterator-state rules directly. It may be useful either as a future direction, or as a lens for evaluating which parts of iteration semantics should remain convention-based versus statically enforced.