If the rule is that lifetime ends at the end of scope if not consumed, then any intervening code can only shorten (never lengthen) that lifetime; this would be strictly more "reliable" (if that is the goal) than lifetime ending after last use, because then as you say editing the code could silently either shorten or lengthen lifetime.
That consuming a noncopyable value will shorten lifetime is, well, inherent to what consuming means; that it can also lengthen lifetime would in my view be suboptimal.
What I meant is that consuming could end up moving the value to a variable not bound by the current scope. Maybe it'll store it in a class or even in a global variable. The lifetime will then be longer. But in general I expect most unintentional consumptions would shorten the lifetime.
Edit: it looks like I'm talking about the lifetime of the value while you're talking about the lifetime of the local variable holding that value. What I'm really interested in when the deinit is called.
This is the behavior I would expect. Shortening lifetimes can cause benign ordering dependencies to suddenly become fatal errors when refactoring lengthens the lifespan of a variable.
Could you provide an example where you think this might be a problem specifically for a noncopyable type? (Such an example would help a lot; I've been trying and failing to come up with such.)
I think that it's easy to think of things when there's global state involved, but I would classify them as bad habits. For instance, if autoreleasepool was implemented as a (theoretical) non-copyable, non-movable type, it would almost certainly need to expire at the end of its scope to be useful. However, that's not how Swift encourages you to implement autoreleasepool.
That is not correct. Instead the language is using lexical lifetimes which explicitly does allow for lifetime shortening. We just do not shorten past deinit barriers and in specific cases (consider an escaping pointer), do not shorten at all.
IOW, we looked at the cases where the strict lifetime shrinking behavior was causing issues for users and discovered it was a very specific set of cases and changed the lifetime shrinking behavior to be conservative around those cases.
It's a nice consistent story to follow the same rules for all types, but it'd be interesting to consider those situations that led us to be more conservative in general and how likely it is they could apply to noncopyable types. Setting up a parent-child object relationship with a weak reference in one direction, for instance, isn't really an issue with single-ownership values (though not the only tricky case, to be fair).
Sure. I was just attempting to be clear about what is actually happening today with lexical lifetimes that way there isn't any confusion during the review since the review period remaining is short.
Off the top of my head, the main lexical lifetime conservative case that would apply to noncopyable types would be situations where one is exposing an unsafe pointer that depends on the noncopyable type being around. As an example just off the top of my head, consider a noncopyable type that has a class field and exposes an unsafe pointer into the class's memory. If the noncopyable type is the last owner of the class, we could with those semantics potentially introduce a use after free or require the user to add a fixLifetime or the like on the class field to ensure that we do not shorten too much.
One thing that I think would be helpful in this discussion is to lay out the options. In the following I am going to lay that out and put in italics my personal thoughts. First to discuss this, it is helpful to consider a piece of code:
// Definition
let x = NonCopyable()
if condition {
_ = consume x
// (1)
} else {
// (2)
callUnknownFunction()
// (3)
doSomethingElseThatCanBeProvenToNotHaveSideEffects()
// (4)
}
// (5)
Shorten lifetime to after last use, more eagerly than we do for copyable bindings.
This would mean that the lifetime of x would end at (1) and (2).
Michael: I don't think that this is really an option due to the ability to create an unsafe pointer potentially into a noncopyable type. In such a case, when the user has escaped the pointer, we really do not want the optimizer to do anything with shortening lifetimes along non-consuming paths since we may not be able to track the escaping value.
Shorten lifetime respecting destroy barriers ("lexical lifetimes") like we do copyable bindings today. NOTE: This is not the same semantically as lexical lifetimes. We are just using similar restrictions.
What this means is that when optimizing we would not shrink lifetimes at all if one escapes a pointer into a field of the noncopyable type or past things that could have synchronization like a memory barrier, access a lock, or sys call. In practice these are things like calls to unknown functions. So in this case, we would shrink the lifetime to (1), (3) since callUnknownFunction() calls some unknown system function and would block our ability to shorten the lifetime.
Michael: I prefer this one since it ensures that we have a single model for users to learn rather than adding a special rule to remember to the language. Swift IMO already has enough special cases, avoiding one here would make noncopyable types not a special case in terms of these semantics, reducing cognitive overhead by requiring users to remember another special rule
Non-RAII-scoped maximized lifetimes.
This is similar to scoped RAII, but instead of having strict RAII lifetimes, the compiler shrinks the lifetime of the value along non-consumed paths so that the lifetime of the value ends before any code reachable from a consume runs. In terms of the example above, this would mean the lifetime of x would end at (1) [the consuming pointer] and at (4) since that is the last point in the code that is along the non-consuming path that is not reachable from (1) (that is (5) is reachable from (1)).
By ensuring that program points along non-consumed paths are not reachable from consumed paths, we are able to ensure that:
We preserve the invariant that once consumed, a noncopyable typed value cannot in a strong way (including along other paths) affect program behavior.
The compiler does not need to implement the complexity of having to handle conditionally destroyed values since we still have the invariant from the first bullet point. Beyond just inserting the conditionally destroyed values themselves, conditionally destroyed values are not-SSA and would either need to be represented in memory or perhaps as optional values under the hood.
Michael: If we want to have more of an RAII like solution, I would prefer this approach. I still would prefer not to have any special rule so would prefer 2. But this at least avoids much compiler complexity and is in fact what we already do at -Onone for copyable types (so we would in a sense for noncopyable types have the -Onone behavior for copyable types both at -Onone and -O)
Strict RAII-scoped lifetimes.
This would mean that the compiler would be forced to strictly enforce lifetimes and thus jump through hoops to create conditional lifetimes. In the case of the example above, this would mean that the lifetime would end at (1) if we dynamically went along the true path and (5) if we went dynamically along the false path.
Michael: I really do not like this approach since we lose the strong invariant I mentioned above and also additionally introduce an extra rule. It would also introduce compiler complexity and make it harder to optimize code without much benefit over approach 3.
How do you escape that pointer in practice? Using withUnsafe*Pointer(to:_:), I would expect the object to stay alive at least until withUnsafePointer returns.
Speaking of UnsafePointer, the fact that noncopyable values can’t be used in generics means you can’t get pointers to them, right? (Aside from Unsafe[Mutable]RawPointer, of course.) This likely isn’t a huge deal in Swift code, but I imagine it would get frustrating when calling, for example, a C++ function using inout-to-UnsafePointer conversion.
For copyable values, opting into ownership control means:
consuming values are eagerly consumed.
borrowing values are associated with a lexical scope.
With consuming, there aren't any subtle do-what-I-mean rules (that probabilistically guess the author's intention). There is no association with a lexical scope, and "deinit barriers" are irrelevant. Lifetimes are fully, progressively optimized. That's necessary to optimize CoW and ARC. If the programmer escapes an unsafe pointer or weak reference, they need to borrow the parent value for the duration of that escape.
A noncopyable let should be defined as consuming since we can't mimic the behavior of a copyable let. I do not think that noncopyable lets should have anything to do with a lexical scopes or with "deinit barriers". There's no coherent way to do that. We could do a best effort "maximal" lifetime, but that would be highly confusing. It would mean that programmers can escape pointers, and sometimes it would just happen to work. Changing a consume in an unrelated part of the code could easily break their assumptions. The simple rule should be: use a borrow scope to protect escaping pointers. If they fail to do this, we want the code to break sooner rather than later.
The more interesting question is whether noncopyable types should have strict lifetimes vs. optimized lifetimes. For owned noncopyable values, an optimized lifetime can result in running a struct deinit early after dead code elimination. For borrowed noncopyable values, it means that a struct deinit can be reordered with other side effects inside the borrow scope.
I'm in favor of strict lifetimes for noncopyable types, particularly when the struct has a deinit, because this ensures consistent behavior independent of the optimizer. There is a compelling argument that programmers should not rely on struct deinit side effects, but not compelling enough to me.
Yeah, I think this is a really important point. deinit is meant for basic resource management, and it's okay for us to state that as a policy. with functions are the right way of doing things where it's semantically critical what code in inside the with, because deinit is inherently implicit, subtle, and imprecise to naive readers of code even if the language gives rigid semantics to when it'll actually be called.
Then in haste, I started to argue against my own design...
[Correction and clarification]:
let NonCopyable should have "maximal lifetimes" (which we proactively implemented a while back in copy propagation):
In Michael's example, this means that x is deinitialized at #1 and #4:
let x = NonCopyable()
if condition {
_ = consume x
// (1)
} else {
// (2)
callUnknownFunction()
// (3)
doSomethingElseThatCanBeProvenToNotHaveSideEffects()
// (4)
}
// (5)
There are three key reasons for this:
With NonCopyable, let is an alias for consuming. Special let semantics only make sense when the compiler can insert implicit copies. Noncopyable let should, therefore, have the same lifetime rules as Copyable consuming.
Copyable consuming variables are designed to have "maximal" lifetimes at -Onone. Their lifetime only ends on a path in which it was consumed. This is important for the debugging experience.
NonCopyable lifetimes should be the same at -Onone and -O. This way programmers can use a debugger to understand the behavior of a struct deinit.
The discussion above went off the rails with using "pointer escapes" as a reason for extending lifetimes. That's really what I wanted to correct in my first post. Attempt #2:
An implicit consume does not protect weak references or pointer escapes
Any model that relies on this would lead to confusing action-at-a-distance bugs. The example below is dangerous, and we want to add compiler diagnostics to make it an error:
[EDIT] We probably should not promote this last workaround because it can't be used consistently across both Copyable and NonCopyable types. We might even consider a rule that storage must be force-unwrapped with a borrow scope in all cases:
If this example used a class instead of a noncopyable struct, wouldn’t we have the same problem and tell ourselves that withExtendedLifetime is the solution for keeping the strong reference alive?
Right. withExtendedLifetime works in all cases. But if you need it in "normal" situations, I consider that a language or API bug. We should try to replace it with ownership controls, namely borrow scopes, which can be implicit when using with _read accessors.
With Copyable types, let conservatively handles cases like this as a concession to "legacy" code that doesn't explicitly guard weak refs and unsafe pointers. But that stands in the way of optimizing copies. consuming is the way for programmers to opt back into those optimizations.
Above, I mentioned that NonCopyable lifetimes can be controlled with an explicit consume:
} else {
_ = storage!
_ = consume container
}
But that does not work for Copyable types, whose lifetime can be optimized. Removing dead consumes in order to eliminate copies is a routine optimization. We could give an empty consume stronger semantics than a regular consume, but that would only lead to more confusion.
It's probably better to just have one simple rule for dependent values:
Any use of an escaped pointer or force-unwrapped weak reference must be within a borrow scope.
The language workgroup has decided to accept the proposal in principle, while kicking off a focused re-review on some of the details that have been expanded on and clarified by the proposal authors. There is a new review thread, please continue discussion there.