__shared should be considered inherently incorrect, as it is not actually stable.
Wait, does this mean you could basically opt into a form of Rust-style ownership management with that attribute? That’s very interesting.
I imagine production-ready Rust inter-op is not particularly high on the to-do list, but maybe it isn’t that far off.
I heard Joe say this a while back, and it remains the best explanation of Rust’s borrow check I’ve heard.
I had the same question: is @noImplicitCopy on a reference type essentially isomorphic (perhaps modulo some fussy details) to Rust’s borrow checker?
The pitch, indeed, sounds like a Christmas present. It is a delight to read.
Essentially, it seems to boil down to just two effects:
What I'm worried about is the amount of new keywords and attributes added to the language. For a newcomer to Swift this may be overwhelming. Can we consider reusing the same or similar terms for listed application, e.g. we have inout, then we can have in and out in some places, or similar?
My point is that in a bid to avoid confusion maybe it's worth to use the same language terms for different cases which are expected to behave in a similar way by a programmer.
do we have plan to reduce the atomic accesses of ARC when data will not cross threads or tasks?
I'm so excited to see this moving forward. I think it's one of the last few things that Swift lacks to make it powerful enough for world domination. ^^
I have to say that the only and very little experience I have with ownership control is mostly from Rust so please bare with me if I say nonsense things. Also sorry for the long post, seems like I got late to the party (>60 replies! nice!).
This seems sensible.
If I understand correctly this means we won't have to "strongify" weak bindings to keep them alive during the same lexical scope? It always felt too extreme to have to do that.
What is not really clear is if a binding is no longer used after the middle point of a scope, will it still be "released" at the end of it or it will be immediately after its final use? I have a disconnect between what assumptions can be made as a general rule vs. in specific code. And I wonder what Rust does here.
I've been following the other thread and naming aside this looks very nice. But I have a question about the array example...
I'm still not sure I understand what provokes a change on the behaviour.
Are you saying that right now there is no copy when sorting because the values lifetime ends before the sort call and so we get the good performance we want? and that with the changes that won't be true anymore and thus we will have to explicitly move ownership to get avoid the copy?
If so, even tho I understand the overall goals, this feels like a regression on the smartness of the compiler/language :(
This feels very good. I didn't know inits and setters used "consume" on their arguments already! Being able to improve appends on data strcutures is awesome. And I think that it seems to have the right level of progressive disclosure since this bit doesn't seem something we will need on "normal" Swift code.
It will be interesting to see the bikesheddingon names. I thought that Rust naming was quite clear but consuming seems like a good analogy too. :)
As explained, this is in place in the standard library already (and a recent post has shown how prevalent is in the community too) so I'm just glad it's becoming public.
One thing I'm wondering tho is if there is a way the language can expose a more consistent view on coroutines. If I'm not mistaken, won't we have now multiple "coroutine like" systems? async/await and read/modify. It feels like those features are in reality just one. (I remember the starting days of async discussions and how it was decided to give them more specific names, now it feels like a more general was a better idea). I'm thinking other places of the language/stdlib would benefit from it.
This looks quite powerful to control copies in our code. My comments are just about naming so not really important: the @ annotation for no implicit copy feels out of place with the rest of this roadmap. And the copy function in my eyes reminds me too much of copy constructors on other langs which is not really what this is.
This is awesome! Letting the compiler force that parms on certain APIs can escape is something that has been needed for a while. Right now you just have to rely on docs and making sure you don't do weird things.
I think is the place where more benefit we will get as "normal" Swift programmers, specially the inout bit.
One thing I would like to get clarified is: does this effectively let us take an inout variable to an element in an array and mutate it trough the binding?
var array = ...
array.mutatingFunc() // super nice to use!
var element = array[1]
element.mutatingFunc() // doesn't change the original element... sometimes annoying
New:
inout element = array[1]
element.mutatingFunc() // did it change the original element inside the array? <3
This is one of the pain points I have still with arrays being value types. Sometimes makes it very awkward to work with them cause you need to go trough the indices and subscripts to have in place mutations. One of the best things about swift is how value types have "safe" mutability so it would be awesome if we could get that with bindings too :)
(EDIT: I see same question above, sorry for the dupe)
With this said I love this direction. I'm just a bit unconvinced by the naming which I guess can be discussed on the respective proposals. I think that looking for a word that can be used in all places will be better. That ref is very weird looking.
Great plans! Whatever brings Swift more in the line of being able to write critical paths, like, audio, video, and handling sensors and actors (embedded) is extremely appreciated.
Thanks.
Really glad to see all this communicated together! The appeal of ARC is predictable performance after compilation, so any way to make compilation predictable is essential to the model.
I started off skeptical of this change, but after a lot of thought I am enthusiastically +1. I don't have time to write up my journey on this topic, but can do so in a review thread if it helps.
Without this, we are potentially putting a nightmare scenario on developers and library authors. Code works for years until it suddenly breaks in optimized builds, in ways that are very difficult and time consuming to track down. And, there's usually no recourse, other than to rewrite or redesign the system to be in accordance with lifetime rules that are invisible in the source code. Constant programmer vigilance shouldn't be required under normal conditions.
Although it seems unrelated to lexical lifetimes, this goes hand-in-hand. It makes sense to shift the burden of reasoning about lifetimes from unannoated non-expert code to annotated expert code. This is a way to, explicitly, get non-lexical lifetimes.
I slightly prefer the keyword-like spelling for the move operator, like foo(move x), especially since the most common situations are to move a value into a function parameter or into an assignment. The extra set of parenthesis add noise in those cases. But I haven't thought through the impact.
Alternatively, a namespace for intrinsics would make move() more palatable and less magical.
I am embarassed to say that even though I have learned what __owned and __shared means dozens of times, I still have to relearn them nearly every time. I just want consistent terminology instead of having to consult multiple decoder rings to know what something actually does or means. +1 for unifying on "consuming" and "nonconsuming" (or anything consistent and understandable, really) with good documentation. Especially documentation.
Removing the underscore and adding documentation is vital for libraries (as well as avoiding editor embarassment when the auto-generated protocol conformance stubs adds them). I'm especially looking fowards to documentation.
Are assignments considered explicit or implicit copies? Similarly for subscript and var setters that appear in source code as assignments.
Yes please! Have you thought about whether this is const poisonous and how to avoid it if so? For much of the code that I write, I almost want this to be the default for non-trivial values, but only if every function that I need call has been similarly carefully annotated and considered. But this is great for leaf functions!
What about "nonescaping" return values, that is return values that cannot escape the lexical scope of the caller? That seems like it would work alongside lexical lifetimes for APIs that need to be closure-based for lifetime management reasons alone.
This makes sense to me. When I first came to Swift I found it confusing what mutates in-place and what just modifies a copy. Learning about accessors helps (i.e. not thinking in terms of l-values), and I feel like this is a sensible extension to that.
Actually, we can be considerably more efficient than UnsafeBufferPointer, ideally without binding the type of the memory and having a consistent opt-out bounds checking story.
We may investigate this more when the concurrency model has been better fleshed out. Some of the features planned to enforce data race safety may allow us to know when certain references are thread-isolated and can use nonatomic refcounting.
Working with the weak reference itself shouldn't change much under this proposal. If you're going to perform multiple operations on the same object, you'll still want to "strongify" the reference to ensure you have a stable reference for the duration of those operations. But if let and guard let in Swift already do that for you. Lexical lifetimes intend to make the side that holds the one strong reference to the object less hazardous, by making it less likely you need to use withExtendedLifetime to stop the optimizer from shortening the lifetime of the strong reference.
Rust has "non-lexical lifetimes" too, though it's stricter than Swift's traditional rule, since the end of a binding's lifetime is pinned to its last use in source, whereas Swift didn't consider no-op loads like _ = x to be formal uses of x, and so inlining and other optimizations could allow lifetimes to shorten even further than the last source-level use of the variable. Rust's non-lexical lifetime rule could however still pose a hazard for weak references in the example from the original post, since the last source-level use of the strong reference to the delegate object still happens before the access into the weak reference.
In this particular example, there would be no change from today, since the initializer contains no deinitialization barriers. Both today and with lexical lifetimes, you will likely see an extra copy on write in debug builds, because people generally expect local variables to be readable anywhere in scope in the debugger, and maybe that copy will be elided in optimized builds, but there's no strong guarantee either way. move provides a way to assert an important performance property, but it still isn't strictly necessary to get the benefit of optimization.
I would say that async is different from other coroutines, because the coroutine transform for async functions is really an implementation detail; they otherwise behave like regular functions at the surface level. The ownership manifesto does discuss supporting generator coroutines in the future, which would allow for more efficient for loops that borrow their underlying sequence's storage, similar to how accessor coroutines allow for more efficient property access than get/set functions.
An assignment is effectively destroying the old value in the LHS, and consuming the new value from the RHS. It would make sense to me if we still required an explicit copy or move if the RHS is a non-implicitly-copyable expression, and the assignment would necessitate a copy of the underlying value, so if you had:
@noImplicitCopy var x: ExpensiveType = expensiveValue()
var y: ExpensiveType, z: ExpensiveType
...
...
y = copy(x)
z = x
the copy on the assignment into y would be required at minimum, since expensiveValue() has to be copied to be able to assign into both y and z.
I think that, in order for it to be effective, @nonescaping will have to be viral to some degree like const, and it'll need to be retrofitted to a good chunk of the standard library in order for @nonescaping operations to be expressive. As part of the eventual proposal for @nonescaping, we shouuld discuss how we can manage that annotation burden.
It's more that we want to enable more aggresive lifetime optimization. New ARC rules are needed to constrain those optimizations enough to avoid breaking existing use cases.
In this specific example, the compiler eliminates the copy today, and, ideally, that optimization should continue to work with lexical lifetimes.
Optimization only works today because of a combination of heroics on the part of the compiler and trivial nature of the example. Add anything to that initializer that the compiler can't fully analyze, and the copy will stick around. A programmer could never be expected to guess what will happen. There are many ways the compiler approaches ARC optimization, and I had to do a lot of digging to understand which technique was doing the job here.
In the successful approach, the compiler figures out that the initializer has two copies of (references to) the same object. It proves that
Lexical lifetime rules do indeed make that optimization more difficult. Now the compiler either needs to prove that sort() isn't a deinitializer barrier, or that the referenced object has a default deinitializer. It helps that Array<String> is assumed to have a default deinitializer.
This lexical lifetime "problem" could also be solved without any compiler magic by introducing a @noLifetimeDependence attribute and annotating the sort() method as such. Only the @usableFromInline API entry points would need that annotation. Alternatively, a @defaultDeinit attribute could be added to types being sorted.
I think predictably optimizing lifetimes requires either intentional lifetime management in the API design or intentional lifetime management when using the API. Fortunately it doesn't require both.
The user of an API can annotate arguments using move,
@noImplicitCopy, @nonescaping, etc. That's where we are today
until we can migrate frameworks.
But ultimately, an API can annotate entry points with
@noLifetimeDependence, types with @defaultDeinit, or properties with
@nonescaping. In the long run, designing APIs around move-only types can
avoid the proliferation of attributes.
The next roadmap should talk about those API and type attributes, along with prohibiting associated objects on native Swift classes.
One thing I've been wondering since you mentioned it earlier - what makes default deinitialisers so special? Is it just that it's "default", or is there some other property of the deinit function which needs to be proven?
For example, if you use a ManagedBuffer for tail-allocated storage with a custom header, you need to write a custom deinit to clean up the tail-allocated elements. But that alone shouldn't prohibit any lifetime-related optimisations AFAICT, because it's basically the same thing Array needs to do.
I imagine proving that a deinitializer does/doesn’t have side effects might have something to do with this. If the following code were run with with a lexical lifetime model, then foo’s deinitializer would have to be called after print(0), even though foo’s last use is before that line.
class Foo {
deinit {
print(1)
}
}
do {
let foo = Foo()
print(0)
}
Right. What I was alluding to is that IIRC, in C++, any user-defined destructor at all (even something with no side-effects, such as an empty destructor) makes a class non-trivial. I wonder if that's something we inherited, or whether Swift's definition is different in some way that could allow more things to be considered trivially-destructible.
Thanks @Andrew_Trick and @Joe_Groff your answers clarified my doubts. Very insightful to know how things work under the hood. 
One thing that I don't think has been answered yet (maybe because we need concrete proposals which is fine) is this:
I think this will be the improvement that will have more use so I'm keen on knowing if it's realistic or not.
Cheers
"Default deinit" is not being proposed yet, but to give you an idea...
For the purpose of reordering deinitialization with unknown side effects, we only care that the deinitializer doesn't make external calls (including UnsafePointer::deallocate).
That's not sufficient to completely ignore deinitialization barriers. We also need to know there aren't weak refs or "interior pointers" that aren't covered by a withExtendedLifetime. We could extend "default deinit" with restrictions on the non-deinit code as well, prohibiting those code patterns on references of the type.
The benefit of a "default deinit" type attribute or marker protocol goes way beyond reordering deinitialization though. It helps solve the bigger problem of optimizing in the presence of synchronous deinitializers by allowing the compiler to treat deinitialization as a side-effect free operation. This means releasing a reference no longer interferes with other optimizations--general optimizations that are unrelated to optimizing the object's lifetime.
I like to refer to anything that constrains the deinitializer as "default deinit" because the meaning is obvious without getting into those details, and it covers a broad set of issues. It really says that the deinitializer doesn't access shared state (beyond refcounts). The deinitializer only releases references owned by the object. Programmers could safely opt-in to the attribute (marker protocol) with compiler enforcement. A very small set of programmers could opt out of the compiler's enforcement by adding @defaultDeinit(unsafe) to their custom deinitializer, promising that it only releases owned references. That covers the ManagedBuffer case.
Is this a good time to propose a better system for actually making functions conform to protocols?
Personally, I’d like to see every nominal type become capable of protocol conformance. Including protocols, once SE-0335 makes them less of a nightmare to work with directly.
Granted, the function type isn’t what needs conformance so much as the function value… At any rate, whether or not that is in scope for this roadmap, I think the issue deserves more than an endless torrent of special attributes and keywords. Retroactively if necessary.
Speaking of which, I’m of the firm belief that major releases of Swift should not be overly averse to fixing old mistakes. Even the most fundamental parts of the language, if there is a truly immense amount of support, should be open to revision.
In languages like Python, where errors are often extremely difficult to uncover until runtime, changes of that nature are considered anathema. But Swift’s focus on compile-time safety, combined with increasingly impressive migration tooling, ABI stability, and language modes, means that we should not be afraid to break things for the sake of future code.
In particular, we should be willing to replace short-term fixes with long-term solutions. Especially when the meaning in one major version can be unambiguously (and therefore automatically) translated to the meaning in another.
A few points:
Naturally, as someone who's struggled to wring performance from high-level Swift code, I'm in favor of the intent of this proposal.
As you might predict, though, I have concerns about the approach.
IMO guaranteeing the lifetime of locals until end-of-scope is one of the worst design tradeoffs in C++. The “RAII trick” of attaching cleanups to the lifetimes of locals is not inherently better than defer—to the contrary, it's only an indirect expression of intent—and requiring explicit move() on last use (or penalizing performance when it is omitted) for the convenience of low-level code that exposes pointers or interoperates with C/C++ seems like an inversion of design priorities. Do we really believe this change is consistent with the spirit of Swift?
At Google, the explicit moving of constructor arguments, exactly analogous to the SortedArray example in the pitch, is either in the C++ coding guidelines or routinely inserted by linters (I don't remember which). In other words, it is standard boilerplate. Of course the initializer case is the one that's easy to remember and get right, but to get it right in the many other cases where it matters is quite error prone and requires vigilance from programmers. To ask programmers for extra effort in the case of unsafe code, where they already have to exercise vigilance, and to run high-level code as efficiently as possible, seems much more consistent with the design philosophy of Swift.
It seems a bit premature to make other ARC changes in the name of performance predictability while semantic ARC is (IIUC) as yet not-fully-realized. Swift's performance and performance predictability has always been hobbled by an ARC implementation that doesn't work at -O0 and tries to do its job based on a fraction of the information available in the source. Shouldn't that work be carried through to its conclusion so we can see what the actual remaining problems are? /cc @Michael_Gottesman
The problem of withXXX { ... } pyramids-of-doom exists even if we eliminate the need for withExtendedLifetime, and ought to be addressed with a language feature. In fact I proposed one to the Swift team when I was working on SwiftUI for completely different reasons. Just as we see here, these pyramids come up often and tend to drive the introduction of mitigating language changes, which after all are expensive.
The complexity of what's being proposed here seems quite significant and, at least in part, easily avoided. Observe that the cases where you want to pass arguments by consuming them are exactly those where the argument needs to escape. Can't we leverage this fact to avoid introducing consuming, nonConsuming, @escaping, and @nonEscaping as separate concepts?
I was wondering about that last point myself.
I generally assume that the lifetime of locals end at last use, give or take compiler optimizations. Whether or not that is currently accurate, I agree that performant code should not require extra work.
It seems clear that there is demand for such functionality, however. For the sake of C++ interop, at least, it is probably for the best that these tools exist (similar to the string-based dynamic key paths used for Python), so long as it is made clear they should not be used often. Perhaps only for wrappers?
But the tools for explicit lifetime extension already exist: withUnsafePointer, withExtendedLifetime, et al. Unless I'm misunderstanding something, the proposal is to make the tools unnecessary by implicitly extending lifetimes, everywhere, by default.