[Manifesto] Ownership

I ran into this exact issue today. Would something like inout returns help to fix it?

I have a struct called StyledString which gets rid of much of the frustration of dealing with NSAttributedStrings in Swift. Basically, you make selections on the string and set some property of it like so:

  styled.paragraph(2).words(3…7).bold = true

and then at the end, you ask for an attributed string:

  myLabel.attributedText = styled.attributedString

The language I originally designed this for (almost 20 years ago) had a special type of function (called a selector) which could be in the middle of a chain of calls, but not at the end of it… which avoided the problem you are talking about here. In Swift, I basically have a note in the documentation saying not to store the selections in a variable… but that is not ideal.

It would be nice to have a way to say that the return value is not allowed to be stored (only called as part of a chain). This would allow mutable views without the issue of them breaking value semantics.

Thanks,
Jon

···

On Feb 27, 2017, at 9:08 AM, John McCall via swift-evolution <swift-evolution@swift.org> wrote:

3: Mutable Views

Currently a major pain point of Swift has been exposing “mutable views” into a type in a reasonable way. Apologies for length, but here’s an extended example:

  /// Implements a 2D grid (of size fixed upon initialization).
  /// **Not** a `Collection` itself but has many *views* into it that *are* `Collection`s.
  ///
  struct Grid2D<T> {
    // typical CoW backing storage
    fileprivate var storage: Grid2DStorage<T>
    
    /// Obtain a linear traversal of our grid:
    /// - parameter corner: The corner at which we start (e.g. `northWest`)
    /// - parameter motion: The order in which we travel (`latitudeThenLongitude` or `longitudeThanLatitude`)
    ///
    /// - returns: A read-only `Collection` that visits each grid location in `self`, following the requested traversal
    func traversal(from corner: Grid2DCorner, following motion: Grid2DMotion) -> Grid2DTraversal<T>
  }

…wherein `Grid2DTraversal<T>` is, as noted, a "read-only view” into its parent that also adopts `Collection`.

What would be nice is to be able to make that `Grid2DTraversal<T>` into a mutable view in a way that’s also *safe*; what I mean is something like:

  extension Grid2D {
  
    mutating func mutatingTraversal<T>(from corner: Grid2DCorner, following motion: Grid2DMotion, _ mutator: (inout Grid2DTraversal<T>) -> R) -> R {
      var t = self.traversal(from: corner, following: motion)
      return mutator(&t)
    }

  }

…with the specific goals of (a) having mutations applied via `Grid2DTraversal` get directly written-through to the underlying storage (w/out pointless copying) but also (b) making `Grid2DTraversal` generally act safely in all other contexts.

As it is the “best” way to squeeze this into the type system at this time is arguably:

- define Grid2DTraversal:
  - to have as a strong reference to the backing storage
  - as only having the read-only methods
- *also* define MutableGrid2DTraversal:
  - to have an unowned reference to the backing storage
  - also include the mutating methods

…and then just be careful with API design and patterns of use; in particular only provide access to `MutableGrid2DTraversal<T>` values in the context of closures passed into dedicated functions like `mutatingTraversal`, above…and then just be disciplined never to extract those passed-in values.

Needless to say this isn’t a very satisfactory state of affairs—it is well into “why am I bothering with all this?” territory—and it’d be *far* nicer if the ownership system would allow for `Grid2DTraversal`:

- to adopt `Collection` without restriction
- to adopt `MutableCollection` only in certain ownership contexts
- to have reasonable control over where those contexts apply

It's not sensible to have a type that conditionally conforms to a protocol based on context. However, the requirements of MutableCollection are all 'mutating', so you can't actually use them unless you have a mutable value. So the way to fit what you're asking for into the ownership proposal is to make sure that clients of your view type only have a mutable value when the base was mutable, and the easiest way of getting that is to have the view be vended as storage rather than a return value. If you think about it, a mutable view can't be an independent value anyway, because if code like this could work:

  var grid = ... // note that this is mutable
  var view = grid.traversal(from: p, following: m) // should return a mutable view by design, right?
  view.mutate() // reflected in grid, right?

then it completely destroys the value-type properties of 'grid', because 'view' should really be an independent value.

The proposal suggests doing this instead with storage, so that the view is logically a (mutable) component of the grid. So on the use side:

  var grid = ...
  inout view = &grid[traversingFrom: p, following: m]
  view.mutate()

and on the declaration side:

  struct Grid2D<T> {
    subscript(traversingFrom corner: Grid2DCorner, following motion: Grid2DMotion) -> Grid2DTraversal<T> {
      read {
        yield Grid2DTraversal(...)
      }
      modify {
        var traversal = Grid2DTraversal(...)
        yield &traversal
        // ensure changes were applied here
      }
    }
  }

If you feel that the aesthetics of this leave something to be desired, that's a totally reasonable thing to discuss.

Thanks very much for writing this up, it’s very interesting.

The one thing which I think needs improvement is the Copyable protocol. I think that this is actually part of a larger problem in Swift, which is that we don’t expose enough information to generic code for it to operate safely. This goes back to people asking for a value-equivalent to the “class” or “AnyObject” magic protocols.

For example, often you would like to wrap a Collection and keep some information about what it contains. In order to do that generically, you need to ensure a client can hand you an exclusive view of a collection’s contents, so you know they will not mutate underneath your feet.

Currently, that means you need a RangeReplaceableCollection because it includes an empty initialiser which allows you to create a unique copy of the Collection’s contents. It’s a workaround, but it means we have this unnecessary protocol requirement in the standard library, and in the implementation, which is making copies that may not be required. If my CollectionWrapper is initialised with something which has ValueSemantics, I don’t need to create a whole new instance with equal contents to ensure exclusivity of those contents. If this was an Array, for example, I could simply assign it and it would ensure the underlying contiguous buffer is never mutated.

struct CollectionWrapper<C: Collection> {
    var _underlying: C

    init(contentsOf other: C) where C: RangeReplaceableCollection {
        _underlying = C();
        _underlying.replaceSubrange(_underlying.startIndex..<_underlying.endIndex, with: other)
    }
}

// Would love to do this…

extension CollectionWrapper {
    init(contentsOf other: C) where C: ValueSemantics {
        _underlying = other
    }
}

As the proposal notes (with the File example), these are semantic considerations which go beyond simple struct/class distinctions: structs may have reference semantics, and classes may have value semantics. Would it be possible to model Copyable/move-only in terms of two new protocols, ValueSemantics and ReferenceSemantics, with trivial types/non-trivial types given appropriate defaults (overridable by the programmer, such as for “File”)?

- Karl

1: Collection Composition

As much as the high-level picture is focused on safety, I’d just like to request a strong focus on making sure the eventual ownership system addresses the common issues like avoiding copies when mutating through optionals, or when nesting collections, and so on.

Avoiding copies is basically the entire core of the proposal. The interaction with safety is that sometimes copies are necessary to avoid memory unsafety, and this proposal recommends an approach that minimizes that.

Thanks for the extend reply and sorry for my own belated reply.

What motivated me to ask for more focus on the common cases was that it was unclear from reading the proposal whether the `inout` binding--as used in `for inout`--could be used with `if`: in other words, we currently have `if let` and `if var`, but will we have `if inout` under this proposal?

I am embarassed to admit I honestly couldn't tell if the proposal didn't contain any examples of `if inout` due to (a) the capability following trivially from the rest of the proposal or (b) the capability falling outside the scope of the proposal.

If it's (a) I apologize and feel free to skip down to the next section, if it's (b) I'll reiterate my request to for more focus on streamlining the common cases. In particular it'd be nice if there was a relatively homogeneous syntax as sketched below:

for related operations starts to diverge a bit:

  var dict: [K:Set<V>] = // ...
  
  // read-only uses `if-let` (or `if shared`?)
  if let setForKey = dict[key] {
    // read `setForKey`
  }
  
  // simple in-place mutation
  if inout setForKey = dict[key] {
    setForKey.mutateSomehow()
  }
  
  // compound "in-place" mutation
  if inout setForKey = dict[key] {
    setForKey.mutateSomehow()
    setForKey.mutateSomeOtherWay()
  }

  // simple "in-place" mutation w/follow-up
  if inout setForKey = dict[key] {
    setForKey.mutateSomehow()
    let shouldRemove = setForKey.isEmpty
    endScope(setForKey)
    if shouldRemove {
      dict.removeValue(forKey: key)
    }
  }

  // compound "in-place" mutation w/follow-up
  if inout setForKey = dict[key] {
    setForKey.mutateSomehow()
    setForKey.mutateSomeOtherWay()
    let shouldRemove = setForKey.isEmpty
    endScope(setForKey)
    if shouldRemove {
      dict.removeValue(forKey: key)
    }
  }
  
...which, again, all use approximately the same syntax for each of these scenarios.

Contrast with what would be the ideal ways to write the above under the proposal AIUI:

  var dict: [K:Set<V>] = // ...
  
  // read-only uses `if-let` (or `if shared`?)
  if let setForKey = dict[key] {
    // read `setForKey`
  }
  
  // simple in-place mutation
  if let index = dict.index(of: key) {
    dict.values[index].mutateSomehow()
  }
  
  // compound "in-place" mutation, I
  // shorter, less-efficent due to repeated lookup
  if let index = dict.index(of: key) {
    dict.values[index].mutateSomehow()
    dict.values[index].mutateSomeOtherWay()
  }

  // compound "in-place" mutation, II
  // longer, more-efficent
  if let index = dict.index(of: key) {
    inout setForKey = &dict.values[index]
    setForKey.mutateSomehow()
    setForKey.mutateSomeOtherWay()
  }

  // simple "in-place" mutation w/follow-up, I
  // longer, less-efficient due to repeated lookup
  if let index = dict.index(of: key) {
    dict.values[index].mutateSomehow()
    if dict.values[index].isEmpty {
      dict.remove(at: index)
    }
  }

  // simple "in-place" mutation w/follow-up, II
  // longer, less-efficient due to repeated lookup
  if let index = dict.index(of: key) {
    inout setForKey = &dict.values[index]
    setForKey.mutateSomehow()
    let shouldRemove = setForKey.isEmpty
    endScope(shouldRemove)
    if shouldRemove {
      dict.remove(at: index)
    }
  }

  // compound "in-place" mutation w/follow-up, I
  // longer, less-efficient due to repeated lookups
  if let index = dict.index(of: key) {
    dict.values[index].mutateSomehow()
    dict.values[index].mutateSomeOtherWay()
    if dict.values[index].isEmpty {
      dict.remove(at: index)
    }
  }

  // compound "in-place" mutation w/follow-up, II
  // longer, less-efficient due to repeated lookup
  if let index = dict.index(of: key) {
    inout setForKey = &dict.values[index]
    setForKey.mutateSomehow()
    setForKey.mutateSomeOtherWay()
    let shouldRemove = setForKey.isEmpty
    endScope(shouldRemove)
    if shouldRemove {
      dict.remove(at: index)
    }
  }

...where my concern is less about what how particular scenario winds up looking and more that there’s such variance *between* the scenarios, at least when compared to a hypothetical `if inout` construct.

Again if `if inout` is actually part of the proposal I apologize for illustrating the consequence of its absence at such length.

The technical obstacle to this is just that, so far, we've tried to make language features like for-loops use formal protocols.

An iteration protocol is going to have requirements like this:
  generator iterate() -> Element
  mutating generator iterateMutable() -> inout Element
But there's no valid substitution that makes '(Key, inout Value)' equal either 'Element' or 'inout Element'. So we'd have to do some special to make this work.

That said, no, there's no intrinsic technical reason this can't be made to work.

The explanation of wanting to stick to formal protocols makes perfect sense. I don’t think this should be a show-stopper, but I do think it’ll be a common request for a subsequent enhancement.

Even in the interim it seems quite feasible to emulate the capability in any concrete case with enough willingness to crank out boilerplate; thus e.g. if someone truly needs `for (index, inout value) in collection.enumerated()` or `for (a, inout b) in zip(as,bs)` it isn’t as if they’re entirely stuck.

3: Mutable Views

It's not sensible to have a type that conditionally conforms to a protocol based on context.

That did indeed seem like a big ask!

I’ll put in an early request to consider

  @moveonly {
    struct File { }
  }

…(or @scope(moveonly) or @context(moveonly) or @dialect(moveonly), etc.).

It’s confusing that `moveonly` essentially applies “inward”—it flags the code *within* a scope as using different assumptions, etc.—in contrast to most of the modifiers like `open`, `final`, `mutating`, etc., apply “outward” (by e.g. describing how visible the code in the scope is from other scopes, etc.).

However, the requirements of MutableCollection are all 'mutating', so you can't actually use them unless you have a mutable value. So the way to fit what you're asking for into the ownership proposal is to make sure that clients of your view type only have a mutable value when the base was mutable, and the easiest way of getting that is to have the view be vended as storage rather than a return value. If you think about it, a mutable view can't be an independent value anyway, because if code like this could work:

  var grid = ... // note that this is mutable
  var view = grid.traversal(from: p, following: m) // should return a mutable view by design, right?
  view.mutate() // reflected in grid, right?

then it completely destroys the value-type properties of 'grid', because 'view' should really be an independent value.

Without belaboring this, the point is indeed to “destroy the value-type properties of `grid`”, while trying to keep things “safe” by quarantining `view`—and the temporary relaxation of value semantics its existence implies—to a specific scope; thus under the status quo the use-site looks like this:

  var grid = // note that this is mutable
  // also note that `mutatingTraversal` is a `mutating` method...
  grid.mutatingTraversal(from: c, following: m) {
    (inout view: MutableGrid2DTraversal<T>) -> Void
    in
    // ^ this is the only public method that vends `MutableGrid2DTraversal<T>`, and
    // `MutableGrid2DTraversal<T>` has no public constructors, thus `view` is
    // “quarantined” to this scope unless we actively attempt to leak `view`...
    view.mutate()
  }

…which currently has two major drawbacks:

- (a) lots of code duplication: the immutable `Grid2DTraversal<T>` and the mutable `Mutable2DTraversal<T>`
- (b) the “quarantine” isn’t airtight, in that there are at least these three ways a `view` could escape:

  var grid = // …
  var leakedView: MutatingGrid2DTraversal<T>? = nil
  var otherLeakedView = grid.mutatingTraversal(from: c, following: m) {
    (inout view: MutableGrid2DTraversal<T>) -> MutableGrid2DTraversal<T>
    in
    view.mutate()
    // leak #1:
    leakedView = view
    // leak #2:
    self.liveDangerously(leaking: view)
    // leak #3:
    return view
  }

…which imho makes it fair to say “the above approach *encourages* ‘safe’ usage, but can’t *prevent* unsafe usage”.

Under the ownership proposal nothing changes for drawback (a): to stick with the design and behavior sketched above requires distinct types and thus a lot of duplicate or near-duplicate boilerplate.

Drawback (b) seems to fare better under the ownership proposal, provided that I make `MutatingGrid2DTraversal` a non-Copyable type; at least AIUI making `MutatingGrid2DTraversal` non-Copyable would effectively close leaks #1, #2, and #3, b/c:

- I would explicitly have to write e.g. `leakedView = move(view)` (a *very* unlikely accident)
- I would also have to supply a "replacement value” for `view` by the end of the scope (impossible due to lack of public constructors)

…at least if I understand correctly. Is this accurate? If accurate, how likely would it be that “failed to provide replacement value for `view` after move” would get caught at compile time?

Thanks again for providing such detailed clarifications about this proposal.

···

On Feb 27, 2017, at 11:08 AM, John McCall <rjmccall@apple.com> wrote:

On Feb 25, 2017, at 11:41 AM, plx via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

The proposal suggests doing this instead with storage, so that the view is logically a (mutable) component of the grid. So on the use side:

  var grid = ...
  inout view = &grid[traversingFrom: p, following: m]
  view.mutate()

and on the declaration side:

  struct Grid2D<T> {
    subscript(traversingFrom corner: Grid2DCorner, following motion: Grid2DMotion) -> Grid2DTraversal<T> {
      read {
        yield Grid2DTraversal(...)
      }
      modify {
        var traversal = Grid2DTraversal(...)
        yield &traversal
        // ensure changes were applied here
      }
    }
  }

If you feel that the aesthetics of this leave something to be desired, that's a totally reasonable thing to discuss.

John.

Anyways, it’s not clear to me, personally, whether or not the above is within the scope of any likely, concrete ownership system that’d follow from the manifesto or not…but if at all possible I’d prefer the eventual ownership system make it reasonable—and reasonably safe—to implement “small-c ‘collection’s that can safely-and-efficiently expose various *mutable* views as big-C `Collection`s”.

Apologies if all of the above considerations have answers that follow trivially from the manifesto; it’s just unclear personally whether the features described in the manifesto would work together to allow something like the above to be implemented more-reasonably than currently the case.

On Feb 17, 2017, at 11:08 AM, John McCall via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Feb 17, 2017, at 4:50 AM, Adrian Zubarev <adrian.zubarev@devandartist.com <mailto:adrian.zubarev@devandartist.com>> wrote:
Hi John, would you mind creating a markdown document for this manifesto in https://github.com/apple/swift/tree/master/docs? :)

Yes, it should go in the repository. That commit is pending, but the in meantime, you can see the document properly rendered at:
  https://github.com/rjmccall/swift/blob/4c67c1d45b6f9649cc39bbb296d63663c1ef841f/docs/OwnershipManifesto.md

John.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

I think the idea of dynamic bookkeeping state is pretty great, and, in particular, not having to be exact about the state in the presence of concurrent access is a nice way to make it lightweight.

The way I read the manifesto, the bookkeeping field would work like this:

(a) Thread 1 sets bookkeeping state to read, saves previous state (unaccessed)
(b) Thread 2 sets bookkeeping state to read, saves previous state (read)
(c) Thread 1 ends access, restores state to unacceseed -- too early, but whatever!
(d) Thread 2 ends access, restores state to read? No, you NEVER restore a read.

If you never restore a previously saved "read" state, then you'll never get stuck in a "read" state even if the restore order is wrong (like it is here, because of the multiple threads).

In this scenario, thread 2 could also restore the state to "unaccessed" instead of doing that conditionally on the saved state. In general however, for reentrant functions to keep single-thearded bookkeeping always exact, setting the state to "unaccessed" has to be conditional to the saved state being the same.

If (a) and (b) are not atomic exchanges, a third thread could write a "modify" state during the non-atomic writing of your "read" state, which would make the overlap of the two incompatible accesses undetected.

With atomic exchanges in (a) and (b), an incorrect "modify" can only pass undetected during the time between (c) and (d). If by chance (c) and (d) were to occur in reverse order (making it proper nesting time-wise), an incorrect "modify" between the two would become detectable.

This is all very interesting.

···

Le 17 févr. 2017 à 15:08, John McCall via swift-evolution <swift-evolution@swift.org> a écrit :

We don't have an initial implementation yet, sorry. I'm sure it'll vary a lot by the application. My hope is that it'll just be a few extra non-atomic loads and stores around every access, but it's possible that we'll need to make at least one of those an atomic exchange because it's legal for multiple threads to read concurrently. It hinges in part on how comfortable we are with allowing conflicts to escape dynamic detection in complex concurrent cases.

--
Michel Fortin
https://michelf.ca

Hi John,

This is fantastic! I’ve been eagerly anticipating this. It looks very much as I was hoping it would. I can’t wait to see this vision come to life!

You only show a mutating generator example. Do you also plan to allow shared generators?

Yes; a generator yielding a shared reference would be used for non-mutating iteration.

One topic you don’t touch on that feels related is the ability to use RAII and rely on deferred deinit to ensure an owned resource isn’t released before the scope exits. I can imagine something like `defer let resource = MyResource()`. It may also be interesting to support “defer only” types to force a compiler error where non-deferred use would be incorrect. What do you think?

My intuition is the use cases for this that I'm aware of are really a mis-use of values. You might design an API that way in C++, where destructors are really the only language mechanism for injecting code into a function "later", but in Swift I think you would want to use either (1) a generalized accessor (if the protected resource could meaningfully be described as a single value) or (2) a callback that explicitly scopes what happens with the resource.

So e.g. if you wanted a Rust-style mutex API, you could do it like so:

moveonly struct Locked<T> {
  var unlockedMemory: T { // probably not the best name
    read {
      mutex.acquire()
      defer { mutex.release() } // There are reasons why I think this needs to be in a defer, but we can talk about them when we get to the detailed proposals for co-routines. I'm still looking for a better language design.
      yield actualMemory // Recall that 'read' yields a shared value, so this doesn't implicitly copy.
    }
    nonmutating modify { // 'nonmutating' is a misnomer, but it's an existing misnomer. Setters are 'mutating' methods by default and normally demand exclusive access to self, but we don't need that here because we're dynamically enforcing exclusive access to the memory, so all we need is shared access, and this is how we express that.
      mutex.acquire()
      defer { mutex.release() }
      yield &actualMemory
    }
  }

  private var mutex: Mutex
  private mutable var actualMemory: T // Make this variable mutable even within nonmutating methods; whether that makes sense is the programmer's problem. I didn't cover this in the proposal, because it's speculative, but it's useful for things like our nonmutating modify. Lots of open questions here.
}

I like this approach. It scopes access to the resource as necessary without requiring nesting.

Or if you wanted a more C-like mutex API, you'd do it like so:

moveonly struct Mutex {
  func withLock<T>(_ function: () throws -> T) rethrows -> T {
    acquire()
    defer { release() }
    return try function()
  }
}

But I just don't see the motivation to try to do this by making a function return a C++-style lock guard.

Do you have a use case which clearly benefits from an exact scoping and really needs to be an independent value?

To be honest, I don’t know. The primary thing I am looking for is a way to encapsulate the scoping of a resource without requiring the introduction of a new lexical scope like is required in your second example.

Yes, I agree that the nesting isn't ideal. In particular, it's awkward to get values out of the scope.

The first approach you show above using generalized accessors may well be able to cover our needs. Thanks for taking time to show the example!

Okay, glad to hear it. If you *do* find something which really benefits from an RAII-style value, please let me know; I'm certainly open to the idea.

John.

···

On Feb 20, 2017, at 4:26 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Feb 19, 2017, at 11:24 PM, John McCall <rjmccall@apple.com <mailto:rjmccall@apple.com>> wrote:

On Feb 18, 2017, at 11:08 AM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

John.

On Feb 17, 2017, at 11:08 AM, John McCall via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Feb 17, 2017, at 4:50 AM, Adrian Zubarev <adrian.zubarev@devandartist.com <mailto:adrian.zubarev@devandartist.com>> wrote:
Hi John, would you mind creating a markdown document for this manifesto in https://github.com/apple/swift/tree/master/docs? :)

Yes, it should go in the repository. That commit is pending, but the in meantime, you can see the document properly rendered at:
  https://github.com/rjmccall/swift/blob/4c67c1d45b6f9649cc39bbb296d63663c1ef841f/docs/OwnershipManifesto.md

John.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

I think it might not be very related to ownership, but the use of the
word “Copyable” may be a problem. For most programmers when applied to
reference types “Copyable” will connote “Clonable,” i.e. that the
referent (rather than the reference, which is what you're referring to
when you say class types are copyable) can be explicitly copied. So I'm
betting Karl read “Copyable” and was led to this other topic which is of
concern to many people. In that sense, claiming “Copyable” for
ownership without also addressing “Clonable” could be a problem.

The idea behind “Clonable” would be that it gives you a *generic* way to
create a logically *independent* copy of a value, that would work both
for value types and reference types.

···

on Mon Mar 06 2017, John McCall <swift-evolution@swift.org> wrote:

On Mar 6, 2017, at 3:46 PM, Karl Wagner <razielim-Re5JQEeQqe8AvxtiuMwx3w@public.gane.org> wrote:
Thanks very much for writing this up, it’s very interesting.

The one thing which I think needs improvement is the Copyable protocol. I think that this is

actually part of a larger problem in Swift, which is that we don’t expose enough information to
generic code for it to operate safely. This goes back to people asking for a value-equivalent to the
“class” or “AnyObject” magic protocols.

For example, often you would like to wrap a Collection and keep some information about what it contains. In order to do that generically, you need to ensure a client can hand you an exclusive view of a collection’s contents, so you know they will not mutate underneath your feet.

Currently, that means you need a RangeReplaceableCollection because it includes an empty initialiser which allows you to create a unique copy of the Collection’s contents. It’s a workaround, but it means we have this unnecessary protocol requirement in the standard library, and in the implementation, which is making copies that may not be required. If my CollectionWrapper is initialised with something which has ValueSemantics, I don’t need to create a whole new instance with equal contents to ensure exclusivity of those contents. If this was an Array, for example, I could simply assign it and it would ensure the underlying contiguous buffer is never mutated.

struct CollectionWrapper<C: Collection> {
    var _underlying: C

    init(contentsOf other: C) where C: RangeReplaceableCollection {
        _underlying = C();
        _underlying.replaceSubrange(_underlying.startIndex..<_underlying.endIndex, with: other)
    }
}

// Would love to do this…

extension CollectionWrapper {
    init(contentsOf other: C) where C: ValueSemantics {
        _underlying = other
    }
}

As the proposal notes (with the File example), these are semantic
considerations which go beyond simple struct/class distinctions:
structs may have reference semantics, and classes may have value
semantics. Would it be possible to model Copyable/move-only in terms
of two new protocols, ValueSemantics and ReferenceSemantics, with
trivial types/non-trivial types given appropriate defaults
(overridable by the programmer, such as for “File”)?

Class types have reference semantics and are still copyable; ownership
is not going to change that. More generally, I don't see how anything
we can do in ownership could ever do something like eliminate the
possibility of a reference-semantics collection from the language.

--
-Dave

Class types have reference semantics and are still copyable; ownership is not going to change that. More generally, I don't see how anything we can do in ownership could ever do something like eliminate the possibility of a reference-semantics collection from the language.

John.

···

On Mar 6, 2017, at 3:46 PM, Karl Wagner <razielim@gmail.com> wrote:
Thanks very much for writing this up, it’s very interesting.

The one thing which I think needs improvement is the Copyable protocol. I think that this is actually part of a larger problem in Swift, which is that we don’t expose enough information to generic code for it to operate safely. This goes back to people asking for a value-equivalent to the “class” or “AnyObject” magic protocols.

For example, often you would like to wrap a Collection and keep some information about what it contains. In order to do that generically, you need to ensure a client can hand you an exclusive view of a collection’s contents, so you know they will not mutate underneath your feet.

Currently, that means you need a RangeReplaceableCollection because it includes an empty initialiser which allows you to create a unique copy of the Collection’s contents. It’s a workaround, but it means we have this unnecessary protocol requirement in the standard library, and in the implementation, which is making copies that may not be required. If my CollectionWrapper is initialised with something which has ValueSemantics, I don’t need to create a whole new instance with equal contents to ensure exclusivity of those contents. If this was an Array, for example, I could simply assign it and it would ensure the underlying contiguous buffer is never mutated.

struct CollectionWrapper<C: Collection> {
    var _underlying: C

    init(contentsOf other: C) where C: RangeReplaceableCollection {
        _underlying = C();
        _underlying.replaceSubrange(_underlying.startIndex..<_underlying.endIndex, with: other)
    }
}

// Would love to do this…

extension CollectionWrapper {
    init(contentsOf other: C) where C: ValueSemantics {
        _underlying = other
    }
}

As the proposal notes (with the File example), these are semantic considerations which go beyond simple struct/class distinctions: structs may have reference semantics, and classes may have value semantics. Would it be possible to model Copyable/move-only in terms of two new protocols, ValueSemantics and ReferenceSemantics, with trivial types/non-trivial types given appropriate defaults (overridable by the programmer, such as for “File”)?

That's a very fair point.

John.

···

On Mar 6, 2017, at 4:30 PM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:
on Mon Mar 06 2017, John McCall <swift-evolution@swift.org> wrote:

On Mar 6, 2017, at 3:46 PM, Karl Wagner <razielim-Re5JQEeQqe8AvxtiuMwx3w@public.gane.org> wrote:
Thanks very much for writing this up, it’s very interesting.

The one thing which I think needs improvement is the Copyable protocol. I think that this is

actually part of a larger problem in Swift, which is that we don’t expose enough information to
generic code for it to operate safely. This goes back to people asking for a value-equivalent to the
“class” or “AnyObject” magic protocols.

For example, often you would like to wrap a Collection and keep some information about what it contains. In order to do that generically, you need to ensure a client can hand you an exclusive view of a collection’s contents, so you know they will not mutate underneath your feet.

Currently, that means you need a RangeReplaceableCollection because it includes an empty initialiser which allows you to create a unique copy of the Collection’s contents. It’s a workaround, but it means we have this unnecessary protocol requirement in the standard library, and in the implementation, which is making copies that may not be required. If my CollectionWrapper is initialised with something which has ValueSemantics, I don’t need to create a whole new instance with equal contents to ensure exclusivity of those contents. If this was an Array, for example, I could simply assign it and it would ensure the underlying contiguous buffer is never mutated.

struct CollectionWrapper<C: Collection> {
   var _underlying: C

   init(contentsOf other: C) where C: RangeReplaceableCollection {
       _underlying = C();
       _underlying.replaceSubrange(_underlying.startIndex..<_underlying.endIndex, with: other)
   }
}

// Would love to do this…

extension CollectionWrapper {
   init(contentsOf other: C) where C: ValueSemantics {
       _underlying = other
   }
}

As the proposal notes (with the File example), these are semantic
considerations which go beyond simple struct/class distinctions:
structs may have reference semantics, and classes may have value
semantics. Would it be possible to model Copyable/move-only in terms
of two new protocols, ValueSemantics and ReferenceSemantics, with
trivial types/non-trivial types given appropriate defaults
(overridable by the programmer, such as for “File”)?

Class types have reference semantics and are still copyable; ownership
is not going to change that. More generally, I don't see how anything
we can do in ownership could ever do something like eliminate the
possibility of a reference-semantics collection from the language.

I think it might not be very related to ownership, but the use of the
word “Copyable” may be a problem. For most programmers when applied to
reference types “Copyable” will connote “Clonable,” i.e. that the
referent (rather than the reference, which is what you're referring to
when you say class types are copyable) can be explicitly copied. So I'm
betting Karl read “Copyable” and was led to this other topic which is of
concern to many people. In that sense, claiming “Copyable” for
ownership without also addressing “Clonable” could be a problem.

The idea behind “Clonable” would be that it gives you a *generic* way to
create a logically *independent* copy of a value, that would work both
for value types and reference types.

I'll just drop a note to say that I explained how we could implement provable value-type semantics in the discussion about "pure" a few weeks ago. This was the overview:
https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170220/032582.html

The part that might have some relation to ownership is the notion in the type system that a pointer is guarantied to be uniquely referenced. You can check for that at runtime with `isUniquelyReferenced`, but that information is lost as soon as the call returns. If there was a way to safely flag a variable as a "unique reference" and for the compiler to detect when the uniqueness might be broken, then it'd open the door to the compiler being able to enforce value-type semantics.

···

On 6 mars 2017, at 15:53, John McCall via swift-evolution <swift-evolution@swift.org> wrote:

Class types have reference semantics and are still copyable; ownership is not going to change that. More generally, I don't see how anything we can do in ownership could ever do something like eliminate the possibility of a reference-semantics collection from the language.

--
Michel Fortin
https://michelf.ca

Part of it is the name; partly I was led there by the examples in the document. For example, a move-only struct with a deinitialiser looks an awful lot like it’s trying to express ReferenceSemantics, since it infers identity. How would using a move-only struct differ from using a class?

I’ve been thinking that we could do with a collection of magic protocols to constrain generic code based on type layout in general. There is a lot of generic code which could benefit from optimised implementations if they know that T is trivial, for example. At a higher level, we often want to know whether a particular type (regardless of struct/class) has reference/value semantics because we care about exclusivity over its contents.

Some of the constraints of the “law of exclusivity” sound like they are providing a kind of value semantic of contents at the variable-level (e.g. shared references are allowed as long as they do not change the contents). Several of the concepts in the document are new to me, but at some broad level there appear to be conceptual similarities.

At the same time, while references to classes are “Copyable” in the ownership sense, those copies are very different from copies of structs. For classes, those copies are basically worthless to the optimiser because it can’t guarantee anything about who else has references to the instance. I’m not really sure classes actually benefit at all from being “Copyable”. Perhaps they should be some other, closely-related thing instead?

- Karl

···

On 6 Mar 2017, at 22:30, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Mon Mar 06 2017, John McCall <swift-evolution@swift.org> wrote:

On Mar 6, 2017, at 3:46 PM, Karl Wagner <razielim-Re5JQEeQqe8AvxtiuMwx3w@public.gane.org> wrote:
Thanks very much for writing this up, it’s very interesting.

The one thing which I think needs improvement is the Copyable protocol. I think that this is

actually part of a larger problem in Swift, which is that we don’t expose enough information to
generic code for it to operate safely. This goes back to people asking for a value-equivalent to the
“class” or “AnyObject” magic protocols.

For example, often you would like to wrap a Collection and keep some information about what it contains. In order to do that generically, you need to ensure a client can hand you an exclusive view of a collection’s contents, so you know they will not mutate underneath your feet.

Currently, that means you need a RangeReplaceableCollection because it includes an empty initialiser which allows you to create a unique copy of the Collection’s contents. It’s a workaround, but it means we have this unnecessary protocol requirement in the standard library, and in the implementation, which is making copies that may not be required. If my CollectionWrapper is initialised with something which has ValueSemantics, I don’t need to create a whole new instance with equal contents to ensure exclusivity of those contents. If this was an Array, for example, I could simply assign it and it would ensure the underlying contiguous buffer is never mutated.

struct CollectionWrapper<C: Collection> {
   var _underlying: C

   init(contentsOf other: C) where C: RangeReplaceableCollection {
       _underlying = C();
       _underlying.replaceSubrange(_underlying.startIndex..<_underlying.endIndex, with: other)
   }
}

// Would love to do this…

extension CollectionWrapper {
   init(contentsOf other: C) where C: ValueSemantics {
       _underlying = other
   }
}

As the proposal notes (with the File example), these are semantic
considerations which go beyond simple struct/class distinctions:
structs may have reference semantics, and classes may have value
semantics. Would it be possible to model Copyable/move-only in terms
of two new protocols, ValueSemantics and ReferenceSemantics, with
trivial types/non-trivial types given appropriate defaults
(overridable by the programmer, such as for “File”)?

Class types have reference semantics and are still copyable; ownership
is not going to change that. More generally, I don't see how anything
we can do in ownership could ever do something like eliminate the
possibility of a reference-semantics collection from the language.

I think it might not be very related to ownership, but the use of the
word “Copyable” may be a problem. For most programmers when applied to
reference types “Copyable” will connote “Clonable,” i.e. that the
referent (rather than the reference, which is what you're referring to
when you say class types are copyable) can be explicitly copied. So I'm
betting Karl read “Copyable” and was led to this other topic which is of
concern to many people. In that sense, claiming “Copyable” for
ownership without also addressing “Clonable” could be a problem.

The idea behind “Clonable” would be that it gives you a *generic* way to
create a logically *independent* copy of a value, that would work both
for value types and reference types.

--
-Dave

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

1: Collection Composition

As much as the high-level picture is focused on safety, I’d just like to request a strong focus on making sure the eventual ownership system addresses the common issues like avoiding copies when mutating through optionals, or when nesting collections, and so on.

Avoiding copies is basically the entire core of the proposal. The interaction with safety is that sometimes copies are necessary to avoid memory unsafety, and this proposal recommends an approach that minimizes that.

Thanks for the extend reply and sorry for my own belated reply.

What motivated me to ask for more focus on the common cases was that it was unclear from reading the proposal whether the `inout` binding--as used in `for inout`--could be used with `if`: in other words, we currently have `if let` and `if var`, but will we have `if inout` under this proposal?

I am embarassed to admit I honestly couldn't tell if the proposal didn't contain any examples of `if inout` due to (a) the capability following trivially from the rest of the proposal or (b) the capability falling outside the scope of the proposal.

I just didn't think about it. "if inout" is quite supportable under the basic model.

The technical obstacle to this is just that, so far, we've tried to make language features like for-loops use formal protocols.

An iteration protocol is going to have requirements like this:
  generator iterate() -> Element
  mutating generator iterateMutable() -> inout Element
But there's no valid substitution that makes '(Key, inout Value)' equal either 'Element' or 'inout Element'. So we'd have to do some special to make this work.

That said, no, there's no intrinsic technical reason this can't be made to work.

The explanation of wanting to stick to formal protocols makes perfect sense. I don’t think this should be a show-stopper, but I do think it’ll be a common request for a subsequent enhancement.

Even in the interim it seems quite feasible to emulate the capability in any concrete case with enough willingness to crank out boilerplate; thus e.g. if someone truly needs `for (index, inout value) in collection.enumerated()` or `for (a, inout b) in zip(as,bs)` it isn’t as if they’re entirely stuck.

I actually spent some more time thinking about it after I wrote it, and there are some nice things that come out of abandoning formal protocols here. You could allow generators to be overloaded, not by name, but by the "inout-shape" of their return value, so that something like
  for (a, inout b) in ...
would specifically look for a generator like:
  generator() -> (T, inout U)
etc.

3: Mutable Views

It's not sensible to have a type that conditionally conforms to a protocol based on context.

That did indeed seem like a big ask!

I’ll put in an early request to consider

  @moveonly {
    struct File { }
  }

…(or @scope(moveonly) or @context(moveonly) or @dialect(moveonly), etc.).

It’s confusing that `moveonly` essentially applies “inward”—it flags the code *within* a scope as using different assumptions, etc.—in contrast to most of the modifiers like `open`, `final`, `mutating`, etc., apply “outward” (by e.g. describing how visible the code in the scope is from other scopes, etc.).

An interesting idea.

However, the requirements of MutableCollection are all 'mutating', so you can't actually use them unless you have a mutable value. So the way to fit what you're asking for into the ownership proposal is to make sure that clients of your view type only have a mutable value when the base was mutable, and the easiest way of getting that is to have the view be vended as storage rather than a return value. If you think about it, a mutable view can't be an independent value anyway, because if code like this could work:

  var grid = ... // note that this is mutable
  var view = grid.traversal(from: p, following: m) // should return a mutable view by design, right?
  view.mutate() // reflected in grid, right?

then it completely destroys the value-type properties of 'grid', because 'view' should really be an independent value.

Without belaboring this, the point is indeed to “destroy the value-type properties of `grid`”, while trying to keep things “safe” by quarantining `view`—and the temporary relaxation of value semantics its existence implies—to a specific scope; thus under the status quo the use-site looks like this:

  var grid = // note that this is mutable
  // also note that `mutatingTraversal` is a `mutating` method...
  grid.mutatingTraversal(from: c, following: m) {
    (inout view: MutableGrid2DTraversal<T>) -> Void
    in
    // ^ this is the only public method that vends `MutableGrid2DTraversal<T>`, and
    // `MutableGrid2DTraversal<T>` has no public constructors, thus `view` is
    // “quarantined” to this scope unless we actively attempt to leak `view`...
    view.mutate()
  }

…which currently has two major drawbacks:

- (a) lots of code duplication: the immutable `Grid2DTraversal<T>` and the mutable `Mutable2DTraversal<T>`
- (b) the “quarantine” isn’t airtight, in that there are at least these three ways a `view` could escape:

  var grid = // …
  var leakedView: MutatingGrid2DTraversal<T>? = nil
  var otherLeakedView = grid.mutatingTraversal(from: c, following: m) {
    (inout view: MutableGrid2DTraversal<T>) -> MutableGrid2DTraversal<T>
    in
    view.mutate()
    // leak #1:
    leakedView = view
    // leak #2:
    self.liveDangerously(leaking: view)
    // leak #3:
    return view
  }

…which imho makes it fair to say “the above approach *encourages* ‘safe’ usage, but can’t *prevent* unsafe usage”.

Under the ownership proposal nothing changes for drawback (a): to stick with the design and behavior sketched above requires distinct types and thus a lot of duplicate or near-duplicate boilerplate.

Only if you insist that you want a traversal to behave like a linked value even when everything about the traversing code makes it look independent. You're trying to shoe-horn in a kind of reference semantics where it doesn't belong.

Look, your goals are an exact match for the basic ownership design here. You have a type (the grid), it has a thing you can get from it (the traversal), that thing has both mutating and non-mutating operations, you want the mutating operations to be usable if and only if the original value was mutable, you don't want to allow simultaneous accesses to the original value while you have the thing, etc. These are exactly the properties you get by default if starting a traversal is just protecting a component of the value via a property/subscript access. Just accept that a let or var containing a traversal is an independent value.

Drawback (b) seems to fare better under the ownership proposal, provided that I make `MutatingGrid2DTraversal` a non-Copyable type; at least AIUI making `MutatingGrid2DTraversal` non-Copyable would effectively close leaks #1, #2, and #3, b/c:

- I would explicitly have to write e.g. `leakedView = move(view)` (a *very* unlikely accident)
- I would also have to supply a "replacement value” for `view` by the end of the scope (impossible due to lack of public constructors)

…at least if I understand correctly. Is this accurate? If accurate, how likely would it be that “failed to provide replacement value for `view` after move” would get caught at compile time?

It would be guaranteed to get caught at compile time.

This leaking issue is something I didn't get into in the manifesto, but we've actually thought a fair amount about it. Generalized accessors come up a lot in the context of array slices, which share a lot of similar properties with your scenario. With array slices, we have some additional problems:
  - arrays are copy-on-write values, and the original array does need to keep its buffer alive while a slice is being accessed
  - slices need to be usable as independent (and copyable) value types; thus at least sometimes they need to have a strong reference to the buffer
  - forming a mutable slice really shouldn't form a second strong reference to the buffer because then mutations of the slice will trigger a buffer copy
The current line of thinking is that maybe we can have explicit copy and move hooks so that e.g. a projected slice could hold an unowned reference to the buffer which would be promoted to a strong reference on move/copy. But that's a lot of complexity, and I don't think it helps you that much.

Your analysis is correct: making the type move-only solves many of these problems, but not all of them because of the ability to move values aside. Rust has a concept of types that can't even be moved, I think; maybe we need to explore that.

John.

···

On Mar 7, 2017, at 11:47 AM, plx via swift-evolution <swift-evolution@swift.org> wrote:

On Feb 27, 2017, at 11:08 AM, John McCall <rjmccall@apple.com <mailto:rjmccall@apple.com>> wrote:

On Feb 25, 2017, at 11:41 AM, plx via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Thanks again for providing such detailed clarifications about this proposal.

The proposal suggests doing this instead with storage, so that the view is logically a (mutable) component of the grid. So on the use side:

  var grid = ...
  inout view = &grid[traversingFrom: p, following: m]
  view.mutate()

and on the declaration side:

  struct Grid2D<T> {
    subscript(traversingFrom corner: Grid2DCorner, following motion: Grid2DMotion) -> Grid2DTraversal<T> {
      read {
        yield Grid2DTraversal(...)
      }
      modify {
        var traversal = Grid2DTraversal(...)
        yield &traversal
        // ensure changes were applied here
      }
    }
  }

If you feel that the aesthetics of this leave something to be desired, that's a totally reasonable thing to discuss.

John.

Anyways, it’s not clear to me, personally, whether or not the above is within the scope of any likely, concrete ownership system that’d follow from the manifesto or not…but if at all possible I’d prefer the eventual ownership system make it reasonable—and reasonably safe—to implement “small-c ‘collection’s that can safely-and-efficiently expose various *mutable* views as big-C `Collection`s”.

Apologies if all of the above considerations have answers that follow trivially from the manifesto; it’s just unclear personally whether the features described in the manifesto would work together to allow something like the above to be implemented more-reasonably than currently the case.

On Feb 17, 2017, at 11:08 AM, John McCall via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Feb 17, 2017, at 4:50 AM, Adrian Zubarev <adrian.zubarev@devandartist.com <mailto:adrian.zubarev@devandartist.com>> wrote:
Hi John, would you mind creating a markdown document for this manifesto in https://github.com/apple/swift/tree/master/docs? :)

Yes, it should go in the repository. That commit is pending, but the in meantime, you can see the document properly rendered at:
  https://github.com/rjmccall/swift/blob/4c67c1d45b6f9649cc39bbb296d63663c1ef841f/docs/OwnershipManifesto.md

John.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

Hi John,

This is fantastic! I’ve been eagerly anticipating this. It looks very much as I was hoping it would. I can’t wait to see this vision come to life!

You only show a mutating generator example. Do you also plan to allow shared generators?

Yes; a generator yielding a shared reference would be used for non-mutating iteration.

One topic you don’t touch on that feels related is the ability to use RAII and rely on deferred deinit to ensure an owned resource isn’t released before the scope exits. I can imagine something like `defer let resource = MyResource()`. It may also be interesting to support “defer only” types to force a compiler error where non-deferred use would be incorrect. What do you think?

My intuition is the use cases for this that I'm aware of are really a mis-use of values. You might design an API that way in C++, where destructors are really the only language mechanism for injecting code into a function "later", but in Swift I think you would want to use either (1) a generalized accessor (if the protected resource could meaningfully be described as a single value) or (2) a callback that explicitly scopes what happens with the resource.

So e.g. if you wanted a Rust-style mutex API, you could do it like so:

moveonly struct Locked<T> {
  var unlockedMemory: T { // probably not the best name
    read {
      mutex.acquire()
      defer { mutex.release() } // There are reasons why I think this needs to be in a defer, but we can talk about them when we get to the detailed proposals for co-routines. I'm still looking for a better language design.
      yield actualMemory // Recall that 'read' yields a shared value, so this doesn't implicitly copy.
    }
    nonmutating modify { // 'nonmutating' is a misnomer, but it's an existing misnomer. Setters are 'mutating' methods by default and normally demand exclusive access to self, but we don't need that here because we're dynamically enforcing exclusive access to the memory, so all we need is shared access, and this is how we express that.
      mutex.acquire()
      defer { mutex.release() }
      yield &actualMemory
    }
  }

  private var mutex: Mutex
  private mutable var actualMemory: T // Make this variable mutable even within nonmutating methods; whether that makes sense is the programmer's problem. I didn't cover this in the proposal, because it's speculative, but it's useful for things like our nonmutating modify. Lots of open questions here.
}

I like this approach. It scopes access to the resource as necessary without requiring nesting.

Or if you wanted a more C-like mutex API, you'd do it like so:

moveonly struct Mutex {
  func withLock<T>(_ function: () throws -> T) rethrows -> T {
    acquire()
    defer { release() }
    return try function()
  }
}

But I just don't see the motivation to try to do this by making a function return a C++-style lock guard.

Do you have a use case which clearly benefits from an exact scoping and really needs to be an independent value?

To be honest, I don’t know. The primary thing I am looking for is a way to encapsulate the scoping of a resource without requiring the introduction of a new lexical scope like is required in your second example.

Yes, I agree that the nesting isn't ideal. In particular, it's awkward to get values out of the scope.

The first approach you show above using generalized accessors may well be able to cover our needs. Thanks for taking time to show the example!

Okay, glad to hear it. If you *do* find something which really benefits from an RAII-style value, please let me know; I'm certainly open to the idea.

Will do. I’ll keep this in the back of my mind and we’ll see if anything bubbles to the surface! :)

···

On Feb 20, 2017, at 3:38 PM, John McCall <rjmccall@apple.com> wrote:

On Feb 20, 2017, at 4:26 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Feb 19, 2017, at 11:24 PM, John McCall <rjmccall@apple.com <mailto:rjmccall@apple.com>> wrote:

On Feb 18, 2017, at 11:08 AM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

John.

John.

On Feb 17, 2017, at 11:08 AM, John McCall via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Feb 17, 2017, at 4:50 AM, Adrian Zubarev <adrian.zubarev@devandartist.com <mailto:adrian.zubarev@devandartist.com>> wrote:
Hi John, would you mind creating a markdown document for this manifesto in https://github.com/apple/swift/tree/master/docs? :)

Yes, it should go in the repository. That commit is pending, but the in meantime, you can see the document properly rendered at:
  https://github.com/rjmccall/swift/blob/4c67c1d45b6f9649cc39bbb296d63663c1ef841f/docs/OwnershipManifesto.md

John.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

Thanks very much for writing this up, it’s very interesting.

The one thing which I think needs improvement is the Copyable protocol. I think that this is

actually part of a larger problem in Swift, which is that we don’t expose enough information to
generic code for it to operate safely. This goes back to people asking for a value-equivalent to the
“class” or “AnyObject” magic protocols.

For example, often you would like to wrap a Collection and keep some information about what it contains. In order to do that generically, you need to ensure a client can hand you an exclusive view of a collection’s contents, so you know they will not mutate underneath your feet.

Currently, that means you need a RangeReplaceableCollection because it includes an empty initialiser which allows you to create a unique copy of the Collection’s contents. It’s a workaround, but it means we have this unnecessary protocol requirement in the standard library, and in the implementation, which is making copies that may not be required. If my CollectionWrapper is initialised with something which has ValueSemantics, I don’t need to create a whole new instance with equal contents to ensure exclusivity of those contents. If this was an Array, for example, I could simply assign it and it would ensure the underlying contiguous buffer is never mutated.

struct CollectionWrapper<C: Collection> {
   var _underlying: C

   init(contentsOf other: C) where C: RangeReplaceableCollection {
       _underlying = C();
       _underlying.replaceSubrange(_underlying.startIndex..<_underlying.endIndex, with: other)
   }
}

// Would love to do this…

extension CollectionWrapper {
   init(contentsOf other: C) where C: ValueSemantics {
       _underlying = other
   }
}

As the proposal notes (with the File example), these are semantic
considerations which go beyond simple struct/class distinctions:
structs may have reference semantics, and classes may have value
semantics. Would it be possible to model Copyable/move-only in terms
of two new protocols, ValueSemantics and ReferenceSemantics, with
trivial types/non-trivial types given appropriate defaults
(overridable by the programmer, such as for “File”)?

Class types have reference semantics and are still copyable; ownership
is not going to change that. More generally, I don't see how anything
we can do in ownership could ever do something like eliminate the
possibility of a reference-semantics collection from the language.

I think it might not be very related to ownership, but the use of the
word “Copyable” may be a problem. For most programmers when applied to
reference types “Copyable” will connote “Clonable,” i.e. that the
referent (rather than the reference, which is what you're referring to
when you say class types are copyable) can be explicitly copied. So I'm
betting Karl read “Copyable” and was led to this other topic which is of
concern to many people. In that sense, claiming “Copyable” for
ownership without also addressing “Clonable” could be a problem.

The idea behind “Clonable” would be that it gives you a *generic* way to
create a logically *independent* copy of a value, that would work both
for value types and reference types.

--
-Dave

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Part of it is the name; partly I was led there by the examples in the
document. For example, a move-only struct with a deinitialiser looks
an awful lot like it’s trying to express ReferenceSemantics, since it
infers identity. How would using a move-only struct differ from using
a class?

The distinction between value and reference semantics lies in what
happens when you copy or assign, and then mutate. You can't observe
that distinction with move-only types because they have no copy or
assignment. Similarly, you can't observe the difference between value
and reference types when types are immutable. Both immutable and
move-only types eliminate half of the operations you need to be able to
see the distinction.

The confusion about identity occurs because Swift (like most OOP
languages these days) tries hard to erase your ability to see the
difference between the reference and the thing it refers to. References
(what John refers to as Class types above) don't have identity, but the
instances they refer to do. So in that sense, move-only instances are
like class instances.

I’ve been thinking that we could do with a collection of magic
protocols to constrain generic code based on type layout in
general. There is a lot of generic code which could benefit from
optimised implementations if they know that T is trivial, for
example. At a higher level, we often want to know whether a particular
type (regardless of struct/class) has reference/value semantics
because we care about exclusivity over its contents.

Some of the constraints of the “law of exclusivity” sound like they
are providing a kind of value semantic of contents at the
variable-level (e.g. shared references are allowed as long as they do
not change the contents). Several of the concepts in the document are
new to me, but at some broad level there appear to be conceptual
similarities.

At the same time, while references to classes are “Copyable” in the
ownership sense, those copies are very different from copies of
structs. For classes, those copies are basically worthless to the
optimiser because it can’t guarantee anything about who else has
references to the instance.

Except when it can, but yeah, it's harder.

I’m not really sure classes actually benefit at all from being
“Copyable”. Perhaps they should be some other, closely-related thing
instead?

Class references have to be “Copyable” or the world falls apart :-)

···

on Mon Mar 06 2017, Karl Wagner <razielim-AT-gmail.com> wrote:

On 6 Mar 2017, at 22:30, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:
on Mon Mar 06 2017, John McCall <swift-evolution@swift.org> wrote:

On Mar 6, 2017, at 3:46 PM, Karl Wagner <razielim-Re5JQEeQqe8AvxtiuMwx3w@public.gane.org> wrote:

--
-Dave

1 Like

To put this in code:

magic-protocol HasReferenceSemantics { func retain() }
magic-protocol MoveOnly: HasReferenceSemantics {}

magic-protocol HasValueSemantics {}
magic-protocol Copyable: HasValueSemantics { func copy() -> Self }

extension MoveOnly {
mutating func move() -> Self {
    defer { self.deinitialise() }
    return self
}
}

imaginary-struct Reference<To: HasReferenceSemantics>: HasValueSemantics {
    let referencedThing: To

    init(_ p: inout To) where To: MoveOnly {
        referencedThing = p.move()
    }
    init(_ p: To) {
        p.retain()
        referencedThing = p
    }
    func copy() -> Self {
       referencedThing.retain()
       return self
    }
}

imaginary-class Shared<Thing: HasValueSemanics>: HasReferenceSemantics {
    let sharedThing: Thing
    func retain() { /* no-op */ }
}

That’s basically my understanding from the document. Is that more-or-less correct?

- Karl

···

On 7 Mar 2017, at 02:12, Karl Wagner <karl.swift@springsup.com> wrote:

On 6 Mar 2017, at 22:30, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Mon Mar 06 2017, John McCall <swift-evolution@swift.org> wrote:

On Mar 6, 2017, at 3:46 PM, Karl Wagner <razielim-Re5JQEeQqe8AvxtiuMwx3w@public.gane.org> wrote:
Thanks very much for writing this up, it’s very interesting.

The one thing which I think needs improvement is the Copyable protocol. I think that this is

actually part of a larger problem in Swift, which is that we don’t expose enough information to
generic code for it to operate safely. This goes back to people asking for a value-equivalent to the
“class” or “AnyObject” magic protocols.

For example, often you would like to wrap a Collection and keep some information about what it contains. In order to do that generically, you need to ensure a client can hand you an exclusive view of a collection’s contents, so you know they will not mutate underneath your feet.

Currently, that means you need a RangeReplaceableCollection because it includes an empty initialiser which allows you to create a unique copy of the Collection’s contents. It’s a workaround, but it means we have this unnecessary protocol requirement in the standard library, and in the implementation, which is making copies that may not be required. If my CollectionWrapper is initialised with something which has ValueSemantics, I don’t need to create a whole new instance with equal contents to ensure exclusivity of those contents. If this was an Array, for example, I could simply assign it and it would ensure the underlying contiguous buffer is never mutated.

struct CollectionWrapper<C: Collection> {
  var _underlying: C

  init(contentsOf other: C) where C: RangeReplaceableCollection {
      _underlying = C();
      _underlying.replaceSubrange(_underlying.startIndex..<_underlying.endIndex, with: other)
  }
}

// Would love to do this…

extension CollectionWrapper {
  init(contentsOf other: C) where C: ValueSemantics {
      _underlying = other
  }
}

As the proposal notes (with the File example), these are semantic
considerations which go beyond simple struct/class distinctions:
structs may have reference semantics, and classes may have value
semantics. Would it be possible to model Copyable/move-only in terms
of two new protocols, ValueSemantics and ReferenceSemantics, with
trivial types/non-trivial types given appropriate defaults
(overridable by the programmer, such as for “File”)?

Class types have reference semantics and are still copyable; ownership
is not going to change that. More generally, I don't see how anything
we can do in ownership could ever do something like eliminate the
possibility of a reference-semantics collection from the language.

I think it might not be very related to ownership, but the use of the
word “Copyable” may be a problem. For most programmers when applied to
reference types “Copyable” will connote “Clonable,” i.e. that the
referent (rather than the reference, which is what you're referring to
when you say class types are copyable) can be explicitly copied. So I'm
betting Karl read “Copyable” and was led to this other topic which is of
concern to many people. In that sense, claiming “Copyable” for
ownership without also addressing “Clonable” could be a problem.

The idea behind “Clonable” would be that it gives you a *generic* way to
create a logically *independent* copy of a value, that would work both
for value types and reference types.

--
-Dave

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Part of it is the name; partly I was led there by the examples in the document. For example, a move-only struct with a deinitialiser looks an awful lot like it’s trying to express ReferenceSemantics, since it infers identity. How would using a move-only struct differ from using a class?

I’ve been thinking that we could do with a collection of magic protocols to constrain generic code based on type layout in general. There is a lot of generic code which could benefit from optimised implementations if they know that T is trivial, for example. At a higher level, we often want to know whether a particular type (regardless of struct/class) has reference/value semantics because we care about exclusivity over its contents.

Some of the constraints of the “law of exclusivity” sound like they are providing a kind of value semantic of contents at the variable-level (e.g. shared references are allowed as long as they do not change the contents). Several of the concepts in the document are new to me, but at some broad level there appear to be conceptual similarities.

At the same time, while references to classes are “Copyable” in the ownership sense, those copies are very different from copies of structs. For classes, those copies are basically worthless to the optimiser because it can’t guarantee anything about who else has references to the instance. I’m not really sure classes actually benefit at all from being “Copyable”. Perhaps they should be some other, closely-related thing instead?

- Karl

1: Collection Composition

As much as the high-level picture is focused on safety, I’d just like to request a strong focus on making sure the eventual ownership system addresses the common issues like avoiding copies when mutating through optionals, or when nesting collections, and so on.

Avoiding copies is basically the entire core of the proposal. The interaction with safety is that sometimes copies are necessary to avoid memory unsafety, and this proposal recommends an approach that minimizes that.

Thanks for the extend reply and sorry for my own belated reply.

What motivated me to ask for more focus on the common cases was that it was unclear from reading the proposal whether the `inout` binding--as used in `for inout`--could be used with `if`: in other words, we currently have `if let` and `if var`, but will we have `if inout` under this proposal?

I am embarassed to admit I honestly couldn't tell if the proposal didn't contain any examples of `if inout` due to (a) the capability following trivially from the rest of the proposal or (b) the capability falling outside the scope of the proposal.

I just didn't think about it. "if inout" is quite supportable under the basic model.

The technical obstacle to this is just that, so far, we've tried to make language features like for-loops use formal protocols.

An iteration protocol is going to have requirements like this:
  generator iterate() -> Element
  mutating generator iterateMutable() -> inout Element
But there's no valid substitution that makes '(Key, inout Value)' equal either 'Element' or 'inout Element'. So we'd have to do some special to make this work.

That said, no, there's no intrinsic technical reason this can't be made to work.

The explanation of wanting to stick to formal protocols makes perfect sense. I don’t think this should be a show-stopper, but I do think it’ll be a common request for a subsequent enhancement.

Even in the interim it seems quite feasible to emulate the capability in any concrete case with enough willingness to crank out boilerplate; thus e.g. if someone truly needs `for (index, inout value) in collection.enumerated()` or `for (a, inout b) in zip(as,bs)` it isn’t as if they’re entirely stuck.

I actually spent some more time thinking about it after I wrote it, and there are some nice things that come out of abandoning formal protocols here. You could allow generators to be overloaded, not by name, but by the "inout-shape" of their return value, so that something like
  for (a, inout b) in ...
would specifically look for a generator like:
  generator() -> (T, inout U)
etc.

Would it be valid write:

var iter = foo.makeIterator()
inout x = iter.next()

If not, are special case `inout` returns really the right way to model this feature? It seems that a inout-map interface would be better as it doesn't require any special case language features.

Or am I misunderstanding, and `inout` reruns in the general case are supported by this proposal? If so, `inout` seems like a poor name.

···

On Mar 7, 2017, at 10:02 AM, John McCall via swift-evolution <swift-evolution@swift.org> wrote:

On Mar 7, 2017, at 11:47 AM, plx via swift-evolution <swift-evolution@swift.org> wrote:

On Feb 27, 2017, at 11:08 AM, John McCall <rjmccall@apple.com> wrote:
On Feb 25, 2017, at 11:41 AM, plx via swift-evolution <swift-evolution@swift.org> wrote:

3: Mutable Views

It's not sensible to have a type that conditionally conforms to a protocol based on context.

That did indeed seem like a big ask!

I’ll put in an early request to consider

  @moveonly {
    struct File { }
  }

…(or @scope(moveonly) or @context(moveonly) or @dialect(moveonly), etc.).

It’s confusing that `moveonly` essentially applies “inward”—it flags the code *within* a scope as using different assumptions, etc.—in contrast to most of the modifiers like `open`, `final`, `mutating`, etc., apply “outward” (by e.g. describing how visible the code in the scope is from other scopes, etc.).

An interesting idea.

However, the requirements of MutableCollection are all 'mutating', so you can't actually use them unless you have a mutable value. So the way to fit what you're asking for into the ownership proposal is to make sure that clients of your view type only have a mutable value when the base was mutable, and the easiest way of getting that is to have the view be vended as storage rather than a return value. If you think about it, a mutable view can't be an independent value anyway, because if code like this could work:

  var grid = ... // note that this is mutable
  var view = grid.traversal(from: p, following: m) // should return a mutable view by design, right?
  view.mutate() // reflected in grid, right?

then it completely destroys the value-type properties of 'grid', because 'view' should really be an independent value.

Without belaboring this, the point is indeed to “destroy the value-type properties of `grid`”, while trying to keep things “safe” by quarantining `view`—and the temporary relaxation of value semantics its existence implies—to a specific scope; thus under the status quo the use-site looks like this:

  var grid = // note that this is mutable
  // also note that `mutatingTraversal` is a `mutating` method...
  grid.mutatingTraversal(from: c, following: m) {
    (inout view: MutableGrid2DTraversal<T>) -> Void
    in
    // ^ this is the only public method that vends `MutableGrid2DTraversal<T>`, and
    // `MutableGrid2DTraversal<T>` has no public constructors, thus `view` is
    // “quarantined” to this scope unless we actively attempt to leak `view`...
    view.mutate()
  }

…which currently has two major drawbacks:

- (a) lots of code duplication: the immutable `Grid2DTraversal<T>` and the mutable `Mutable2DTraversal<T>`
- (b) the “quarantine” isn’t airtight, in that there are at least these three ways a `view` could escape:

  var grid = // …
  var leakedView: MutatingGrid2DTraversal<T>? = nil
  var otherLeakedView = grid.mutatingTraversal(from: c, following: m) {
    (inout view: MutableGrid2DTraversal<T>) -> MutableGrid2DTraversal<T>
    in
    view.mutate()
    // leak #1:
    leakedView = view
    // leak #2:
    self.liveDangerously(leaking: view)
    // leak #3:
    return view
  }

…which imho makes it fair to say “the above approach *encourages* ‘safe’ usage, but can’t *prevent* unsafe usage”.

Under the ownership proposal nothing changes for drawback (a): to stick with the design and behavior sketched above requires distinct types and thus a lot of duplicate or near-duplicate boilerplate.

Only if you insist that you want a traversal to behave like a linked value even when everything about the traversing code makes it look independent. You're trying to shoe-horn in a kind of reference semantics where it doesn't belong.

Look, your goals are an exact match for the basic ownership design here. You have a type (the grid), it has a thing you can get from it (the traversal), that thing has both mutating and non-mutating operations, you want the mutating operations to be usable if and only if the original value was mutable, you don't want to allow simultaneous accesses to the original value while you have the thing, etc. These are exactly the properties you get by default if starting a traversal is just protecting a component of the value via a property/subscript access. Just accept that a let or var containing a traversal is an independent value.

Drawback (b) seems to fare better under the ownership proposal, provided that I make `MutatingGrid2DTraversal` a non-Copyable type; at least AIUI making `MutatingGrid2DTraversal` non-Copyable would effectively close leaks #1, #2, and #3, b/c:

- I would explicitly have to write e.g. `leakedView = move(view)` (a *very* unlikely accident)
- I would also have to supply a "replacement value” for `view` by the end of the scope (impossible due to lack of public constructors)

…at least if I understand correctly. Is this accurate? If accurate, how likely would it be that “failed to provide replacement value for `view` after move” would get caught at compile time?

It would be guaranteed to get caught at compile time.

This leaking issue is something I didn't get into in the manifesto, but we've actually thought a fair amount about it. Generalized accessors come up a lot in the context of array slices, which share a lot of similar properties with your scenario. With array slices, we have some additional problems:
  - arrays are copy-on-write values, and the original array does need to keep its buffer alive while a slice is being accessed
  - slices need to be usable as independent (and copyable) value types; thus at least sometimes they need to have a strong reference to the buffer
  - forming a mutable slice really shouldn't form a second strong reference to the buffer because then mutations of the slice will trigger a buffer copy
The current line of thinking is that maybe we can have explicit copy and move hooks so that e.g. a projected slice could hold an unowned reference to the buffer which would be promoted to a strong reference on move/copy. But that's a lot of complexity, and I don't think it helps you that much.

Your analysis is correct: making the type move-only solves many of these problems, but not all of them because of the ability to move values aside. Rust has a concept of types that can't even be moved, I think; maybe we need to explore that.

John.

Thanks again for providing such detailed clarifications about this proposal.

The proposal suggests doing this instead with storage, so that the view is logically a (mutable) component of the grid. So on the use side:

  var grid = ...
  inout view = &grid[traversingFrom: p, following: m]
  view.mutate()

and on the declaration side:

  struct Grid2D<T> {
    subscript(traversingFrom corner: Grid2DCorner, following motion: Grid2DMotion) -> Grid2DTraversal<T> {
      read {
        yield Grid2DTraversal(...)
      }
      modify {
        var traversal = Grid2DTraversal(...)
        yield &traversal
        // ensure changes were applied here
      }
    }
  }

If you feel that the aesthetics of this leave something to be desired, that's a totally reasonable thing to discuss.

John.

Anyways, it’s not clear to me, personally, whether or not the above is within the scope of any likely, concrete ownership system that’d follow from the manifesto or not…but if at all possible I’d prefer the eventual ownership system make it reasonable—and reasonably safe—to implement “small-c ‘collection’s that can safely-and-efficiently expose various *mutable* views as big-C `Collection`s”.

Apologies if all of the above considerations have answers that follow trivially from the manifesto; it’s just unclear personally whether the features described in the manifesto would work together to allow something like the above to be implemented more-reasonably than currently the case.

On Feb 17, 2017, at 11:08 AM, John McCall via swift-evolution <swift-evolution@swift.org> wrote:

On Feb 17, 2017, at 4:50 AM, Adrian Zubarev <adrian.zubarev@devandartist.com> wrote:
Hi John, would you mind creating a markdown document for this manifesto in https://github.com/apple/swift/tree/master/docs? :)

Yes, it should go in the repository. That commit is pending, but the in meantime, you can see the document properly rendered at:
  https://github.com/rjmccall/swift/blob/4c67c1d45b6f9649cc39bbb296d63663c1ef841f/docs/OwnershipManifesto.md

John.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

1: Collection Composition

As much as the high-level picture is focused on safety, I’d just like to request a strong focus on making sure the eventual ownership system addresses the common issues like avoiding copies when mutating through optionals, or when nesting collections, and so on.

Avoiding copies is basically the entire core of the proposal. The interaction with safety is that sometimes copies are necessary to avoid memory unsafety, and this proposal recommends an approach that minimizes that.

Thanks for the extend reply and sorry for my own belated reply.

What motivated me to ask for more focus on the common cases was that it was unclear from reading the proposal whether the `inout` binding--as used in `for inout`--could be used with `if`: in other words, we currently have `if let` and `if var`, but will we have `if inout` under this proposal?

I am embarassed to admit I honestly couldn't tell if the proposal didn't contain any examples of `if inout` due to (a) the capability following trivially from the rest of the proposal or (b) the capability falling outside the scope of the proposal.

I just didn't think about it. "if inout" is quite supportable under the basic model.

The technical obstacle to this is just that, so far, we've tried to make language features like for-loops use formal protocols.

An iteration protocol is going to have requirements like this:
  generator iterate() -> Element
  mutating generator iterateMutable() -> inout Element
But there's no valid substitution that makes '(Key, inout Value)' equal either 'Element' or 'inout Element'. So we'd have to do some special to make this work.

That said, no, there's no intrinsic technical reason this can't be made to work.

The explanation of wanting to stick to formal protocols makes perfect sense. I don’t think this should be a show-stopper, but I do think it’ll be a common request for a subsequent enhancement.

Even in the interim it seems quite feasible to emulate the capability in any concrete case with enough willingness to crank out boilerplate; thus e.g. if someone truly needs `for (index, inout value) in collection.enumerated()` or `for (a, inout b) in zip(as,bs)` it isn’t as if they’re entirely stuck.

I actually spent some more time thinking about it after I wrote it, and there are some nice things that come out of abandoning formal protocols here. You could allow generators to be overloaded, not by name, but by the "inout-shape" of their return value, so that something like
  for (a, inout b) in ...
would specifically look for a generator like:
  generator() -> (T, inout U)
etc.

Would it be valid write:

var iter = foo.makeIterator()
inout x = iter.next()

If not, are special case `inout` returns really the right way to model this feature? It seems that a inout-map interface would be better as it doesn't require any special case language features.

Or am I misunderstanding, and `inout` reruns in the general case are supported by this proposal? If so, `inout` seems like a poor name.

inout returns are not supported, for reasons covered at length in the manifesto. The specific sub-proposal for for loops is to base them around co-routines, for which inouts would be supported, and for which the "inout" name is somewhat more appropriate (because, you might say, exclusive access to the value in the variable yielded inout is temporarily passed into the caller of the co-routine and then back out to the co-routine). Co-routine yields are quite different from returns in a number of ways.

John.

···

On Mar 7, 2017, at 1:28 PM, jaden.geller@gmail.com wrote:
On Mar 7, 2017, at 10:02 AM, John McCall via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Mar 7, 2017, at 11:47 AM, plx via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Feb 27, 2017, at 11:08 AM, John McCall <rjmccall@apple.com <mailto:rjmccall@apple.com>> wrote:

On Feb 25, 2017, at 11:41 AM, plx via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

3: Mutable Views

It's not sensible to have a type that conditionally conforms to a protocol based on context.

That did indeed seem like a big ask!

I’ll put in an early request to consider

  @moveonly {
    struct File { }
  }

…(or @scope(moveonly) or @context(moveonly) or @dialect(moveonly), etc.).

It’s confusing that `moveonly` essentially applies “inward”—it flags the code *within* a scope as using different assumptions, etc.—in contrast to most of the modifiers like `open`, `final`, `mutating`, etc., apply “outward” (by e.g. describing how visible the code in the scope is from other scopes, etc.).

An interesting idea.

However, the requirements of MutableCollection are all 'mutating', so you can't actually use them unless you have a mutable value. So the way to fit what you're asking for into the ownership proposal is to make sure that clients of your view type only have a mutable value when the base was mutable, and the easiest way of getting that is to have the view be vended as storage rather than a return value. If you think about it, a mutable view can't be an independent value anyway, because if code like this could work:

  var grid = ... // note that this is mutable
  var view = grid.traversal(from: p, following: m) // should return a mutable view by design, right?
  view.mutate() // reflected in grid, right?

then it completely destroys the value-type properties of 'grid', because 'view' should really be an independent value.

Without belaboring this, the point is indeed to “destroy the value-type properties of `grid`”, while trying to keep things “safe” by quarantining `view`—and the temporary relaxation of value semantics its existence implies—to a specific scope; thus under the status quo the use-site looks like this:

  var grid = // note that this is mutable
  // also note that `mutatingTraversal` is a `mutating` method...
  grid.mutatingTraversal(from: c, following: m) {
    (inout view: MutableGrid2DTraversal<T>) -> Void
    in
    // ^ this is the only public method that vends `MutableGrid2DTraversal<T>`, and
    // `MutableGrid2DTraversal<T>` has no public constructors, thus `view` is
    // “quarantined” to this scope unless we actively attempt to leak `view`...
    view.mutate()
  }

…which currently has two major drawbacks:

- (a) lots of code duplication: the immutable `Grid2DTraversal<T>` and the mutable `Mutable2DTraversal<T>`
- (b) the “quarantine” isn’t airtight, in that there are at least these three ways a `view` could escape:

  var grid = // …
  var leakedView: MutatingGrid2DTraversal<T>? = nil
  var otherLeakedView = grid.mutatingTraversal(from: c, following: m) {
    (inout view: MutableGrid2DTraversal<T>) -> MutableGrid2DTraversal<T>
    in
    view.mutate()
    // leak #1:
    leakedView = view
    // leak #2:
    self.liveDangerously(leaking: view)
    // leak #3:
    return view
  }

…which imho makes it fair to say “the above approach *encourages* ‘safe’ usage, but can’t *prevent* unsafe usage”.

Under the ownership proposal nothing changes for drawback (a): to stick with the design and behavior sketched above requires distinct types and thus a lot of duplicate or near-duplicate boilerplate.

Only if you insist that you want a traversal to behave like a linked value even when everything about the traversing code makes it look independent. You're trying to shoe-horn in a kind of reference semantics where it doesn't belong.

Look, your goals are an exact match for the basic ownership design here. You have a type (the grid), it has a thing you can get from it (the traversal), that thing has both mutating and non-mutating operations, you want the mutating operations to be usable if and only if the original value was mutable, you don't want to allow simultaneous accesses to the original value while you have the thing, etc. These are exactly the properties you get by default if starting a traversal is just protecting a component of the value via a property/subscript access. Just accept that a let or var containing a traversal is an independent value.

Drawback (b) seems to fare better under the ownership proposal, provided that I make `MutatingGrid2DTraversal` a non-Copyable type; at least AIUI making `MutatingGrid2DTraversal` non-Copyable would effectively close leaks #1, #2, and #3, b/c:

- I would explicitly have to write e.g. `leakedView = move(view)` (a *very* unlikely accident)
- I would also have to supply a "replacement value” for `view` by the end of the scope (impossible due to lack of public constructors)

…at least if I understand correctly. Is this accurate? If accurate, how likely would it be that “failed to provide replacement value for `view` after move” would get caught at compile time?

It would be guaranteed to get caught at compile time.

This leaking issue is something I didn't get into in the manifesto, but we've actually thought a fair amount about it. Generalized accessors come up a lot in the context of array slices, which share a lot of similar properties with your scenario. With array slices, we have some additional problems:
  - arrays are copy-on-write values, and the original array does need to keep its buffer alive while a slice is being accessed
  - slices need to be usable as independent (and copyable) value types; thus at least sometimes they need to have a strong reference to the buffer
  - forming a mutable slice really shouldn't form a second strong reference to the buffer because then mutations of the slice will trigger a buffer copy
The current line of thinking is that maybe we can have explicit copy and move hooks so that e.g. a projected slice could hold an unowned reference to the buffer which would be promoted to a strong reference on move/copy. But that's a lot of complexity, and I don't think it helps you that much.

Your analysis is correct: making the type move-only solves many of these problems, but not all of them because of the ability to move values aside. Rust has a concept of types that can't even be moved, I think; maybe we need to explore that.

John.

Thanks again for providing such detailed clarifications about this proposal.

The proposal suggests doing this instead with storage, so that the view is logically a (mutable) component of the grid. So on the use side:

  var grid = ...
  inout view = &grid[traversingFrom: p, following: m]
  view.mutate()

and on the declaration side:

  struct Grid2D<T> {
    subscript(traversingFrom corner: Grid2DCorner, following motion: Grid2DMotion) -> Grid2DTraversal<T> {
      read {
        yield Grid2DTraversal(...)
      }
      modify {
        var traversal = Grid2DTraversal(...)
        yield &traversal
        // ensure changes were applied here
      }
    }
  }

If you feel that the aesthetics of this leave something to be desired, that's a totally reasonable thing to discuss.

John.

Anyways, it’s not clear to me, personally, whether or not the above is within the scope of any likely, concrete ownership system that’d follow from the manifesto or not…but if at all possible I’d prefer the eventual ownership system make it reasonable—and reasonably safe—to implement “small-c ‘collection’s that can safely-and-efficiently expose various *mutable* views as big-C `Collection`s”.

Apologies if all of the above considerations have answers that follow trivially from the manifesto; it’s just unclear personally whether the features described in the manifesto would work together to allow something like the above to be implemented more-reasonably than currently the case.

On Feb 17, 2017, at 11:08 AM, John McCall via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Feb 17, 2017, at 4:50 AM, Adrian Zubarev <adrian.zubarev@devandartist.com <mailto:adrian.zubarev@devandartist.com>> wrote:
Hi John, would you mind creating a markdown document for this manifesto in https://github.com/apple/swift/tree/master/docs? :)

Yes, it should go in the repository. That commit is pending, but the in meantime, you can see the document properly rendered at:
  https://github.com/rjmccall/swift/blob/4c67c1d45b6f9649cc39bbb296d63663c1ef841f/docs/OwnershipManifesto.md

John.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

What is the status of ownership in Swift?

A couple of days ago protocol “Sendable” in SE-302 was accepted, I got the impression that this might be a building stone in Ownership implementation? Or am I completely lost?

Is Ownership on any roadmap with a coarsed grained scheduled release?

10 Likes

ARC became quick enough to be bothered? Swift Performance - #105 by Joe_Groff
other optimizations are also either done or possible (e.g. immortal objects or thread bound objects).