How is __shared keyword supposed to work in Swift 5.0?

@jrose Confirm. I haven't been updating the ownership manifesto as changes go in, but that's probably a good idea.

4 Likes

The __shared keyword is simply not the same thing as "Shared values" explained in the Ownership Manifesto. It does not change the argument passing semantics and is not analagous to Rust's immutable borrow. Your argument is still passed by value. That's the reason you don't see any exclusivity violation.

@John_McCall , if the Ownership Manifesto is updated it should really say that __shared does not implement shared values.

__shared is really an ABI convention that tells the compiler to destroy references on the caller vs. callee side. It's nothing more than that.

The terminology will need to be straightened out before we expose either __shared or move only types as public features.

5 Likes

You’re encountering this known issue:

Swift command line projects crash on launch with “dyld: Library not loaded” errors. (46824656) Workaround: Add a user-defined build setting SWIFT_FORCE_STATIC_LINK_STDLIB=YES .

2 Likes

The implicit copying behavior seems to me like it's independent of the argument passing semantics. We copy when we have to because, unlike Rust, our model introduces implicit copies where necessary to satisfy the conventions of the operations being performed instead of raising borrow check errors. When the argument to a shared parameter is sharably borrowable without copying, then we can and in some situations do share the value. The fact that it's passed by value is an optimization allowed because most of our types are address-independent and can be moved around freely in memory; a shared class reference still represents at the semantic level shared ownership of a single strong reference to the underlying object, for instance. If we added move-only types, or the ability to opt in to requiring explicit copies, then it seems like the only thing that changes is that the implicit copies become errors if they can't be avoided. The convention would otherwise be identical.

2 Likes

@Joe_Groff, @jrose

The implicit copying behavior seems to me like it's independent of the argument passing semantics.

No, the copying behavior changes semantics as the example above makes obvious.

fact that it's passed by value is an optimization allowed

No, no. The argument absolutely must be passed by value per the language semantics. Passing by reference in the above code would print the wrong value!

The OP's question is why doesn't Swift __shared have the semantics of Rust's immutable borrow. The answer is that was not designed to. Literally the only thing it does is force a +0 calling convention at the ABI level. The source of confusion is that Swift's terminology and documentation are currently misleading.

The __shared keyword does not today and cannot ever guarantee that the argument won't be copied. Nor will __shared ever be sufficient to enforce borrowed value semantics (We aren't going to break the code above in some future release).

The only way we will ever get the semantics that Max expects for copyable types is by introducing a new keyword.

I am strongly opposed to publicly documenting the __shared keyword. It will cause unending confusion with the new keyword that we introduce to actually do whatever people now think __shared does. We can call it unowned and document that it is the required convention for borrowed values, but otherwise has absolutely no semantic effect and certainly does not provide pass-by-reference semantics.

2 Likes

Which example? If you're referring to @Max_Desiatov's OP, then it indicates that the caller must copy x in order to satisfy the constraints of f's interface.

But you can't pass by reference in this case, because it would violate the borrow model. The code would be illegal if x were move-only. The fact that x is copyable is what allows the caller to insert the copies necessary to form the call. That's independent of whether the callee takes its arguments shared.

Please do!

I'm a normal Swift user trying to understand the semantics of this code--does the code compile, and what value is printed? What happens on the caller or callee side at run time is neither part of the language semantics nor relevant to me.

func f(sh: shared Int, io: inout Int) {
  io += 1
  print(sh)
}

var x = 5
f(sh: x, io: &x)

The possiblities are:

  1. The shared argument is an immutable borrow and the code is illegal.

  2. The shared argument has pass-by-reference semantics.

  3. shared does not affect program semantics for any code that I can write today.

First I read Swift's documentation. That leads me to the first possiblity. __shared -> "shared values" -> "immutable borrow". But the code does compile. What could this mean?

The only reasonable assumption now is that __shared provides pass-by-reference semantics, what else could it possibly mean? After all, language designers just told me that:

  • __shared does affect semantics by suppressing copies!
  • "__shared acts like a borrow"!
  • "passed by value is [just] an optimization"!

How could this be more misleading?

What language designers should do about this:

Surface __shared as unowned. Document that it is the inverse of owned which allows user control over caller vs. callee ownership of references, that it has no effect on program semantics, that it does not enforce exclusivity (which would be clearly implied by "shared"), and that value types are passed by value either way.

I did try to nip this in the bud last year:

Back to the OP:

But even at run time this prints 5, not 6 as I'd expect if __shared prevents it from creating a copy. Does it mean that I misunderstand the intended use of __shared?

Swift language semantics require that the argument be copied. You were being mislead.

Or another possibility I'm thinking, is __shared keyword currently only "decorative" to annotate the code for ABI stability purposes and there's no real borrow checker available for Swift 5.0?

@Max_Desiatov You are absolutely correct. Congratulations for seeing throught the nonsense even though you had to reverse engineer the true behavior of the language.

3 Likes

Official releases from Apple are built with assertions off to speed up compile times. Also the beta 1 release was branched from the 5.0 branch a few weeks ago. Since no work was done on __shared in a while, do you mind re-testing with the latest 5.0 snapshot and filing bugs?

I think we're in violent agreement about what the documentation ought to say about the language feature today. The issues I have are with the future impact onthe model once we have move-only types. I think __shared can and should be both the calling convention feature we have today and the semantic borrow control we have for move-only values in the future, and that these aren't separate contradictory features. Nonetheless a lot of the confusion and contradictory statements we’re making are probably arising because we aren’t being precise in separating the behavior that applies to Swift today from what we think will apply to future language features.

I don't think that's true. What happens on the caller side is all about language semantics. Part of our goal here AIUI is that, by default, users don't have to think about borrowing or copies in Swift unless they use language features that require these be made explicit.

#2 is an implementation detail. #1 is separable into two claims—"the shared argument is an immutable borrow" is true today; "the code is illegal" is not true, because the language semantics allow for implicit copies to be introduced to make borrows legal.

None of this is an "official" finalized language feature yet, so there isn't really an finalized statement one way or another. We don't have move-only types yet, and we don't have any language mechanisms for suppressing implicit copies, so nobody should be saying that "__shared does affect semantics by suppressing copies", because there's no way to suppress copies today. If we say that somewhere, it's wrong. If there is a means to suppress copies in the future, it ought to be an orthogonal concern to the calling convention.

I think it's important we avoid using the terms "pass by value" and "pass by reference", since we've tried to make sure that for most types in our model addresses are semantically irrelevant. A shared borrow of most Swift values can be passed either by value or reference at the machine level. Other than that, I totally agree with your statement. (We may still want a parenthetical note for clients such as the standard library that are planning for forward compatibility with future move only types about the effect the convention will have on values that don't admit implicit copying.)

1 Like

I'm a normal Swift user trying to understand the semantics of this code--does the code compile, and what value is printed? What happens on the caller or callee side at run time is neither part of the language semantics nor relevant to me.

I don't think that's true. What happens on the caller side is all about language semantics. Part of our goal here AIUI is that, by default, users don't have to think about borrowing or copies in Swift unless they use language features that require these be made explicit.

To me, the normal user, the program semantics boil down to: does the code compile and what will be printed. I honestly don't understand what you're saying.

The possiblities are:

  1. The shared argument is an immutable borrow and the code is illegal.
  2. The shared argument has pass-by-reference semantics.
  3. shared does not affect program semantics for any code that I can write today.

#2 is an implementation detail.

No, it is not. It determines whether the program prints "5" or "6". I can't imagine anyone would think otherwise:

#1 is separable into two claims—"the shared argument is an immutable borrow" is true today; "the code is illegal" is not true, because the language semantics allow for implicit copies to be introduced to make borrows legal.

The first claim is has no semantic meaning if a copy will be introduced. The copy changes the behavior of the program, so of course the user needs to know about it.

1 Like

What determines whether the program prints "5" or "6" is whether a caller passed the same variable as arguments to sh and io. It is illegal to do so because io requires exclusive access, so updates to io within f cannot modify anything other than that argument, and likewise, sh requires shared access to its argument, which implies that neither f nor any other code in the program can modify the argument during the call to f. Therefore, for an address-independent type like Int, it does not matter whether f takes sh by reference or by value, or whether it takes io by reference or by value-result. If the code were instead:

func f(sh: __shared Int, sh2: __shared Int) { ... }

var x = 5
f(sh: x, sh2: x)

then it wouldn't matter whether the arguments were passed by value or reference, or where zero, one, or two copies happened inside the caller.

The copy is necessary to form the call to f in the first place, given the constraints on its parameters.

It's true that "owned", "shared access", and "exclusive access" can be roughly corresponded to "by value", "by const reference", and "by mutable reference" for normal value types (and pretty much all Swift types today), but the distinction will become more important when we have move-only types for more interesting things like concurrency primitives.

2 Likes

Part of our goal here AIUI is that, by default, users don't have to think about borrowing or copies in Swift unless they use language features that require these be made explicit.

If the promise of progressive disclosure is to be met, then it cannot be that the program prints either “5” or “6” contingently. It should print exactly what the following prints:

func f(sh: Int, io: inout Int) {
  io += 1
  print(sh)
}
var x = 5
f(sh: x, io: &x)
1 Like

Yes, that's what I'm saying—either it behaves exactly as today with copyable types, or is an error with move-only types. There would be no circumstance under which sh and io can legally mutably alias regardless of how sh is annotated.

Andy and I met in person and discussed this, it turns out we're in agreement on everything being discussed here. Our messaging around this feature is currently doing a poor job of keeping the intended "layers" of disclosure distinct which is leading to the confusion.

6 Likes

Thanks @Joe_Groff. That's right, there's no debate about how we want the code to behave. The debate is about how people will expect the code to behave based on the name __shared and how it was described earlier in this thread and in the current documentation. I'll try restating my position again in case it wasn't clear (yes, I think it's pretty important).

I want to unambiguously communicate the argument passing semantics for a trivial function application. I don't expect users to pick apart separate complex caller/callee semantics and try to infer the simple argument passing semantics from that.

Tweaking the original example:

func f(sh: __shared Int, f: () -> ()) {
  f()
  print(sh)
}

var x = 5
let f = { x += 1 }
f(sh: x, f: f)

By messaging that __shared is a "shared value" convention, we encourage people to interpret it as Rust's 'immutable borrow'. The only other possible interpretation, given that we've messaged it as a change to argument passing semantics would be C++ 'const &'. Both are wrong.

The counter argument is that users should know that it's impossible to pass a mutable variable of copyable type as __shared, and therefore they should expect the compiler to create a new invisible variable that can then be "shared" with the callee. Even if I could accept this logic at face value, it is totally counter intuitive. Swift could easily have made it legal to pass a mutable variable of copyable type as __shared with exclusivity enforced. The fact that it doesn't work this way then just looks like an implementation quirk that no one would expect.

In my mind, when the user writes f(sh: x, f: f), at the highest level of semantics it should mean that x is the "variable" that will be bound to the sh argument. It's unnecessarily confusing to think about this as the caller implicitly passing a new invisible variable to sh. In fact, I would say that, by creating that temporary phantom "variable", the compiler contradicts any reasonable interpretation of __shared.

Instead, I would progressively explain Swift's conventions this way:

Level 1: Argument Passing

Value types have value semantics, reference types have reference semantics. Swift's argument passing semantics are always pass-by-value for value types where inout means copy-in, copy-out. End of story. Whether a callee declares ownership of the argument does not affect these semantics.

Level 2: Lifetime

For lifetime-managed values (references or move-only types) the user can override whether the caller or callee takes ownership of the argument for the purpose of ending its lifetime. This can reduce the copies required at the implementation level. There are no semantics here. The compiler is free to destroy things out of order.

Level 3: Move-only semantics (straw man)

Now the declaration of ownership can affect semantics simply because variables of move-only types are semantically different depending on whether the variable has ownership. So it's really the implicit ownership conversions that have semantics, not the argument passing itself. For arguments of move-only type:

  • owned to owned: implicit move

  • owned to unowned: implicit borrow (exclusivity enforced)

  • unowned to owned: illegal

For mutable variables with copyable types, all of the above implicitly copy, as pass-by-value semantics dictate. The only one left is:

  • unowned to unowned (any type): no implicit operations

Passing an unowned variable to an unowned argument is the only situation that could reasonably be called "shared".

In the owned to unowned case, even though copyable types will be semantically copied, as an optimization and ABI convention, the implementation can share references by guaranteeing lifetimes. This ABI is required to implement move-only semantics, but does not by itself change program behavior!

Level 4: Explicit borrow and move (straw man)

I think it's important for the programmer to be able to achieve the same argument passing semantics for copyable types that move-only types will implicitly adopt. So, to get the behavior of Rust's immutable borrow we could invent syntax like f(sh: borrow(x), f: f). This would give the user a way to declare an expectation of zero abstraction cost, free the implementation to remove physical copies, and, in the running example above, produce a diagnostic that the code is illegal because f will modify x.

8 Likes

@Joe_Groff, @Slava_Pestov
What do you guys think about introduction of accessors at syntactic level to make easier violation checks, like:

var someVar: @shared = "Always copied upon mutation"
var anotherVar: @unique = "Accessible exclusively within a scope"
var  otherVar: @threaded = "Check evaluation from multiple scopes to enforce correct multi-threading behavior"
//so functions may not provide explicit argument capture scheme, and instead be:
var a: @shared, b: @unique, c: @theaded = 1
func add(_ a: Int, _ b: Int) -> Int { a + b }
add(a, &b) \\ok, b captured, a is copied

Also how about removing the definition of structs and classes and introducing only objects, which can be assigned access attributes as needed?

///some examples here
var num1: @unique = 1
var num2: @shared = 2
var num3: @threaded = 3
func e(&num1){
   p(&num1); u(num1) //error: num1 passed to multiple scopes, which is forbidden for @unique 
}
func e(&num2) {
    p(&num2); u(&num2) //error: num2 cannot be mutated from multiple scopes, use @threaded instead
}
func e(&num3) {
    p(&num3); u(&num3) 
    //building an execution graph at run-time to resolve read/write access collisions
   //and inform about things that would go wrong. Omit in release, if possible
}
//there can be an attribute @nonmutating as well to put constraints on funcs
@nonmutating
func someFunc(_ a: Some)  //someFunc(&value) is error
//and some additional syntax for procedures
[1, 2, 3].&map { $0 + 1 }

I'm sure now (shortly before WWDC) is a terrible time to ask this question, but how goes the process of resolving the ownership model?

1 Like

The original example was purely an inevitable misunderstanding of the __shared keyword.

I think everyone would agree Swift should allow us express shared arguments, and that would trigger an exclusivity violation on this code. I would spell it something like:

func f(sh: Int, io: inout Int) { // __shared is meaningless on an Int
  io += 1
  print(sh)
}

var x = 5

f(sh: borrow(x), io: &x) // <== simultaneous access here

The current __shared keyword allows for shared arguments to be passed to generic APIs in the future but does not force shared arguments today, and never will, which is why it badly needs to be renamed. So, I think the only unresolved question is what to call it.

Independent of argument sharing or move-only types, it's important to be able to override the default convention for argument ownership for performance tuning: to be explicit we should say caller_owned (vs. __shared today) and callee_owned (vs. __owned today)

1 Like

That makes sense. Is there a lack of consensus on this front? I ask, because the Ownership Manifesto seems to be out of date despite the good intentions stated up-thread about updating it.

The Ownership Manifesto uses "shared" in the correct sense everywhere. The current implementation of the __shared keyword is not what was outlined in the manifesto. I think the section starting with:

A function argument can be explicitly declared  `shared`

will need to be amended eventually to reflect the reality that it isn't implementable without breaking source.

I think the only way we'll get shared arguments is

  1. when they are type-directed (move-only types will require sharing)
    or
  2. when the storage reference on the caller side is explicitly shared/borrowed

I'm not an authority though. This is just me talking on the forum. I haven't heard of any update to the plan outside of this forum.