Add shared storage to property wrappers

A normal value can come from anywhere: it can be stored in a property, yes, but it can also be returned from a function, passed as an argument to a function, directly constructed by a call to an initializer, etc. By design, when you're using the value, the only extra information you have besides the value itself is its static type. So yeah, maybe all the values constructed by a particular initializer call have some data in common that could be shared between them; but when you're using that value later, you don't know what initializer call was originally used to construct it, so the only way you can get at that shared data is if it's passed along with the value itself. We could maybe optimize the representation so that it passes the shared data along more efficiently, but we can't completely elide it.

In contrast, we know exactly how a property wrapper is constructed, and we know when we're using it as an "independent" value, and the wrapper itself isn't normally passed around separately (so we could potentially forbid that for a particular wrapper if it was advantageous). So we do have an opportunity to split off the shared data and not store it in the value that's stored inline in the property / local frame / whatever. And that gets property wrappers much closer to parity with built-in modifiers like lazy.

Property wrappers also have an opportunity to construct a different, self-contained value to be passed around externally via the projected value. I think it's reasonable to to forbid passing the optimized wrapper storage around, and suggest using projected value for this purpose instead.

2 Likes

I do think there's a way to extend the idea of shared storage to types more broadly, but I'm concerned that it has sufficiently different trade-offs that it might compromise the feature's usefulness for property wrappers.

If you think of shared storage as extending from a particular construction site (e.g. MyType(x: 1, y: 2, z: 3)), and we know that MyType uses shared storage, then we could validate and construct the shared storage from the shared arguments to that specific initializer-call expression in basically the same way that I described it above for property wrappers:

struct MyType {
  @shared var x: Int, y: Int
  var z: Int
  init(@shared x: Int, @shared y: Int, z: Int) { ... }
  @shared init(x: Int, y: Int) { ... }
}

MyType(x: 1, y: 2, z: 3)
// rewritten to
static let anonymous_shared = MyType.Shared(x: 1, y: 2)
MyType(&anonymous_shared, z: 3)

MyType is then rewritten to carry a pointer to a MyType.Shared, so that you get useful sharing between all the values constructed by a specific expression, but the value can still be used in a normal, first-class way. This doesn't completely eliminate the overhead associated with passing around the shared data, but it may significantly lower it if the shared data is large or expensive to copy. It would be very invasive for programmers to make this sort of change manually.

The thing is, I don't know how to efficiently combine that with what we want to do with property wrappers. The natural representation for MyType is something like the following; notice how all the non-shared properties are stored inline in the struct:

struct MyType {
  struct Shared {
    var x: Int, y: Int
  }
  var shared: UnsafePointer<Shared>
  var z: Int
}

But for property wrappers, we really want to be able to store all the non-shared properties somewhere by themselves and then reconstitute a value of MyType from a reference to that without having to copy any of the non-shared data. For MyType, which just has a single Int of non-shared storage, it wouldn't be a big overhead to copy the Int into the reconstituted value (and then copy it back if we needed the MyType for a potentially-mutating operation), but for an arbitrary type that could be a big deal. This gets into the conversation that @Joe_Groff and I had above about in-place access; he's absolutely right that if we need to copy the non-shared storage back and forth, we may have a hard time eliminating those costs. So for property wrappers we really want a representation more like what I had in my earlier posts:

struct MyType {
  struct Shared {
    var x: Int, y: Int
  }
  struct Instance {
    var z: Int
  }
  var shared: UnsafePointer<Shared>
  var instance: UnsafeMutablePointer<Instance>
}

or at least something like the following, which would allow values to be passed around in a first-class way, but also allow efficient reconstitution as long as they weren't:

struct MyType {
  struct Shared {
    var x: Int, y: Int
  }
  struct Instance {
    var z: Int
  }
  enum InstanceStorage {
    case inline(Instance)
    case separate(UnsafeMutablePointer<Instance>)
  }
  var shared: UnsafePointer<Shared>
  var instance: InstanceStorage // when the value is copied, this always transitions to the inline case
}

But this seems like a substantial burden to impose on the property wrapper use case.

4 Likes

Looks like Isa fixed this typo already (the updated draft is on GitHub)

The initialization of the storage value itself follows the same principles as static variables: it can't use instance variables or methods that depend on self being initialized.

1 Like

What about this scheme for lowering?

extension ClosedRange {
  func clamp(_ value: Bound) -> Bound {
  	return Swift.min(Swift.max(value, self.lowerBound), self.upperBound)
  }
}

protocol ClampedShared {
   associatedtype Value: Comparable
   static var bounds: ClosedRange<Value> { get } 
}

@propertyWrapper
struct Clamped<Value, Shared: ClampedShared> where Shared.Value == Value {
    private var value: Value
    init(wrappedValue: Value) {
        self.value = Shared.bounds.clamp(wrappedValue)
    }
    var wrappedValue: Value {
        get { value }
        set {
            value = Shared.bounds.clamp(newValue) 
        }
    }
}

private enum Hud_bar_Clamped_Shared: ClampedShared {
    static var bounds: ClosedRange<Int> { 0...14 }
}

private enum Hud_foo_Clamped_Shared: ClampedShared {
    static var bounds: ClosedRange<Int> { 1...9 }
}

struct Hud {
  @Clamped<Int, Hud_bar_Clamped_Shared> var bar = 5
  @Clamped<Int, Hud_foo_Clamped_Shared> var foo = 1
}

My thinking behind this goes as following:

Currently in Swift everything related to a nominal type and shared across instances is associated with a type. I was thinking how shared storage could keep this as is and avoid introducing new kind of sharing.

Applying this to the example of Clamped means that it should be lowered into the generic type, with extra generic argument for shared data. We don't have values as generic parameters yet, and I'm assuming we won't get them soon, so some sort of poor-man's solution is needed.

To pass values as a generic parameter, as a workaround, we can wrap the value into a type conforming to a protocol with static requirements. This allows simple mapping from types to values. It is possible to come up with a scheme mapping values back to types, preserving equality, but as far as I can see, it is not needed for this proposal.

This lowering scheme organically answers how to handle values of types with shared values outside the context of property wrappers - they are just passed around as generic values:

func inc<Shared: ClampedShared>(value: inout Clamped<Int, Shared>) {
    value.wrappedValue += 1
}

Using generics it is possible to express requirement that different values must be coming from the declaration of the same property:

func transfer<Shared: ClampedShared>(from: inout Clamped<Int, Shared>, to: inout Clamped<Int, Shared>, amount: Int) {
    from.wrappedValue -= amount
    to.wrappedValue += amount
}

var h1: Hud = ...
var h2: Hud = ...
transfer(from: &h1.foo, to: &h2.foo, amount: 1) // OK
transfer(from: &h2.bar, to: &h1.bar, amount: 1) // OK
transfer(from: &h1.foo, to: &h2.bar, amount: 1) // Error
2 Likes

Ok I think I get what you are saying now! I think you are saying that this particular storage sharing optimization relies on the "typical" use of property wrappers, which is to say that usually you are never going to touch the wrapper type outside of the property declaration site.

And now it seems that you are proposing a compile-time enforcement of this "typical" use of property wrappers when they utilize this storage sharing optimization, I think that makes sense so that one doesn't accidentally escape this optimization without realizing it.

Please correct me if this understanding isn't quite right!

For a normal type, there are two "interesting" places to store information:

  • In each instance of the type (instance properties).
  • Shared by all instances of the type (type properties).

For a property wrapper, there is a third "interesting" place to store information:

  • Shared by all instances of the type that are wrapping the same property.

I suppose that in theory, there might be a case for a more general mechanism to share variables between a subset of instances of a type, but…how would you actually do that? How would you describe which instances should be grouped together? How would a user initialize these variables?

9 Likes

I think this third “interesting” place to store information can be generalized as:

  • Shared by all instances of the type that can be linked to the same property.

I think this is possible (with the right set of restrictions), but I will refrain from providing further detail about how I think this can work until I can come up with a concrete example of where I think it could be useful.

2 Likes

How does this play with Decodable?
Decodable requires init(from decoder: Decoder) throws but this proposal requires an additional @shared storage parameter.

Shared storage would be great to implement something like DefaultCodable from the BetterCodable package without the need to define a custom type for each default value. Because you currently need to define a custom type for each default value, the BetterCodable package includes a lot of different property wrappers for the most common default values like empty array, empty dictionary, false and so on. With shared storage and proper support for Decodable this could be greatly simplified.

4 Likes

The Style property wrapper in the "API-level Property Wrappers on function parameters" section has an init(wrappedValue: UIView) which does not take a @shared storage parameter. How does this work?

Thanks for catching that, this initializer would also take a @shared parameter. I’ll correct it on GitHub

1 Like

As I was thinking about a concrete example, I think possibly I had an "a-ha" moment (and maybe this was obvious to everyone else already):

Ok right, so I think the third interesting place is per-property. It just so happens that there is already a mechanism to attach behavior on a per-property basis, yes that's right, it's ... property wrappers :sweat_smile:

So to get what I wanted, I could just utilize a generic "Sharing" property wrapper, something like this:

@propertyWrapper
struct Sharing<T, U> {
    
    var wrappedValue: T
    
    // @shared
    // Edit: sharedValue should be a let
    var sharedValue: U
}

struct Foo { }

struct Hud {
    @Sharing(sharedValue: 5) var foo = Foo()
}

// Every Hud's "foo" property shares the same value 5
let hud1 = Hud()
let hud2 = Hud()
let hud3 = Hud()

For whatever reason, I just wasn't putting this together before, anyway, thanks to those that tried to enlighten me :slight_smile:

3 Likes

I do like the idea of a "sharedValue" property, it fits nicely with wrappedValue and projectedValue and the general structure of a property wrapper.

1 Like

@Syre I think that's still just instance-level sharing, but with an extra level of encapsulation. You can see this if you add to Hud a way to modify the shared value:

var shared: Int {
    get { _foo.sharedValue }
    set { _foo.sharedValue = newValue }
}
var hud1 = Hud()
var hud2 = Hud()
hud1.shared = 1
hud2.shared // 5

Even though the same property, @Sharing(sharedValue: 5) var foo, is created multiple times, the value is not shared.

That is also the case when the property is created by the identical declaration at different times:

for i in 1...2 {
    var hud = Hud()
    if i == 1 { hud.shared = 1 }
    print(hud.shared)
}
// 1
// 5

For simplicity, we can skip the extra level of encapsulation of Hud and use the property wrapper on the local variable (Swift 5.4+):

for i in 1...2 {
    @Sharing(sharedValue: 5) var hud = Foo()
    if i == 1 { _hud.sharedValue = 1 }
    print(_hud.sharedValue)
}
// 1
// 5

So this is really just an instance-level property.

Here's my take on what this pitch is trying to achieve, based especially on these two statements:

My interpretation of this is that we want declaration-level sharing. We would need a way to uniquely identify a declaration, i.e. give it an ID. This wouldn't even necessarily be a property-wrapped declaration, although property wrappers already provide a syntax for attaching extra pieces of information (like the 0...100 range in the OP) to the principal (wrapped) value, and there's no obvious syntax that we could use for non-PW values that wouldn't be effectively the same as PW syntax, so I don't see a reason to extend this sharing feature somehow to non-PW.

Here's an example, with Swift 5.5, of what I mean:

We manually give each PW use site a unique ID and use it to access a persistent storage.

enum SharedStorage {
    static var dict: [Int: Any] = [:]
}

@propertyWrapper
struct Sharing<T, U> {
    
    var wrappedValue: T
    
    var sharedValue: U {
        get { SharedStorage.dict[id] as! U }
        set { SharedStorage.dict[id] = .some(newValue) }
    }
    
    private var id: Int
    
    init(wrappedValue: T, sharedValue: U, id: Int) {
        self.wrappedValue = wrappedValue
        if !SharedStorage.dict.keys.contains(id) {
            SharedStorage.dict[id] = .some(sharedValue)
        }
        self.id = id
    }
}

struct Foo { }

for i in 1...2 {
    @Sharing(sharedValue: 5, id: 1) var hud = Foo()
    if i == 1 { _hud.sharedValue = 1 }
    print(_hud.sharedValue)
}
// 1
// 1

Things to note:

  • The SharedStorage would be behind the scenes. It could also be more sophisticated by avoiding duplicates and using Copy-on-Write. It could be separate and nested for each PW-type (or scope, in the case of local variables) that makes use of this new feature, as in the OP, which would avoid the need for type-casting from Any, although that's something I suspect could be optimized either way.
  • Everything about the id would be automated by the compiler: generating a unique id for each declaration/PW-application, initializing the PW with that id, checking if the shared value has been previously initialized for this declaration, and accessing the storage with that id.
  • I made sharedValue writable for the sake of demonstrating the 'sharedness' but this could simply be disallowed as per the OP's specifications.
  • Here the sharedValue is separate from the wrappedValue, but I don't see a reason why the wrappedValue itself couldn't be the shared value. AFAICT that would be effectively the same as a static let when on a property, or a static let in whichever local scope when on a variable declaration.
Extra example

The @Clamped example could be written like this:

enum SharedStorage {
    static var dict: [Int: Any] = [:]
}

@propertyWrapper
struct Clamped<V: Comparable> {
    
    private var value: V
    var wrappedValue: V {
        get { value }
        set { value = sharedRange.clamping(newValue) }
    }
    
    var sharedRange: ClosedRange<V> {
        get { SharedStorage.dict[id] as! ClosedRange<V> }
        set { SharedStorage.dict[id] = newValue }
    }
    
    private var id: Int
    
    init(wrappedValue: V, _ sharedRange: ClosedRange<V>, id: Int) {
        if !SharedStorage.dict.keys.contains(id) {
            SharedStorage.dict[id] = .some(sharedRange)
        }
        self.id = id
        self.value = wrappedValue // satisfy definite initialization check
        self.value = self.sharedRange.clamping(wrappedValue)
    }
}

for i in 1...2 {
    @Clamped(1...100, id: 1) var foo1 = 5
    @Clamped(1...100, id: 2) var foo2 = 5
    if i == 1 { _foo1.sharedRange = 25...75 }
    print(_foo1.sharedRange, _foo2.sharedRange)
    // 25...75 1...100
    // 25...75 1...100
}

And the same with compiler magic...

-enum SharedStorage {
-    static var dict: [Int: Any] = [:]
-}
 
 @propertyWrapper
 struct Clamped<V: Comparable> {
     
     private var value: V
     var wrappedValue: V {
         get { value }
         set { value = sharedRange.clamping(newValue) }
     }

+    @shared var sharedRange: ClosedRange<V>
-    var sharedRange: ClosedRange<V> {
-        get { SharedStorage.dict[id] as! ClosedRange<V> }
-        set { SharedStorage.dict[id] = newValue }
-    }
    
-    private var id: Int
     
+    init(wrappedValue: V, _ sharedRange: ClosedRange<V>) {
-    init(wrappedValue: V, _ sharedRange: ClosedRange<V>, id: Int) {
-        if !SharedStorage.dict.keys.contains(id) {
-            SharedStorage.dict[id] = .some(sharedRange)
-        }
-        self.id = id
-        self.value = wrappedValue
+        self.sharedRange = sharedRange
         self.value = self.sharedRange.clamping(wrappedValue)
     }
 }

 for i in 1...2 {
+    @Clamped(1...100) var foo1 = 5
+    @Clamped(1...100) var foo2 = 5
-    @Clamped(1...100, id: 1) var foo1 = 5
-    @Clamped(1...100, id: 2) var foo2 = 5
     if i == 1 { _foo1.sharedRange = 10...15 }
     print(_foo1.sharedRange, _foo2.sharedRange)
 }

Final thoughts:

  • Sharing values based on where in the code they're declared is the key point, at least for me. Currently this requires manually identifying declarations, which is unmaintainable.
  • Restricting them to be read-only could make the feature more manageable without making it useless, as could disallowing you to assign one PW'ed variable to another (as mentioned in the OP), since it's not clear which shared value should survive that, or if that can even safely be done. At least that seems reasonable to me as a starting point, which could later be expanded.
  • There are lots of things I haven't fully considered or just can't wrap my head around, like PW composition, or anything low-level really. I expect heavy scrutiny. :sweat_smile:
  • One particular thing I'm not sure about is how my sugared version hides a lot of things under the hood (the shared storage and the id). The OP seems to allow more control which could be useful.

At least I hope I didn't completely mischaracterize the intent of this feature. Have I? That's my 2 cents anyway.

2 Likes

Whoops, this example has caused a bit of confusion, to be clear, I’m not proposing that sharedValue be a “special” property name akin to wrappedValue or projectedValue, it’s just the name I happened to pick for this “Sharing” wrapper. (Edit: maybe blessing sharedValue in this way is a good idea, but I don't feel strongly one way or the other on it.)
Additionally, sharedValue here was supposed to be a let, not a var.

In other words, this example is intended to follow exactly what is being proposed wrt shared storage in propertyWrappers, nothing more, nothing less.

1 Like

Okay, going with the Ownership Manifesto's definitions, each Hud's foo property shares the same semantic value, which may be enough in this case, and in many others. The compiler could maybe even optimize it so that you actually get a single value instance for all of them (as long as these shared properties are strictly read-only)?

A blessed property name could be good, and maybe it could also provide access to a collection of all the other instances with different values for that shared property, via a special syntax like a projectedValue (similar to the examples in the OP):

private var $sharedValue: [Int] { /* ... */ }
// or
private var sharedValue$storage: [Int] { /* ... */ }

Perhaps my post was a bit too eager as I'm still not 100% sure what the goals are. I should go back to reading and then try to rework my code examples when I have a better understanding.

1 Like

Phrasing with the Ownership Manifest terms, the goal is to add a way for property wrappers to store properties in a way that shares a single value instance on a per declaration basis.

2 Likes

I am sorry if I missed this (I haven't had a chance to read through all the posts above, so maybe this is out of scope or unsupported) but I was wondering if it would be possible to assign a shared value to all instances declared in a given type? For example, I might have a @Preference property wrapper and I want to inject an instance of some type that conforms to my PreferencesType protocol into all instances of @Preference in a given type. Right now, you need to inject it into each property wrapper manually, something like:

struct S {
  @Preference(key: "some_key_1") var key1: Bool
  @Preference(key: "some_key_2") var key2: Int
  @Preference(key: "some_key_3") var key3: String
  
  init(defaults: PreferencesType = UserDefaults.shared) {
    self._key1 = .init(defaults: defaults)
    self._key2 = .init(defaults: defaults)
    self._key3 = .init(defaults: defaults)
  }
  
  // Alternatively
  init(defaults: PreferencesType = UserDefaults.shared) {
    self.defaults = defaults
    configureWrappers()
  }

  private func configureWrappers() {
    $key1.defaults = defaults
    $key2.defaults = defaults
    $key3.defaults = defaults
  }
}

It would be nice if you could do something like this instead:

// Assuming I have declared @shared var defaults: UserDefaultsProtocol? in my property wrapper
init(defaults: UserDefaultsProtocol = UserDefaults.shared) {
   self.$preferencesShared.defaults = defaults
}

Effectively, I want to inject a value (provided to me via an initializer) into all instances of the property wrapper within a given type (and not globally i.e. into every instance throughout the app) to avoid having to write a lot of boilerplate.

I've been talking more with @hborla and @Douglas_Gregor about this, and I think we can structure this into a series of three proposals.

Expanded parameters

The first proposal allows a subsequence of call arguments to be implicitly collected together at compiler time to initialize a single aggregate argument. That is, one could write a function signature something like this:

func addOrbit(options: expanded OrbitOptions)

where OrbitOptions has one or more accessible initializers, like so:

struct OrbitOptions {
  init(distance: Distance<Double>,
       eccentricity: Double = 0,
       inclination: Angle<Double> = 0)
}

Callers could then supply the arguments to build an OrbitOptions directly when calling addOrbit:

addOrbit(distance: .gigameters(778.57), eccentricity: 0.0489)

or they could simply provide the aggregate directly:

addOrbit(options: opts)

This is a relatively straightforward proposal that would be much more broadly useful. I think the main subtlety here is that we would need to carefully define the argument-matching rules (and place restrictions on parameter lists with expanded parameters) so that the type-checker can unambiguously decide which arguments are meant to be collected into the expansion without doing a type analysis.

It is in keeping with Swift's general emphasis on keeping control over call sites in the hands of the API author that this feature would be opt-in, rather than something you could do with any argument. That also helps to keep the build-time impact in check.

To apply it to my proposed lowering, we could change the part of the transformation that extracts a Shared type and instead just expect the wrapper type to define that explicitly. Non-shared initializers would be expected to take a typically-expanded parameter that initializes that storage. That parameter would still need to be marked as shared in some way to get the desired effect; see the next section.

Static parameters

The second proposal is that any parameter could be declared to be static. (I think I prefer the spelling static over shared. This adds a new use of static, but I think it's generally consistent with the existing use in Swift of meaning "global rather than instance/local". If people hate it, we can have that discussion.)

func buildOrbit(options: static OrbitOptions)

A static argument is always passed as a pointer to global immutable memory. It's unclear how best to expose this in the callee; maybe we just allow code to safely take the address of it with &, or maybe we make the parameter implicitly an UnsafePointer (or a new StaticPointer type, or something similar).

On the caller side, a static parameter can be satisfied with a pointer (when using the & operator, maybe?). Otherwise, the argument expression must be independent of context (e.g. no references to local variables), and global storage will be lazily initialized with that expression prior to the call.

As someone with quite a bit of implementation experience with this kind of static-local-initializer feature, I think we want to be careful to avoid making too strong a guarantee about either execution or uniqueness of address. The compiler will make a best effort to evaluate the initializer expression as few times as possible and use as little storage as it can; however, the expression may be evaluated either more or less than a strict model of "once per unique context" might imply, and storage is only guaranteed to be unique to the extent that different memory is required to hold the value. Accordingly, initializer expressions should not be side-effectful, and code should not rely on the storage being 1-1 with calls. For example, if you are in a generic function over <T, U>, and you use MemoryLayout<T>.alignment as a static initializer, the storage address might be different for all pairs of <T,U>, or it might be different only for different values of T, or it might be different only for distinct alignment values, or it might even be shared with completely different initializers that happen to evaluate to the same result. Similarly, if you use UUID() as a static initializer in the same context, you might get different UUIDs for all pairs of <T,U>, or you might get one UUID for all calls from that context, or you might even get one UUID that is shared across different calls that happen to use UUID() as an initializing expression.

While the expanded-parameter feature is largely just build-time work, this feature requires runtime support. It's possible that this could be made to back-deploy, but we can't really guarantee that in advance. This same work could be leveraged to allow static stored properties in generic types.

Applying this to my proposed lowering is straightforward: non-shared initializers would take a static parameter to their shared storage (which would often be a static expanded parameter, combining this feature and the previous), and this same value would be used when reconstituting the composite type.

Separating static storage from property wrappers

Finally, the third proposal is to take advantage of static parameters to reduce the inline memory overhead of certain kinds of property wrappers, which is most of what we've been talking about in this thread, and so I won't go over it again. This could be done either by passing the static argument pointer to a wrappedValue: subscript during accesses, as Joe suggests, or by reconstituting the property wrapper during accesses from a combination of the static argument pointer and the instance storage, as I suggest. In either case, I think this builds naturally on the previous two language features, which are independently useful.

30 Likes

This would definitely be independently useful! I’ve often wanted something like this for enum cases with a single associated value. It would be great to be able to write:

enum Shape {
  case ellipse
  case rectangle
  case roundedRectangle(expanded RoundedRectangle)
}

struct RoundedRectangle {
  var topLeft: Double = 0
  var topRight: Double = 0
  var bottomLeft: Double = 0
  var bottomRight: Double = 0
}

let tabShape: Shape = .roundedRectangle(topLeft: 4, topRight: 4)
16 Likes