[Pitch] Type Wrappers

Sorry, I've missed this part! Thanks!

As I mentioned in my previous comment this wouldn't allow fine-grained control over accesses to the properties.

I can imagine this being a compelling use case for swift-protobuf. We currently use some heuristics to decide whether the generated struct is small/simple enough that we generate its fields as stored properties or if we should generate an embedded class with CoW semantics. Property wrappers suffer the problem described in the pitch that we can't wrap all the fields as a set; they would each incur their own heap allocation. Instead, I believe this would let us just conditionally attach a type wrapper to the message struct when we generate it, and then the rest of our generation could be simplified by removing conditionals that say "if we have embedded storage, generate this, else generate that".

4 Likes

For what it's worth, if the enclosing type is a class you can solve it with property wrappers using enclosing instance:

class TrackingTest: GenerationTrackable {
    private var generation: Int = 0

    func increaseGeneration() {
        generation += 1
    }

    @GenerationTracked var something: String = ""
    @GenerationTracked var somethingElse: String = ""
}

protocol GenerationTrackable: AnyObject {
    func increaseGeneration()
}

@propertyWrapper
struct GenerationTracked<Value: Equatable> {
    static subscript<T: GenerationTrackable>(
        _enclosingInstance instance: T,
        wrapped wrappedKeyPath: ReferenceWritableKeyPath<T, Value>,
        storage storageKeyPath: ReferenceWritableKeyPath<T, Self>
    ) -> Value {
        get {
            instance[keyPath: storageKeyPath].storage
        }
        set {
            if newValue != instance[keyPath: storageKeyPath].storage {
                instance.increaseGeneration()
            }
            instance[keyPath: storageKeyPath].storage = newValue
        }
    }

    @available(*, unavailable,
                message: "@Published can only be applied to classes"
    )
    var wrappedValue: Value {
        get { fatalError() }
        set { fatalError() }
    }

    private var storage: Value

    init(wrappedValue: Value) {
        storage = wrappedValue
    }
}

I think this in an interesting point because it shows that this property wrapper feature could be replaced with a type wrapper which is easier to use and is not limited to classes.

2 Likes

Ah, thank you; needed a second read to grasp this (even though it's laid out clearly in the design!).

It felt intuitive to me that the generic type accessible to a type wrapper is the type that it wraps, like how a generic property wrapper is often generic over the type of the wrapped property.

I asked about constraints on the wrapped type because it feels reasonable that a type wrapper might utilize information from the type that's wrapped—here's a rough sketch of what I had in mind:

protocol HistoryTracked {
    /// Defines the maximum number of elements which can be held in memory for an @HistoryTracking type
    var recentHistoryMaxBufferSize: Int { get }
}

// ! Suppose 'Wrapped' here represents the actual wrapped type, rather than $Storage
@typeWrapper
struct HistoryTracking<Wrapped: HistoryTracked> {
  var underlying: Wrapped
  var history: [Wrapped] = []
  
  init(memberwise storage: Wrapped) {
    self.underlying = storage
  }

  subscript<V>(storageKeyPath path: WritableKeyPath<Wrapped, V>) -> V {
    get { 
      return underlying[keyPath: path]
    }
    set {
      // ! use of behavior customized by the wrapped instance
      if history.count == underlying.recentHistoryMaxBufferSize {
         history.removeFirst() // (excuse the performance)
      }
      history.append(underlying)
      underlying[keyPath: path] = newValue
    }
  }
}

The idea here being that a wrapped type can use these hooks to customize the behavior of the type wrapper:

@HistoryTracking
struct Document {
   var title: String
   var content: String

   var recentHistoryMaxBufferSize: Int { 64 } // Up to the implementor!
}

It seems to me that if the wrapper has access only to a synthesized $Storage struct, it can't impose constraints on the wrapped type, and therefore can't allow a wrapped instance to customize the behavior of the wrapper.

(This is one contrived example—just a gut feeling that other use cases may want this ability; a survey of those use cases might help!)

While writing this pitch I was wondering whether it would be useful to add a generic parameter to initializer or subscript which would represent the wrapped type.

For example:

@typeWrapper
struct HistoryTracking<Storage> {
  var underlying: Storage
  var history: [Storage] = []
  
  init(memberwise storage: Storage) {
    self.underlying = storage
  }

  subscript<Wrapped, V>(_ underlyingSelf: Wrapped, storageKeyPath path: WritableKeyPath<Wrapped, V>) -> V {
    get { 
      return underlying[keyPath: path]
    }
    set {
      history.append(underlying)
      underlying[keyPath: path] = newValue
    }
  }

  subscript<Wrapped: HistoryTracked, V>(_ underlyingSelf: Wrapped, storageKeyPath path: WritableKeyPath<Wrapped, V>) -> V {
    get { 
      return underlying[keyPath: path]
    }
    set {
      // ! use of behavior customized by the wrapped instance
      if history.count == underlyingSelf.recentHistoryMaxBufferSize {
         history.removeFirst() // (excuse the performance)
      }
      history.append(underlying)
      underlying[keyPath: path] = newValue
    }
  }
}
1 Like

I think $Storage/$storage would work then, I'll work on a custom diagnostic for that.

memberwise: is just something I used without any other subtext, will change it to storage:.

It would behave as Foo behaves because getDefaultValue() is placed as default expression on a parameter:

struct Baz {
  var i: Int

  init(i: Int = getDefaultValue()) { self.i = i }

  static func getDefaultValue() -> Int {
    print("Yo")
    return 42
  }
}

It's the same with or without wrappers, their init expressions and or wrapper arguments are added to parameters: init(@Wrapper(x: 42) i: Int) or init(@Wrapper i: Int = 42).

I'll expand on that in the proposal.

Good idea, will add that as well!

It would be allowed to write extension <Type> : ProtocolWithTypeWrapper but the type wrapper attribute is not going to be inferred for <Type> which means that the transform is not going to happen. Maybe we should add a warning for this case, I'm not sure...

Maybe this could be considered a future direction.

This is a great question and I don't have a good answer. It's definitely possible that a single TypeWrapper protocol with refinements would do the job just fine because I've only considered using this for adding methods from type wrapper to the wrapped type.

I don't have a use-case in mind, it seemed prudent to add explicit escape because the transformation is explicit, but it could definitely be misused.

I can add that to future directions.

In my opinion the provided examples are too exaggerated. This is very specific task which require specific solutions. You can simply do this such way:

class Document {
  let title: String
  var content: String? { [p: \.content] }
  var creationDate: Date { [p: \.creationDate] }
  var modifyDate: Date { [p: \.modifyDate] }
  var type: String { [p: \.type] }
  
  private var storage = Storage {}
  
  subscript<U>(p keyPath: WritableKeyPath<Storage, U>) -> U {
    get { storage[keyPath: keyPath] }
    set { storage[keyPath: keyPath] = newValue; storage.generation += 1 }
  }
  
  private struct Storage {
    var generation: Int = 0
    
    var content: String?
    var creationDate: Date = .now
    var modifyDate: Date = .now
    var type: String = ""
  }
}

In this example there are four mutable properties and not so much bolierplate. You can protect mutable state by any technique you need or not protect it at all.

The proposed solution does not protect people from doing wrong things. Shared DispatchQueue can be passed to property wrapper initializer for example. If someone create a lot of DispatchQueues it is not a language design problem, it is a programmer's mistake.

1 Like

This is something which is potentially interesting for Realm, although I think we'd need some additional pieces to actually use it.

Currently, users define model classes by inheriting from a base class we provide, and then marking properties with a property wrapper we provide:

class Document: Object {
    @Persisted(primaryKey: true) var _id: ObjectId
    @Persisted var title: String
    @Persisted var content: String?
}

Object instances of these classes can either be "unmanaged", in which case they behave exactly like ordinary objects and @Persisted is just pass-through storage, or they can be "managed", which means that the object is an accessor for data in the database, and the property getters read from the database and the setters write to the database. The normal initializer (i.e. Document()) creates unmanaged objects, reading objects from the Realm produces managed objects (e.g. realm.objects(Document.self).first!), and unmanaged objects can be turned into managed objects with realm.add(document).

The current property wrapper based design has a few drawbacks:

  1. It requires subclassing a base class. This means that it can't be a struct (which may or may not be a good idea even if it was possible), and it means that we don't play well with other libraries which want you to subclass something.
  2. There's some minor performance issues with initializing managed objects due to that we need to loop over each of the properties to initialize the property wrapper. This isn't a big deal, but given a model class with a large number of properties, something like realm.objects(Document.self).map { $0.title } can spend most of its runtime in initializing property wrappers.
  3. The implementation is horribly complicated, and I'm not fully confident that everything we're doing is actually legal and not just things that happen to work. Also it relies on _enclosingInstance.
  4. The complexity of the implementation has forced us to make the API worse in places. For example, we would have preferred @PrimaryKey var _id: ObjectId, but I couldn't figure out how to make it work without duplicating hundreds of lines of code.
  5. The managed or unmanaged enum discriminant is stored per-property, which in practice adds a word to the size of objects for each property (for most property types).

A hypothetical type wrapper based version might look something like:

@RealmObject
class Document {
    @PrimaryKey var _id: ObjectId
    var title: String = ""
    var content: String?
}

@typeWrapper
struct RealmObject<S> {
    enum Storage<S> {
        case unmanaged(S)
        case managed(RLMObject)
    }
    var storage: Storage<s>

    init(memberwise storage: S) {
        self.storage = .unmanaged(storage)
    }

    init(managed object: RLMObject) {
        self.storage = .managed(object)
    }

    subscript<V>(storageKeyPath path: WritableKeyPath<S, V>) -> V
          where V: RealmPersistable {
        get {
            switch storage {
            case let .unmanaged(v):
                return v[keyPath: path]
            case let .managed(obj):
                return V.get(obj, path)
            }
        }
        set {
            switch storage {
            case let .unmanaged(v):
                v[keyPath: path] = newValue
                self = .unmanaged(v)
            case let .managed(obj):
                return V.set(obj, path, newValue)
            }
        }
    }

    func promoteToManaged(object: RLMObject) {
        self.storage = .managed(object)
    }
}

// Each type which can be stored in Realm conforms to RealmPersistable and
// provides functions for reading and writing values of that type
extension String: RealmPersistable {
    static func get<S>(_ object: RLMObject, _ path: WritableKeyPath<S, String>) -> String {
        return RLMGetString(object, keyPathToColumnId(path))
    }

    static func set<S>(_ object: RLMObject, _ path: WritableKeyPath<S, String>, _ value: String) {
        RLMSetString(object, keyPathToColumnId(path), value)
    }
}

func keyPathToColumnId<V: RealmObject, S>(_ path: WritableKeyPath<S, String>) -> Int {
    // ?
}

// @PrimaryKey is just a no-op tag used during schema discovery and to
// constrain the property to valid types
@propertyWrapper
struct PrimaryKey<Value: RealmPrimaryKey> {
    var value: Value
    init(wrappedValue: Value) {
        self.value = wrappedValue
    }

    var projectedValue: Self { return self }

    var wrappedValue: Value {
        get { value }
        set { value = newValue }
    }
}

I see two missing pieces for this to work. The first is the where clause on the subscript. We don't support properties of arbitrary types and instead need them to conform to a specific protocol, which the proposal currently doesn't appear to allow.

The other is something not directly related to this proposal: I don't think keyPathToColumnId is currently implementable. _enclosingInstance has a similar problem where we need to turn a keypath into something that can identify which database column to access. There we take advantage of that the target of the keypath is a property wrapper we define, and store the column id on each of the property wrappers (initializing these property wrappers involves ivar_getOffset() and pointer math, and is the part that I'm not sure is actually valid). With type wrappers we could not use the same approach.

There's probably some other problems which would pop up; this is just what occurred to me with an hour or two of thinking about it.

3 Likes

Trying to think way outside the box.


I still think if the focus of this proposal is to operate over an extracted set of stored properties, this new attribute should also reflect only that specific sub-case with its own name and not burn the generic typeWrapper term. That said, storageWrapper would fit way better here.


Let's get a bit abstract in here. I'll reuse the above example I used where are @typeWrapper (not the @storageWrapper, but the true generic version as mentioned above) could project a similar named $-sign prefixed type.

Also let's put the concrete implementation details aside. I'm sure we would eventually be able to figure out the HOW if we only wanted, but that's not super important just for the sake of this little demonstration. The whole purpose of this is just to visually present one or two examples on how interesting true type wrappers could become.

@ValueSemanticsObject // ──┐                      
class Foo<T>          //   └───────── wrapped type ─────────┐
struct $Foo<T>        // ◀─ projected value semantics type ─┘   

// just for the sake of demonstration, let's assume
// a type wrapper would use a template type which the 
// compiler could use to construct a brand new `$` type
// or simply built a type alias with it

@typeWrapper
struct ValueSemanticsObject<WrappedObject: AnyObject> {
  struct Template
    var _wrappedObject: WrappedObject

    subscript<T>(
      dynamicMember keyPath: ReferenceWritableKeyPath<Value, T>
    ) -> T { /* perform CoW here  */ }

    // add something to like `dynamicMethod` to project methods
    // and not only properties
  }
}

// the compiler could potentially synthesize something like this?
typealias $Foo<T> = ValueSemanticsObject<Foo<T>>.Template

// or if we ever get `newtype`
newtype $Foo<T> = ValueSemanticsObject<Foo<T>>.Template

Such a Template shouldn't be limited to a struct only, it could be anything. Of course we would have to figure out how to define a fairly good set of understandable and predictable rules to pull this off, but truly wrapping a whole type could do so much more besides pulling the stored properties into a new private storage type.


For the next small discussion, let's disable the automatic $Storage generation for the @storageWrapper and see what this would mean.

At first glance we would lose almost all benefits that this pitch provides in regards to automatic synthetization. Before I attempt to try elaboration on how we could fix that, let's see what we actually gained from this.

The strictness of the pitched rules is removed. A @storageWrapper type does not need to have a generic type parameter to directly reference the storage type as it can now be done manually.

@storageWrapper
struct MyClassStorageWrapper {
  var storage: MyClass.MyManualStorage
  // details excluded for simplicity
}

@MyClassStorageWrapper
class MyClass {
  struct MyManualStorage { ... }
}

As several people already mentioned above the pitched $Storage type is fairly limited. With this change however we could restore the manual control over the concrete storage type and we can technically do whatever we want that type.

I previously mentioned that the automatic connection between the stored properties and the storage is lost. How can we fix this without reintroducing all the synthesized boiler plate code?

Let me answer that question with a counter question: What if we would first introduce a property wrapper for exactly that purpose?

class MyClass {
  var wrapper: MyClassStorageWrapper

  @Link(\.wrapper.storage.value)
  var value: String

  @Link(\.value.count)
  var count: Int

  // in the future?
  @Link(\.value.contains(_:)) 
  var contains: (String.Element) -> Bool
}

If such a property wrapper existed, it would allow us to re-link the lost connection. Sure it requires a bit of manual work, but even that seems like a win, as such a wrapper can not only link against the one storage type as was previously pitched, but it can link against anything that is reachable via a key-path, maybe even methods in the future.

At this point I think it becomes a bit more clear that the @storageWrapper that does all that automatically for us is not really brining that much to the table. Don't get me wrong, I'm not trying to say it's not useful, but I think that an alternative solution to that particular problem seems a bit more promising as it also opens up other avenues.

However this would require us to explore further the ability for property wrappers to access the enclosing type, optimization of property wrappers where they could be just 'computed property wrappers' to be also allowed in extensions, etc. etc.


I hope that feedback will not derail the topic and still somehow help the author(s) and the community steering the evolution of these features towards the right direction.

2 Likes

Subscripts references are injected during type-checking so subscript(storageKeyPath:) could be overloaded the way you describe.

On the one hand, I feel like this is something that could definitely be useful. A use case that immediately comes to mind is wrapping a type with automatic undo management:

@typeWrapper
struct UndoManaged<T> {
  private var storage: T
  private var undos: [T] = []
  private var redos: [T] = []
 
  init(storage: T) {
    self.storage = storage
  } 
 
  subscript<U>(storageKeyPath keyPath: WritableKeyPath<T, U>) -> U {
    get { storage[keyPath: keyPath] }
    set {
      undos.append(storage)
      redos.removeAll()
      storage[keyPath: keyPath] = newValue
    }
  }
  
  mutating func undo() {
    if let undo = undos.popLast() {
      redos.append(storage)
      storage = undo
    }
  }

  mutating func redo() {
    if let redo = redos.popLast() {
      undos.append(storage)
      storage = redo
    }
  }
}

On the other hand, it feels like a LOT of added compiler/language complexity for little saved user complexity. I don’t think the proposal does a good job of comparing this to dynamic member lookup. The same wrapper implemented today with @dynamicMemberLookup looks almost identical:

@dynamicMemberLookup
struct UndoManaged<T> {
  private var storage: T
  private var undos: [T] = []
  private var redos: [T] = []
 
  init(_ storage: T) {
    self.storage = storage
  } 
 
  subscript<U>(dynamicMember keyPath: WritableKeyPath<T, U>) -> U {
    get { storage[keyPath: keyPath] }
    set {
      undos.append(storage)
      redos.removeAll()
      storage[keyPath: keyPath] = newValue
    }
  }
  
  mutating func undo() {
    if let undo = undos.popLast() {
      redos.append(storage)
      storage = undo
    }
  }

  mutating func redo() {
    if let redo = redos.popLast() {
      undos.append(storage)
      storage = redo
    }
  }
}

It seems to me that the real difference between type wrappers and dynamic member lookup isn’t really complexity, but the UX when using the wrapper. With dynamic member lookup you need to wrap a type when you use it. With type wrappers you wrap the type when you define it, and the wrapper effectively becomes part of the type.

With dynamic member lookup:

struct Document {
  ...
}

var document: UndoManaged<Document>

With type wrappers:

@UndoManaged
struct Document {
  ...
}

var document: Document
5 Likes

The thing that worries me is that some of the examples are starting to look like protocols where it defines a suite of behavior/capabilities applied to arbitrary types

@dynamicMemberLookup is an attribute to have the compiler support syntax sugar for accessing properties through a well-defined subscript, leaving much of the actual implementation up to the type to define, but this new wrapper is actually "hey, go generate a ton of properties and storage, but don't expose all these details unless someone knows to go looking for them or recognizes the synthesized nature of the generated code".

In the proposal there was mentioned that "this is boilerplate that would have to be duplicated for every object" and traditionally that type of space is solved with metaprogramming through either a template or macro system.

The fact that it's not composable, and in several cases hiding details about how the type works (side-effects), I'm very concerned we'd be introducing footguns that are going to have to be avoided primarily through documentation or strong API design by the implementers.

10 Likes

I think this is a great observation, and I wonder whether the addition of variadic generics will allow us to generalize property wrappers to abstract over a variable number of stored properties.

Type wrappers allow libraries to implement property storage and access patterns for all stored properties in a type, which can be opted into by applying a custom attribute to a type. The functionality is very similar to property wrappers, but there is one single wrapper instance that implements a set of stored properties in a type. Despite being so similar conceptually to property wrappers, type wrappers introduce a lot of new complexity into the language, including:

  • A new attribute @typeWrapper
  • A new attribute @typeWrapperIgnored
  • A new initializer form init(storage:)
  • A new indirect property accessor in the form of a subscript: subscript(propertyKeyPath:storageKeyPath:)
  • A synthesized $Storage struct specific to each type that is annotated with a type wrapper that stores each of the stored properties written in the wrapped type
  • A new transformation using init(storage:), the subscript, and the $Storage type that turns all stored properties into computed properties.
  • DI support for hand written member-wise initialization code that re-writes property initialization to initialization of the wrapper through init(storage:)
  • ...and so on

With variadic generics, there is no longer a need for the synthesized $Storage struct, and type wrappers can simply be parameterized directly on a pack of stored property types, much like how a property wrapper is parameterized directly on its wrapped value type. If we give property wrappers the capability to abstract over a variable number of stored properties, the majority of the bespoke language capabilities for type wrappers go away. We also get property-wise projections without any new concepts to the language if the property wrapper declares a var projectedValue, and the ability to store the values however you want, effectively allowing you to write your own $Storage type.

The property wrapper transform only needs to be slightly tweaked to project out a single element from the pack inside of the computed property accessor. Even without a builtin pack element projection feature, we can already express pack element projection on a variadic generic property wrapper using dynamic member lookup.

Among others, property wrappers that implement different storage patterns, such as CopyOnWrite should be easily generalizable to multiple wrapped values. For example, you might imagine that a variadic generic CopyOnWrite property wrapper looks like this:

class Box<Stored...> {
  var stored: Stored...  
}

@dynamicMemberLookup
@propertyWrapper
struct CopyOnWrite<Value...> {
  var box: Box<Value...>
  
  init(wrappedValue: Value...) {
    box = Box(stored: wrappedValue...)
  }
  
  var wrappedValue: Value... {
    get { box.stored... }
    
    set {
      if (!isKnownUniquelyReferenced(&ref)) {
        box = Box(newValue)
      } else {
        box.stored = newValue
      }
    }
  }
  
  // We might want to support pack element projection through a built-in pack
  // projection feature, but we can also support it through dynamic member lookup
  // using positional tuple key-paths.
  subscript<U>(dynamicMember keyPath: WritableKeyPath<(Value...), U>) -> U {
    get { (wrappedValue...)[keyPath: keyPath] }
    set { (wrappedValue...)[keyPath: keyPath] = newValue }
  }
}

@CopyOnWrite
struct Person {
  var name: String
  var birthdate: Date
}

// or

func test() {
  @CopyOnWrite var buffer: MyBufferType = ...
}

You can imagine that this code gets expanded to

struct Person {
  init(name: String, birthdate: Date) {
    _storage = CopyOnWrite(wrappedValue: name, birthdate)  
  }

  var _storage: CopyOnWrite<String, Date>

  var name: String {
    // using dynamic member lookup, but this could instead use a builtin pack element
    // projection on _storage.wrappedValue
    get { _storage.0 } 
    set { _storage.0 = newValue }
  }

  var birthdate: Date {
    get { _storage.1 }
    set { _storage.1 = newValue }
  }
}

The transformation can easily be tweaked for the experimental enclosing-self subscript by passing in the pack element key-path in addition to the wrappedKeyPath and storageKeyPath.

To my mind, generalizing property wrappers achieves the goals of this pitch while building on top of the concepts the language already has for implementing property access patterns.

10 Likes

I think this makes sense, variadic generics really help a lot in this case!

Just to complete the loop; one fairly useful case for type wrappers is to allow for protocol wrapping in extensions. This allows for the potential of side stepping any composability issues. @hborla came up with quite honestly a pretty darned brilliant strategy to account for this where we could write a type wrapper as:

@SomeTypeWrapper
extension SomeProtocol where SomeAssociatedType: SomeConstraint { }

This allows the type wrapper to provide default implementations and would be very useful not just for the observability pitch but I am sure may other applications.

5 Likes

I've been working on the implementation for attached macros, and I had the chance to implement @Observable from the Future Directions section of the observation pitch. I was able to replicate the functionality in this type wrapper pitch through composition of the various attached macro capabilities.

The macro approach provides more flexibility, because the macro author can decide what code to generate inside the wrapped type. The macro-expanded code then becomes much more transparent to the programmer than a subscript call, which isn't very informative in terms of what that call accomplishes. The macro still has the ability to add backing storage variables, initializers, nested types derived from stored properties of the wrapped type, and more, and the transformation can be customized depending on what kind of type the macro is attached to (e.g. a struct versus an actor).

After taking the motivating use cases surfaced in this pitch thread and implementing them as macros, I'm confident that macros can fully subsume type wrappers while providing more flexibility to library authors.

49 Likes

This is very exciting news. Thank you @hborla for the update.

3 Likes

Yes, thank you Holly! It's promising for macros to see them subsume the need for other complex language features. :slightly_smiling_face:

4 Likes