[Pitch] Observation (Revised)

The pitch has both: values(for:) emits an AsyncSequence of values for a given key path, changes(for:) emits an AsyncSequence of changes for a given set of tracked properties.

Just to clarify, I think the intention for this is that it is a new module that is part of the shipping toolchain who will require an explicit import Observation, similar to RegexBuilder and Distributed. This is not going to be a "package" that one can declare as a dependency (using .package) in SwiftPM.

6 Likes

What happens if you don’t do this? Will the listener to \.someComputedProperty just never receive any updates?

Would it be possible to call the getter when creating the observation and detect which properties are accessed, then dynamically use that to subscribe to updates? (You would have to re-check this each time the property is changed in order to ensure that nothing is missed behind e.g. if conditions that change).


Should the dependencies(of:) method be documented as intended to be a pure function (such that calling it for a given key path will always return the same value)?


How does the macro handle let-bound properties? Do they stay on the top level object or are they also moved into the Storage struct?


I worry that this could lead to people unintentionally running tasks on the main actor. Could there be a better way of maintaining these guarantees without this potential performance issue? (Maybe each type could define its own actor, and actors could use their own internal actor?

That makes sense! This is the bit that was confusing me:

1 Like

Regarding actor isolation: I think it's great that there is a way to isolate observed changes to arrive on a specific actor, this seems it will be especially important in a UI context. I do have a couple of questions regarding this:

  • With other observation techniques it was possible to guarantee that an observation event fired on the main actor (previously the main dispatch queue) would be called in the same event loop cycle as the event that triggered it (e.g. via KVO/KVC or Combine). This is/was essential for avoiding animation glitches. Is this same guarantee provided here?
  • Will there be a way to cascade isolation to the actor originally isolated to if follow-on operators like map/filter are used? It's my understanding that Swift's concurrency system will – unless the method is annotated – always execute non-isolated/annotated async methods on the default executor. So, if you were to follow up your call to object.values(for: \.someProperty) with .map { SomeStruct($0) }, as things stand, that map call may unexpectedly shift your concurrency context to the default executor. That would probably be unexpected/undesirable in a UI context causing the previously mentioned animation glitches due to context switching. I'm especially interested in how usage marries with the advice that has been given to date, which is to avoid frequent context switching when using Swift concurrency, which may manifest as an issue if observing something like the user's mouse position for example.
3 Likes

Thanks for the new pitch!

I have a question about the async sequences. Is the first value the current one (when the iteration starts), or only on the first change that happens after iteration has started?

5 Likes

I'll parse the pitch later, but here's a smallish nit pick. Could the synthesized storage type be called and stored in a different place?

Instead of _storage: Storage could we make it less likely to collide with existing code and make it _observableStorage: ObservableStorage?

9 Likes

Looks really interesting pitch and a great example of macros. I just want to check that this doesn't effect the didSet/willSet of observed properties?

3 Likes

Very nice iteration of the pitch, I really like it!

A comment about the pitch text itself: I find it a bit weird that there is no mention of willSet/didSet since that's really the only "language" feature related to observability. The two mention are part of Objective-C and a framework, not really part of the Swift language. Although I totally get their mention since a lot of learnings come from there it would be good to mention the language native way of observing changes and how this pitch relates to that.

A comment about the SwiftUI example: the userCredential property used in the hasAccount computation doesn't seem to be defined.


I find a bit weird that a goal is:

But then you need to do this for something as common as computed properties:

I'm not fond of this distinction, I would expect any property to be observable no matter if it's computed or stored. And what would happen if you want to observe the computed property but is not declared as a dependency? I assume the keypath will be accessible so it will just silently ignore updates?

I think this becomes very important in the hypothetical integration with SwiftUI. With the code in the example I would expect the body of my view to be updated if I used the hasAccount computed property. If that's not the case it can be a very subtile source of issues.


Besides this I really like this pitch, the integration with async sequences and actor isolation is very nice to see.

I also loved the withTracking explanation, it's like seeing a glimpse of how SwiftUI works under the hood. I could imagine SwiftUI wrapping the calls to body with the apply closure. Eye opening really.

And I'm very glad we're using a language tool (macros) that seems to be covering a lot of ground in multiple pitches :D

Kudos to everybody involved!

11 Likes

What does this mean? Is this talking about the case where e.g. several properties change at once but you only really want a single notification?

1 Like

Good so far!

I'm curious why prior art does not mention Swift's getters and setters or property wrappers, which are both good ways to set up forms of observation that do not require a class to be used?

Or does Observable work with structs?

1 Like

Still haven't read the proposal in detail, but the _storage property kept me thinking. This isn't about the Observation but rather about a limitation we face that is reserved to the Swift compiler. I have not followed the macro proposals in details either so excuse me if I'm going to unintentionally rehash something. Could the new macros be used to rebrand @propertyWrapper attribute as a macro? If that could theoretically work out, then can we open up the creation of $ prefixed type members via macros? That would allow us to spawn a more unique name for _storage (e.g. $observableStorage) and possibly allow its exposure when needed.

cc @Douglas_Gregor I'd like to hear your opinion on the idea of allowing $ type members via macros.

1 Like

More thoughts:

Could we have a semi-opt-out functionality? I can imagine that we could permit the user to manually provide the Storage type. Then the macro will ignore all stored properties on the observable type itself and only synthesis the stored properties into the parent type. That gives us some fine grain control when needed and it also provides us the ability to expose the storage itself when needed via its access modifier.

Some examples:

@Observable
struct S {
  public struct Storage {
    public var a: Int
    internal var b: Int
    internal var c: Int
    private var d: Int
  }

  private var b: Int
  public var c: Int

  public var e: Int
}

// desugars to
struct S: Observable {
  public struct Storage {
    public var a: Int
    internal var b: Int
    internal var c: Int
    private var d: Int
  }

  // storage property exposed as `Storage` is public.
  public _storage: Storage // `public $observableStorage` would be nicer

  // synthesized as public
  public var a: Int {
    get { 
      _registrar.access(self, keyPath: \.a)
      return _storage.order
    }
    set {
      _registrar.withMutation(self, keyPath: \.a) {
        _storage.order = newValue
      }
    }
  }

  // access level through that property reduced from internal to private
  private var b: Int {
    get { 
      _registrar.access(self, keyPath: \.b)
      return _storage.order
    }
    set {
      _registrar.withMutation(self, keyPath: \.b) {
        _storage.order = newValue
      }
    }
  }

  // access level through that property is increased from internal to public
  public var c: Int {
    get { 
      _registrar.access(self, keyPath: \.c)
      return _storage.order
    }
    set {
      _registrar.withMutation(self, keyPath: \.c) {
        _storage.order = newValue
      }
    }
  }

  // synthesized as private
  private var d: Int {
    get { 
      _registrar.access(self, keyPath: \.d)
      return _storage.order
    }
    set {
      _registrar.withMutation(self, keyPath: \.d) {
        _storage.order = newValue
      }
    }
  }

  public var e: Int // has no synthesis, as it's explicitly not part of `Storage`
}

The stored property e cannot be observed. b and c property declarations are still consumed by the macro, as they are part of the explicit Storage type. a and d are projected from Storage into the S type and their access modifier is mirrored with the access modifier inside Storage.

If we could make something like this work, it would provide some great control over this functionality, but only when needed and not as a default requirement. It also shows that the storage type could be potentially exposed, if we want to allow that, and that it should probably have a better name which isn't prefixed by an underscore. If exposing the storage property is a no-go then ignore that part of my suggestion, but we might still want to provide an explicit Storage type declaration for the previously mentioned control.

1 Like

A great pitch! I'd definitely love this addition into the language.

My concern is about its use SwiftUI. If I understand correctly, you expect the View body to get reevaluated – under what conditions? Whenever anything in the model changes, or only when model.searchString changes?

With the former, the performance would get only worse than now.

With the latter, I have an issue here that this may be unexpected – there could be some variables that you do not want to get View to be redrawn with.

Also, the general paradigm in SwiftUI behavior is to use a PropertyWrapper for parts of code that can cause View reevaluation. For that, the SwiftUI code is quite easy to reason about but the proposed does break that – with us never being sure whether var could cause our view to reevaluate on its own.

I was fondly reminded of this document introducing key-value observing while reading this.

I could imagine these switch statements grow to an unwieldy wall of key paths. The document above speaks to the keyPathsForValuesAffecting<Key> convention. Any chance the observation machinery could look for the more precisely named method before hitting dependencies(of:)? e.g.: going with the example class, could it check for static var dependenciesOfSomeComputedProperty: TrackedProperties<Self>).

Really great pitch! Keep up the exciting work.

3 Likes

This is a great revision and I’m really liking the direction this is going. Thanks.


While observation not working for multi-layered keypaths (\.account.name) is unfortunate, it is an easy boundary to explain. Sounds like something that would be good to explore in the future. :+1:


I have some concern over the lack of automatic support for computed properties. It feels like something that should just work, or have similarly easy support like the main macro synthesis provides.

Could we generate the contents of the dependencies(of:) method using macros? Either from inspecting the computed property’s body, or via some annotation macro just to move the contents of the static method’s switch statement closer to the source. I guess the issue with all of that would be computed properties in extensions!

At the moment I’m not able to come up with a better solution here, but it would be ideal if we could reduce the friction of having to maintain a separate list of key paths in a different area of the source code from where the canonical information lies.

2 Likes

Thanks for pitching this again! I think it looks amazing already but I have a few questions

Lifetime implications of observable objects

Both the ObservedChanges and ObservedValues types require the underlying object to be alive to produce new values. Are they retaining the subject or expect it be retained somewhere else. I think we should call this out in the documentation.

Isolation

I understand the reason why we want to add this since it allows callers of the ObservedChanges sequence to coalesce transactions. However, I am not entirely convinced this composes nicely in the broader ecosystem. As @tcldr already pointed out as soon as you add any other algorithm to the async sequence you are incurring hops again.

We have recently explored the custom actor executor feature in NIO and it already brings us substantially close to what we want. However, there is one missing piece which is having a way to tell a task what the global executor is for its scope. The problem that we are seeing in NIO and also here is that the AsyncSequence.AsyncIterator.next() method forces a hop right now on to the global executor. If we would be able to override the global executor on a task basis we wouldn't hop anymore. As far as I can see this would eliminate the need for the isolation parameter on ObservedChanges.

This is quite important to me because I think we are setting a bad precedent here in saying that root AsyncSequences or in general AsyncSequences should allow for an isolation opt-in. IMO we should rather get task executors so we can make this work generically everywhere and give the user a way to control where their code is run.


await withTaskGroup(Void.self) { group
    group.addTask(executor: MainActor.shared.unownedExectuor) {
        let someObservable = SomeObservable()
        let changes = someObservable
            .changes(for: [\.someProperty, \.someOtherProperty])
            .map {  }

        for await element in changes {
            // This runs on the MainActor
        }
    }
}

Castiness of the AsyncSequences

Could you specify and document what the behaviour of the ObservedChanges and ObservedValues sequences are. Can multiple iterators be created? If so what values do iterators get that are created delayed?

Location of API

This API will be housed in a package; outside of the standard library.

This was already brought up-thread from my understanding this library will be living in the toolchain after all. I would love if we could get a statement from the core team when a new API should live in the toolchain vs in a separate package. Furthermore, if we add new Swift modules to the toolchain how they integrate with the evolution process.
Lastly, if we land this in the toolchain directly we are bound to have API/ABI stability going forward. I would love if we could first play around with this within a package and get some larger feedback before integrating it into the toolchain.

Overall, I am very happy with how this shaped up and I a lot of usefulness in these APIs to fill a missing gap in the Concurrency story.

12 Likes

Hi thanks for the pitch.

I'm also interested in having more details about that. Will the AsyncSequences be simulcast / broadcast ?

1 Like

I may be misremembering, but what I remember (and what my opinion is) is that because these are public imports that come by default with Swift, they must go through evolution.

1 Like

The thing that I'm not sure about is composition. Let's say I have 2 observable objects, representing parts of my model, and I want to combine them in a new observable object which exposes data from each of them (possibly transforming their values along the way).

It seems that I would be expected to manually write out those dependencies:

This essentially means that I need to write my code twice, which then becomes a very likely source of bugs - every time I update a computed property, I need to remember to update its entry in this Single Giant List else updates sometimes fail to trigger. I don't like this in KVO, and rather than copying it, I think we should consider it a major defect that observation in Swift should improve on.

This design also seems to be incompatible with extensions - if I add a computed property in an extension, I will also need to go back and update the Single Giant List. That hurts the organisation of my code and makes mistakes even more likely, and in some cases it may not even be possible - for instance, extensions across modules, or computed properties in a different file and marked as (file)private.

Also, is dependencies(of:) called recursively? Can keypath A declare a dependency on keypath B, which itself depends on keypath C? Or does keypath A need to list every leaf node of its dependency tree? Are dependencies also limited to one keypath component, or can they refer to keypaths in other objects (e.g. \.accountName depends on \.account.name)?

Also, is this limited to reference types, or types with reference semantics? It can be useful to wrap multiple reference types in a struct for composition purposes, and the resulting struct actually has reference semantics. Non-copyable structs also have reference semantics and should conceptually be able to support observation (although this particular design looks like it may require copyability; I'm not sure - it's possible that we can't express the lifetime of noncopyable structs well enough to support this design).

11 Likes