[Accepted with revision] SE-0395: Observability

[Ben_Cohen]

Hello, Swift community.

The second review of SE-0395: Observability has complete and the proposal has been accepted with revision.

The second review was of a narrower proposal, that focused on the Observable protocol and the withTracking(_:changes:), with some previously proposed functionality moved to Future Directions.

One of the questions of the review was whether the "marker protocol" approach was correct. Since marker protocols do not have ABI, changing this to a regular protocol would have been impossible later if it was found that the protocol needed to add requirements. Consequently, the language steering group requested that the authors investigate making it a regular protocol. The authors were able to adapt the design to a regular protocol.

Another area discussed was the meaning of observing a stuct type. Reviewers expressed concern that the semantics of what observing a value means were unclear, and while the steering group does agree that it is possible to define them in a way that might work for some cases, this should be deferred out of this proposal and given further consideration, with room left in the ABI to introduce it later. As such, the proposal is being amended to restrict observation to classes.

Some reviewers commented on the fact that property-wrapped properties cannot be observed. The macro transform cannot currently add or remove attributes, and reviewers commented that if that feature were added for macros, or if this were a language feature implemented in the compiler, this limitation could be lifted. However, the steering group notes that adding or removing property wrappers as part of the transform would require that that transform have a full understanding of what exactly each property wrapper did (since property wrappers are defined by users). This can be contrasted with didSet on a property, which macros can change – but the meaning of which is fixed as part of the language. The language steering group felt that this limitation does not have a readily available alternative, and that this should not hold back the acceptance of the proposal.

SE-0395 has therefore been accepted with the above modifications for Swift 5.9. The proposal text will be updated to include them.

Thanks to both the community and the proposal authors for their time on this addition to Swift.

Ben Cohen
Review Manager

[ksluder (Kyle Sluder)]

@Ben_Cohen

However, the steering group notes that adding or removing property wrappers as part of the transform would require that that transform have a full understanding of what exactly each property wrapper did (since property wrappers are defined by users).

At the risk of invoking a hypothetical, does this mean that the language workgroup believes it is a non-goal to enable a developer to vend both a property wrapper (which is back-deployable) and a macro that sugars that property wrapper?

[Ben_Cohen]

That’s a question about whether we should ever add the ability to change attributes to macros, which is a different proposal. Some macro could have special knowledge of a specific property wrapper, like in your example. But this proposal would require the observable macro to understand all property wrappers.

[tcldr]

For the record, this review – or any of the previous proposal threads – are still yet to offer a rationale as to why a purported generalised mechanism for observation generates events exclusively on a willSet basis.

Whilst this fulfils the very specific requirements of Swift UI, for a generalised observation mechanism this seems highly unusual and is a major limitation.

The workaround of re-creating a didSet notification through an async Task will quickly be revealed as an anti-pattern for the reasons expressed below.

For context, here's some further detail on the issue as discussed in the latest proposal thread:

---

The willSet variant is great for allowing SwiftUI to perform its 'before changes' render pass and schedule its 'after changes' render pass for the end of the current event loop cycle, but for anything else that needs to happen between these two points – we're stuck.

Really, the onChange parameter of the withObservationTracking(_:onChange:) function should be called willChange to better reflect the semantics of the API.

A participant in the old pitch thread laid out their [questions surrounding] the current API:

@Daniel_Steinberg

how are we to observe an @Observable from something that is not a SwiftUI view - something that may itself be Observable.

My use case may not be common, but when using Combine, I often broke the pipelines into smaller pieces and various structs would subscribe to upstream publishers and modify or join them before republishing them. In this way we could create small pieces that did one thing that could be appropriately named and flexibly assembled. I was thinking we could do the same with @Observables upstream that we might assemble with things from async algorithms etc.

Then again, I may be trying to hold it wrong.

As I summarised in the pitch thread, the two recommendations we have right now are to a) use computed properties, or b) use this pattern:

@Philippe_Hausler

However if you do have a need to (in non SwiftUI scenarios) to do something similar then the use for the withObservationTracking (note; it got changed to a free floating function).

func resetObservationOnExternalObservable() {
    withObservationTracking {
      self.dependentProperty = "\(external.indirectlyObservedProperty). Woohoo!"
    } onChange: { [weak self] in
      // The onChange is fired on the willSet side of things - so the reset needs to be called asynchronously to prevent a recursion and also update after the property changes.
      Task { @MainActor in
        self?.resetObservationOnExternalObservable()
      }
    }
  }

However, method A suffers from unnecessary view refreshes, while method B suffers from an async hop i.e the observation event happens after the 'after changes' SwiftUI snapshot. It misses SwiftUI's deadline for changes in the current transaction. It's also for this reason that an asynchronous sequence based API wouldn't help us here either.

To make this work how we want it, we really need Observable to generate trailing edge (didSet) synchronous notifications.

This would allow dependent observers to make any changes they need to make within SwiftUI's deadline for the current event loop.

It doesn't need to be anything huge for this version of the proposal, something as simple as changing withObservationTracking(_:onChange:) to withObservationTracking(_:willChange:didChange:) could lay the foundation.

[dmt (Dima Galimzianov)]

@tcldr

This would allow dependent observers to make any changes they need to make within SwiftUI's deadline for the current event loop.

It doesn't need to be anything huge for this version of the proposal, something as simple as changing withObservationTracking(_:onChange:) to withObservationTracking(_:willChange:didChange:) could lay the foundation.

This is what I tried to raise in the review as well, but unfortunately it wasn't discussed.
The thing is observing didSets is not enough. willSet has the invariant "subscribers will receive a notification before anything is changed", but didSet doesn't have the invariant "subscribers will receive a notification after everything is changed". To support this invariant we need transactions.

[tourultimate]

@tcldr

For the record, this review – or any of the previous proposal threads – are still yet to offer a rationale as to why a purported generalised mechanism for observation generates events exclusively on a willSet basis.

I was looking for this rationale as well and I am a bit disappointed it wasn't addressed.

[timothycosta]

Very disappointed also. It seems very clunky to use this for general observation the way you can with @Published. You can write your own, but it's a little weird that you have to. EDIT: Please don't use this. See comments below.

Also, it's quite frustrating that after people bringing up the point about didSet multiple times, the implementation does appear to support it. But it's limited to SwiftUI only.

[tcldr]

@timothycosta

Also, it's quite frustrating that after people bringing up the point about didSet multiple times, the implementation does appear to support it. But it's limited to SwiftUI only .

Thanks for pointing this out – I hadn't seen this before. It seems to have been added on July 12. Well after the issues were raised in the various reviews which makes the lack of acknowledgement even more curious. (And no doubt after the SwiftUI team came up against the same limitations we're all facing with the current API.)

[dmt (Dima Galimzianov)]

What would you do with withObservationTracking(_:didChange:) ?
Consider the following model:

@Observable final class Model {
  var a: Int = 0
  var b: String = ""
}

It's observed by didChange

withObservationTracking {
  _ = model.a
  _ = model.b
} didChange {
  print(model.a, model.b)
}

Somewhere the model is changed:

model.a += 1
model.b += "foo"

What do you expect to see in the output?

[mbrandonw (Brandon Williams)]

@timothycosta

Very disappointed also. It seems very clunky to use this for general observation the way you can with @Published. You can write your own , but it's a little weird that you have to.

Hi @timothycosta, I just mentioned this in another thread, but it should be known that the tool for converting withObservationTracking to a publisher has race conditions and cannot be relied upon:

Tracking properties in `@Observable` models internally

Hi @prathameshk, I highly recommend you do not use that publisher tool. The use of Task { … } in withObservationTracking in order to get the changed value has potential race conditions. This test shows the problem: @Observable class Model { var count = 0 } final class ObservablePublisherTests: XCTestCase { func testBasics() async throws { let model = Model() let countPublisher = model.publisher(keyPath: \Model.count) var values: [Int] = [] let cancellable = countPublisher.…

[timothycosta]

Thanks Brandon. Somebody on a public slack shared that with me and I wasn't planning on actually using it.

The problem just reinforces my point. Since Apple didn't provide any drop in replacement to use this like @Published, everyone is going to build their own naive versions that have issues.

[Panajev (Goffredo Marocchi)]

Has there be any further comments anywhere on these forums on why the didSet use case is missing? From this thread it feels like it was raised multiple times and not even acknowledged. Am I missing something perhaps?

[dmt (Dima Galimzianov)]

@Ben_Cohen @Philippe_Hausler @nnnnnnnn Could you please share if an extension to the Observable API to support end-of-changes event (i.e. didSet) is currently in development? Should we expect a proposal soon?
No pressure. I will be quite satisfied with the answer that this is not the direction of development now.

[Philippe_Hausler]

The initial draft of the Observation proposal included a function for tracking changes to properties accessed within the execution of a closure. This function only provided notifications on the leading edge of events, matching the timing of the willSet property observer. During the proposal review, a number of concerns were raised that leading edge notifications aren't sufficient for all use cases, and that trailing edge (i.e. didSet) observation is sometimes required.

While we recognized those concerns, we didn't feel confident that we had a comprehensive use case that would drive a didSet-based design. Therefore, we reduced the scope of the proposal to ensure a focused and robust implementation while leaving the door open for when a more general case can be added in the future. Working to ensure that such a generalization would be possible under the accepted proposal, we added a didSet variant as an experiment, marked as SPI. In retrospect, it should have been added under a experimental feature flag instead, as it is not actually needed or used by SwiftUI. That SPI is going to be removed for now, but it could be used as a draft of a more general future direction.

In addition, there were two other future directions requested during the evolution review: observation external to the instances per property and internal object observation. Some of the groundwork for these has already been laid, and, as above, learning more about the use cases for these features would help us build a comprehensive system that supports the diverse needs of the community.

[tcldr]

@Philippe_Hausler

While we recognized those concerns, we didn't feel confident that we had a comprehensive use case that would drive a didSet-based design.

This surprises me. One can argue this the other way, too. How many use cases are there outside the very specific observation needs of SwiftUI that are satisfied by the Observable mechanism as it exists today?

Observation is not a novel idea. The use cases are well documented. And as you say:

@Philippe_Hausler

During the proposal review, a number of concerns were raised that leading edge notifications aren't sufficient for all use cases, and that trailing edge (i.e. didSet) observation is sometimes required.

Observable, clearly, serves a very specific use case for SwiftUI. It's a great fit for that module and for SwiftUI layer view models specifically but beyond that, I struggle to see a use case for it.

[scanon (Steve Canon)]

@tcldr

This surprises me. One can argue this the other way, too. How many use cases are there outside the very specific observation needs of SwiftUI that are satisfied by the Observable mechanism as it exists today?

It might be more precise to say something like: we had the SwiftUI team available to give us regular feedback guiding the design of "willSet" observation, and no similar source of day-to-day feedback for "didSet" style observation. We'd prefer not to land a feature and bake it into API without getting that sort of experience first if we can avoid it. We know that there are use cases people have in mind, and we could certainly mock up small examples ourselves, but that's not the same as having a team building a whole system on top of it and giving immediate feedback when it falls short.

It's also really important to be clear that having done one now does not preclude doing the other in the future. Observable is an new feature, and as with all features in Swift we expect it to continue to grow and develop in the years to come, and encourage the community to help provide examples to guide that growth.

[tcldr]

@scanon

It might be more precise to say something like: we had the SwiftUI team available to give us regular feedback guiding the design of "willSet" observation, and no similar source of day-to-day feedback for "didSet" style observation. We'd prefer not to land a feature and bake it into API without getting that sort of experience first if we can avoid it. We know that there are use cases people have in mind, and we could certainly mock up small examples ourselves, but that's not the same as having a team building a whole system on top of it and giving immediate feedback when it falls short.

I totally understand that. The design clearly reflects this process, but simply acknowledging that there's other legitimate use cases goes a long way to reassuring me that things will be resolved in the future.

Right now, there's a void that exists for same actor synchronous observation that was previously covered by KVO/Combine. My concern was that this void was assumed filled by asynchronous sequences and Observable.

@scanon

It's also really important to be clear that having done one now does not preclude doing the other in the future. Observable is an new feature, and as with all features in Swift we expect it to continue to grow and develop in the years to come, and encourage the community to help provide examples to guide that growth.

That's great to hear, thanks for your thoughts.

[Philippe_Hausler]

@tcldr

My concern was that this void was assumed filled by asynchronous sequences and Observable.

Actor observation is a completely different beast than async sequences + Observable. The rub with actors is that we need a way of specifying identification of properties; KeyPaths don't currently suffice. There is an intersection of those "tech trees" - creating an async sequence from a property observed from an actor; but all other regards they are separate items.

Now synchronous observation (e.g. an observation of a property from within the isolation of the type) is a more related item - but those cases are where you need a distinct before/after or self-observation. A touch more related to the async sequence case, but still pretty different.

From the feedback on all the reviews; I see the problem space split out in to distinct areas. Async sequences may for some cases overlap to self-observation but it is not a 100% isomorphism in all cases. I don't see either of those being mutually exclusive, nor do I see them as direct replacements.

Thankfully for the synchronous side we do have the concept of property observers (e.g. the code-block attached to willSet/didSet) that does do a decent amount of heavy lifting. In the case of Observable, clients can write out their own withMutation to be a more "general thing".

[tcldr]

@Philippe_Hausler

Actor observation is a completely different beast than async sequences + Observable. The rub with actors is that we need a way of specifying identification of properties; KeyPaths don't currently suffice. There is an intersection of those "tech trees" - creating an async sequence from a property observed from an actor; but all other regards they are separate items.

We're in agreement here. I was referring to same-actor (rather than inter actor) synchronous observation. i.e. observation within the same isolation context – such as a MainActor instance observing another MainActor instance.

However, for inter-actor asynchronous observation – observation of one isolation context from another – this pattern of Observation seems fraught with challenges.

For anything but the simplest use cases, due to the per property observation, and per destination observation semantics of Observation you'd end up with a bunch of granular asynchronous sequences each emitting fine-grained asynchronous state updates in their own indeterminate order – as is the nature of multiple async sequences. This would be incredibly hard to reason about, and exposing this as part of Observation may mask the difficulties involved in using them safely.

Maybe one way around this would be to manage a per-property multicast async sequence that is managed by the ObservationRegistrar and shared by each concurrent observer of that property. This forces updates through a single ordered channel, guaranteeing order and effectively controlling distribution. It still puts the onus on the programmer to ensure they use a single property to distribute their actor state though.

@Philippe_Hausler

Thankfully for the synchronous side we do have the concept of property observers (e.g. the code-block attached to willSet/didSet) that does do a decent amount of heavy lifting. In the case of Observable, clients can write out their own withMutation to be a more "general thing".

This is what I'm really looking forward to, though. It looks like you were really close to having a trailing edge observation working, so I'm excited to see it realised in the near future.