I am totally fine with my contributions being discarded due to missing details, but I would regret if your interest was not piqued.
At the risk of repeating myself, when one starts a Task that iterates observed changes, no event marks the beginning of the actual observation, so it is impossible to know when it is possible to start performing changes with the guarantee of actual observation.
// Given this task, the rest of the application has no way to know
// when object.someProperty can be modified, with the
// guarantee that the really important processing is done.
Task {
for await value in object.values(for: \.someProperty) {
// Really important processing of the value
}
}
I agree this appears to be a genuine defect in the proposal (unless there is some mitigation in the proposal text that I missed). KVO solves this problem with an "initial" option that causes the observation to fire as soon as it's established. It's usually correct or at least harmless to use this option in KVO.
It seems to me that if SE-0395 is not going to offer any similar option, then the AsyncSequences created in accordance with the proposal should unconditionally return the current value once, the first time their sequence is accessed.
I remember frequently relying on the initial value option in KVO. I'm surprised that's not coming over. However, I think it might be possible to prepend the initial value by chaining an iterator that emits the current value and then ends to the iterator produced by object.values(for: \.someProperty).
I think that could look like this using the async algorithms package
Task {
for await value in chain([object.someProperty].async, object.values(for: \.someProperty)) {
// Really important processing of the value
}
}
It does seem to have been a common enough use case (admittedly anecdotally) from the KVO days, though, that it could be considered for 1st class feature in this system.
Enabling Observable types to react to being observed would not only help with guaranteeing all values are seen, but would enable lazy observation. This is particularly important for expensive (for a variety of meanings of the word) observations, like long running computations or network requests. It would be great if I could have an Observable network stack that only starts requests when observers are added and automatically tears them down when the last observer is removed. You could apply the same thing to location managers on mobile, background computations, or system events. As a general solution to the observability problem in Swift, this proposal should be more expansive, not less.
Thanks for those examples. Can you think of any ways in which these should be considered important for the initial proposal, versus valuable future directions for further proposals?
This specific concern still seems to be a rephrasing of the "how do I avoid missing changes". Would this be addressed by the addition of an "initial + subscribe" capability as @QuinceyMorris described as being there in KVO? Notwithstanding @Jon_Shier's other reasons why observed ties knowing they are being observed being useful. Apologies if I'm missing some subtly here.
It would be great if I could have an Observable network stack that only starts requests when observers are added and automatically tears them down when the last observer is removed
That seems out of scope in the context of this proposal.
As a general solution to the observability problem in Swift, this proposal should be more expansive, not less.
I have to disagree, IMO it should be as basic/slim as possible (while covering the capabilities described in the current proposal)
Optionally including the initial value in the sequence of changes does eliminate the possibility of race conditions assuming it's implemented correctly (which should be a given, but I have seen this implemented as "send initial value and then subscribe" which is invalid).
I think for actor-isolated objects reading the value before observing manually works fine, but for mutable Sendable objects it's inherently racy. This is both an important and relatively easy thing to fix, so IMO it absolutely makes sense to be in the initial version.
The scope of the proposal can be whatever the author wants; there's no inherent set of features for a language observability feature. But it makes no sense to me to limit such a fundamental feature of the language.
As @tgoyne pointed out, there's a fundamental correctness issue here. I was merely providing additional motivation for solving it in a way that useful for other things. It could be solved internally but I see no value in saving features for future proposals that are unlikely to happen. In the history of Swift, how often have major features like this seen major new features? How often has any enhancement happened in a useful timeframe? Pretty much every effort made to document and propose future directions has ended in failure. I'd rather wait longer for a more capable feature than get it now and hope the features I want are added in the future.
First, do composed Observable types do anything special right now? e.g. If I nest an Observable inside another, what happens? Does adding observers to an observed value work?
Second, was there any consideration given to sibling Observables? That is, having one Observable trigger event emission in another? This can be useful if you want a connection between Observables that otherwise don't know about each other. Another solution may be to compose them together in a parent manually, but that was my other question.
I think it would also be useful to have a more thorough explanation of the role isolation plays in this system. This would appear to be the first standard library API that uses an isolation actor, and the proposal mentions isolation in regards to atomicity, but further explanation in this thread has complicated the situation. Namely, the (apparent) limitation that Observables can only be observed from their isolating actor. This would be rather surprising coming from Combine or KVO and for an async system in general, especially if the observed values were Sendable.
Makes me think perhaps moving to async sequences was not the right call. It's an inherited limitation. A simple callback-based observer would have solved the problem:
await object.values(for: \.someProperty) { value in
// Really important processing of the value
}
// guaranteed to have at least one observer from here on
Prepending the initial value to the async sequence does (indirectly) solve it too, but also complicates other matters. Since it needs to be an opt-in behavior, it will push the decision of getting the initial value to the user, which is error prone for multiple reasons (and also not what we want to achieve in the first place).
I guess another way to approach this and keep async sequences would be to have an explicit sendable observer:
let observer = await object.observe(\.someProperty)
// guaranteed to have at least one observer from here on
Task {
for await value in observer.values {
// Really important processing of the value
// Would receive the last value wrt to the point were observation started (possibly no value)
}
}
Needless to say both of these complicate the API, but I think the issue you described is valid and covering it should be at least considered for the initial iteration.
This is not true. Several proposals have had follow-on proposals that build on them. SE-0244 (opaque result types) called out future direction of using the same syntax for arguments, later implemented in SE-0328 and SE-0341. SE-0309 (unlock existentials) called out future directions implemented in SE-0353. While not called out in the proposals, if/switch expressions came up as a future direction during SE-0255, and strongly-typed dynamic member lookup came about as a future direction discussed during SE-0195.
Lest these be dismissed with "ah, but no true future scotsman ever gets implemented" I should note that many future directions are along the lines of "this is maybe a bad idea, but this proposal doesn't rule it out" (personally I put reasync in this camp, but there are plenty of less controversial examples), or "this is a future possible direction, but meh" (SE-0268 has one of these) which are unsurprisingly unimplemented.
Other future directions are extremely complex to implement, and so are still expected but not yet here – for example, variadic generics bring us very close to generalized tuple conformances as anticipated in SE-0283, and improvements in the generics system mean we are close to getting self-conformance as anticipated by SE-0309. Others have proven too difficult after many attempts, for example, property wrapper enclosing self access (future direction of SE-0258, possibly now superseded by macros).
Where the future direction of observable types being notified they're being observed fits in this spectrum, I don't know.
One thing I do know is that the language workgroup as more than once given guidance that being against accepting a proposal purely because you want to force a future direction to be included now is not a valid issue to raise during a review. So
Right – I'm just trying to tease apart the two things for the purpose of reviewing this proposal:
There is clearly a correctness need for getting the initial value, then getting updates, without missing updates in-between. If the proposal doesn't give a good way of achieving this (I'd like to hear from @Philippe_Hausler on whether actually this is easily done with what's proposed after all) then that probably needs fixing in this proposal.
You've given several different good motivations for knowledge of being observed, but those may belong in future directions unless there's a reason why they might not be possible to add such a feature later with this design.
Can we experiment with some sort of implementation of this? As I understand it, proposals are supposed to have at least a prototype implementation prior to review. I think there is something because I remember seeing Observability PRs into the swift repo but how should we access them? Should this and usage instructions be specified in the proposal and in the forums?
Next question. Can we observe the key path \.self? I might like to do this if the list of key paths will always fairly exhaustively list the properties of the type. And if we can use \.self, is this even the right API to use? e.g.:
func processChanges(_ object: MyObject) async {
for await theSameObject in object.values(for: \.self) {
print(theSameObject)
}
}
I believe you can access the proposal in a nightly toolchain via import _Observation. @Philippe_Hausler is that right? If so, please can you add that to the proposal?
_Observation is available in the toolchain, but trying to use @Observable fails with an error.
External macro implementation type 'ObservationMacros.ObservableMacro' could not be found for macro 'Observable'; the type must be public and provided via '-load-plugin-library'
So there may be more setup needed for anyone to test this out. Or is the macro not available yet?
To fully answer this question we'd have to design and implement the feature. Barring that I guess we can make a few assumptions.
Adding additional protocol requirements is possible, as long as a default implementation can be provided. This may be further enhanced by the @Observable macro, as it could allow more complex implementations to be generated for free. However, that doesn't help any manual implementations, though we don't know how popular those will be yet.
Additional protocol requirements which add types are much harder to add in a compatible way, perhaps even impossible. And even if possible, back deployment is currently impossible, except in extreme circumstances (like concurrency). So, at best, this future direction would have limited deployment potential.
While actual observer storage isn't exposed in the proposal, the ObservationRegistrar vends the changes and values sequences. Without knowing exactly how things are connected under the hood I speculate that the required API could live on this type.
While the Subject under observation is modeled in these APIs, there doesn't seem to be any representation of the consuming observer. We could create API to indicate when new sequences are vended, and another that's called when the sequences have been cancelled (the important events under consideration), but it seems like there wouldn't be a way to provide information about the observer, or model (much) state about the observations themselves without introducing a new type.
Altogether I'd say that, if it's possible to add new protocol hooks to Observable and concrete methods backing those hooks to ObservationRegistrar in a stable way, it may be possible to build a very simple version of this feature which, while deployment limited, may accomplish the goal. However, it seems likely that version will be less capable than a version built now that can expose a full encapsulation of observation state. Someone with more specific requirements will have to evaluate this hypothetical.
Yet "missing frames" is not quite the problem. The main problem is the "missing last frame". It's OK to coalesce several changes together (and drop frames). But it must be possible to eventually process the last. In other words, if I see a persistent "foo" on the screen when the last set value was "bar", this is a fail. It's unclear how to achieve this basic task with the proposed asynchronous sequences, and I'm afraid this is not possible at all, or only possible with caveats that are yet to be 1. disclosed and 2. evaluated in this review.
Apologies if I'm missing some subtly here
I wish I could help, but I've already written a lot here and in the [Pitch] Observation [Revised] thread. Considering the lack of answers to several concerns there (this one, but others, and of course not only mine), I don't know what to answer.
but even with that, the simplest code fails to compile, though I do get a macro expansion.
import _Observation
@Observable public final class MyObject {
}
Macro Expansion
let _registrar = ObservationRegistrar<MyObject >()
public nonisolated func changes<Isolation: Actor>(
for properties: TrackedProperties<MyObject >,
isolatedTo isolation: Isolation
) -> ObservedChanges<MyObject , Isolation> {
_registrar.changes(for: properties, isolatedTo: isolation)
}
public nonisolated func values<Member: Sendable>(
for keyPath: KeyPath<MyObject , Member>
) -> ObservedValues<MyObject , Member> {
_registrar.values(for: keyPath)
}
private struct _Storage {
}
private var _storage = _Storage()
// original-source-range: /Users/allenh/personal-projects/ObservibilityThing/ObservibilityThing/Model.swift:11:1-11:1
There appears to be something wrong with how the generated swift code is compiled because it produces a bunch of nonsensical errors. I took Xcode out of the equation just to be sure, and compiled via /Library/Developer/Toolchains/swift-5.9-DEVELOPMENT-SNAPSHOT-2023-04-12-a.xctoolchain/usr/bin/swiftc Model.swift
Errors
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:5: error: circular reference
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
Model.swift:10:32: note: through reference here @Observable public final class MyObject {
^
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:1: note: through reference here
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:5: note: through reference here
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:5: note: through reference here
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:5: note: through reference here
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:18: error: 'ObservationRegistrar' is only available in macOS 9999 or newer
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:5: error: circular reference
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:1: note: through reference here
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:5: note: through reference here
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:5: note: through reference here
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:3:5: note: through reference here
let _registrar = ObservationRegistrar()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
Model.swift:10:32: note: add @available attribute to enclosing class @Observable public final class MyObject {
^
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:22:13: error: circular reference
private var _storage = _Storage()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
Model.swift:10:32: note: through reference here @Observable public final class MyObject {
^
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:22:9: note: through reference here
private var _storage = _Storage()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:22:13: note: through reference here
private var _storage = _Storage()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:22:13: note: through reference here
private var _storage = _Storage()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:22:13: note: through reference here
private var _storage = _Storage()
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:6:19: error: 'TrackedProperties' is only available in macOS 9999 or newer
for properties: TrackedProperties,
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:5:25: note: add @available attribute to enclosing instance method
public nonisolated func changes<Isolation: Actor>(
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
Model.swift:10:32: note: add @available attribute to enclosing class @Observable public final class MyObject {
^
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:8:6: error: 'ObservedChanges' is only available in macOS 9999 or newer
) -> ObservedChanges<MyObject , Isolation> {
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:5:25: note: add @available attribute to enclosing instance method
public nonisolated func changes<Isolation: Actor>(
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
Model.swift:10:32: note: add @available attribute to enclosing class @Observable public final class MyObject {
^
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:14:6: error: 'ObservedValues' is only available in macOS 9999 or newer
) -> ObservedValues<MyObject , Member> {
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
/var/folders/7r/pftmwhnx6_l1418syflx25_80000gn/T/swift-generated-sources/@_swiftmacro_5Model8MyObjectC10ObservablefMm.swift:12:25: note: add @available attribute to enclosing instance method
public nonisolated func values<Member: Sendable>(
^
Model.swift:10:26: note: in expansion of macro 'Observable' here @Observable public final class MyObject {
^~~~~~~~~~~~~~~~
Model.swift:10:32: note: add @available attribute to enclosing class @Observable public final class MyObject {
^
