[Pitch] Transactional observation of values

Evolution Pitches
, ,
62

The issue with enumerating that in the initializer is that there may be more than one weak reference and the weak references may be deep inside layers of abstraction.

Observed { 
  thingThatContainsAWeakRef.someWeakRef.property + someOtherThing.containsAWeakRef.anotherWeakRef.property
}

in that case the someWeakRef and anotherWeakRef both can be Observable and may not have direct access for construction; since the containers may also be Observable and have their properties changed!

63

When the object referenced by a weak reference gets deinitialized, does that manage to trigger an update to the observation machinery? Or would existing iterators simply stop receiving updates and fail to realize that the referenced object has gone away?

I also think this from @Nickolas_Pohilets is concerning:

It seems like a fairly large footgun that the sequence generated:

let values = Observed { obj.someOptionalProperty }

would just terminate if the produced value happened to be nil. It's perfectly reasonable to want to observe a sequence of optional values which can become nil and then turn non-nil again!

4 Likes
64

Yea that is actually an issue for SwiftUI and other uses of Observation today. I have a fix incoming for that [Observation] ensure event triggers on deinitialization passes as if all properties that are being observed have changed (for weak storage) by phausler ¡ Pull Request #79823 ¡ swiftlang/swift ¡ GitHub

Sequence et al faces this same problem; the way to fix those is to specify the return value explicitly .some(.none) etc and/or specify the type of the generic signature of the Observed.

1 Like
65

When conforming to Sequence manually you must necessarily specify the function signatures to get the level of optionality right. For the standard library's 'on the fly' sequence generation methods (which I think are most analogous to Observed as proposed here), sequence(first:next:) requires you to provide the first value to 'fix' the intended type. Only sequence(state:next:) is really vulnerable in the same way and that API feels substantially more niche to me than this general-purpose "produce an async sequence of observed values".

I realize there are workarounds—the problem in my eyes is that with the most straightforward, natural code, this API will not do what it appears to for optionally-typed properties. Especially considering it may not always be obvious that a property is optional! Consider:

@Observed
class WebPage {
  var currentURL: URL
}

let pageHostStream = Observed { webPage.currentURL.host }

IMO this is perfectly reasonable code—aside from being the most natural way to write it, I think this is how we should want users to write it. Except! Foundation.URL.host has type String?, and so, as proposed, this sequence implicitly terminates upon the first nil host, and then fails to produce further values. It's all the more concerning because a nil host may be a rare circumstance—it's entirely possible that someone could forget to test this case before shipping, and it doesn't just drop those nil values, but breaks delivery of all future values.

Consider also the process of updating existing code. If an upstream dependency changes one of their properties to be of optional type in a version upgrade, and a downstream client was observing it, I think it's eminently reasonable for the client to think

Oh, no problem, I'll just update this:

let values = Observed { obj.someProp.value }

to this:

let values = Observed { obj.someProp?.value }

But the semantics of this sequence have fundamentally changed. Worse yet, if the property that changed to be of optional type was value instead of someProp, the user may not even get a compiler error at all! If all downstream uses were such that the optionality is immaterial (e.g., assigning back to a value property somewhere else which also changed to be optional), then this will silently break.

12 Likes
66

I'd expect that the code @Jumhyn describes will be more common than the equivalent Sequence scenarios, though:

  • Many Sequence implementations are generic, and thus don't have to think about the layering of optionals; if a Sequence implementation is not generic and does need to handle optional layering intentionally, that's the responsibility of the Sequence author, not each client
  • AnyIterator, which is perhaps closest to the proposed shape of Observed, is rarely used in general (likely even more so with a concrete Optional as its element type)

In contrast, having a property of a model flip between non-nil and nil over time is common, e.g.

  • States describing single selections in user interfaces
  • Cached values, which begin as nil and become populated later

It feels particularly subtle that, given

let values = Observed { thing.property }

if property was initially of a non-Optional type, and later evolved to become Optional, that values would continue to compile without issue (given reliance on optional promotion previously), but would change significantly in behavior (never emitting a value if property's initial value were nil, for example).

EDIT: @Jumhyn beat me by seconds!

2 Likes
67

Jinx, @mpangburn! :slight_smile:

1 Like
68

I don't think that adding a type that replicates Optional is really that well formed. Would a context parameter to the closure be better then? it would be a bit odd since the termination could happen via throwing OR by calling a terminator function on the context.

69

Yeah, I definitely see your point that nil is also the idiomatic way to indicate a sequence end, and so it would be kinda odd to reinvent the wheel on that front for this API alone.

I'm kind of ambivalent as to what exactly the 'right' shape for the value-driven termination version of this API is—I haven't thought about it enough to have strong opinions. A well-established pattern throughout the language is to pass a metatype value for disambiguation when necessary, so perhaps that would work here? I.e., Observed would offer the following two initializers:

// Some details/annotations omitted for brevity
public init(_ emit: () throws(Failure) -> Element)
public init(_ type: Element.Type, _ emit: () throws(Failure) -> Element?)

So that the 'intentionally-terminate-when-optional` case could be written:

let pageURLStream = Observed(URL.self) { [weak webPage] webPage?.currentURL }

and for the non-optional version you'd write:

let pageHostStream = Observed { webPage.currentURL.host }

as desired, which would not terminate on String?.none because Element == String?.

After writing it out, this modification still isn't quite satisfying to me, though, due to the monadic nature of optional chaining. E.g., with the above code for the pageURLStream, if WebPage later changed its currentURL property to be of type URL?, we still wouldn't get a compilation failure, and a webPage going away entirely would be treated the same as the URL (perhaps temporarily!) becoming nil.

So I think there may be good reasons here to require a totally 'out-of-band' method for signaling sequence termination separately from "value became nil". I realize this puts a damper on the 'happy path' for easy observation of weak reference properties, though. :confused:

3 Likes
70

Having a context object would be a great way to disambiguate yes.

public init(_ emit: () throws(Failure) -> Element)
public init(_ emit: (Context) throws(Failure) -> Element?)

This would still require to explicitly specify the type (via the type parameter or via a metatype value) as there is no way to make the first init disappear when Element is Optional

71

To make that work id say there are a few changes needed there (but im still not fully convinced that is measurably better for all cases).

public init(_ emit: () throws(Failure) -> Element?)
public init(_ emit: (Context) throws(Failure) -> Element)

public struct Context: ~Escapable {
  public finish()
}

We likely wouldn't want the context to escape out of the scope. The other issue I'd say is important to consider is that the overload on those two might be difficult to reason about.

Is that somehow better than requiring folks to write Observed<Int?, Never> { ... }?

72

I agree we likely don't want the Context to Escape.

But I don't think offering the emit: () -> Element? overload would be necessary.
I should have been clearer.

Basically, emit: () -> Element would not support termination and emit: (Context) -> Element? would (but only through the Context).

The goal here is mostly that if someone writes

let pageHostStream = Observed<String> { webPage.currentURL.host }

And host or currentURL becomes nullable later, they would get an error (because the Context parameter is being ignored).

I was hoping to allow emit: () -> Element if Element is NOT Optional only, but that's not something Swift supports at the moment.

So in summary:

If someone cares about termination, then they'd use

Observed { [weak webPage] context in
   guard let webPage = webPage else
   {
      context.finish()
      return nil
   }
   return webPage.currentUrl.host
}

If they don't, they can simply use Observed { webPage.currentUrl.host }

73

I feel that requiring an explicit type annotation to disambiguate between "sequence continuously emitting possibly-nil values" and "sequence capable of self-terminating" is too subtle. I appreciate the goal of making weak captures easier to work with, but nil doing double-duty makes the usage patterns more challenging to reason about.

One quirk of a context parameter compared to an explicit enum return type is that it doesn't enforce any relationship between termination and emitted values in the type system:

Observed { [weak webPage] context in
   guard let webPage else {
      context.finish()
      return someOtherNonNilURL // <- Is this value emitted? Ignored? Runtime error?
   }
   return webPage.currentURL
}

To me, the implication that multiple layers of Optional might be required is enough to suggest that an explicitly named type for the outermost layer is worthwhile, even if its shape resembles Optional algebraically. (AsyncThrowingStream.Continuation.Termination is Error?? if you squint, too!)

To that end, my preference is that of @Nickolas_Pohilets:

Separating these concepts improves code readability, because when you see an example like

let pageHostStream = Observed { webPage.currentURL.host }

it's unambiguous that the produced stream never terminates, and whether you're triaging a behavioral bug or a retain cycle, that's a valuable cue.

3 Likes
74

I just tried to implement this overload and im not sure it can actually be written as intended. The root cause is that Element can be Optional itself (and there is no real way to prevent that).

I can continue to investigate ways to accomplish that but I get the feeling that it might be pushing the language into territory it may not be ideal to do so.

75

I'm sorry if I'm missing a deeper point here, but I think the idea was not to prevent Element from being Optional but simply not giving nil any special meaning in case it was.

Termination would be achieved by returning .finish instead (as opposed to, say, .next(value) or .next(nil)).

2 Likes
76

So that is presuming there is no overload as previously suggested:

Given a type that is generic upon an element (e.g. no other constraints other than say Sendable) it is not possible to disambiguate between a type that is Optional<String> and a type that is not Optional unless there is some other disambiguation in the signature.

So here is an exploration of that signature:

 struct Foo<Element> {
  init(_ emit: () -> Element) {
    print("initializer A")
  }
  
  enum Emission {
    case next(Element)
    case finish
  }
  
  init<T>(_ emit: () -> Emission) where Element == Optional<T> {
    print("initializer B")
  }
}

struct Bar {
  var baz: String
  var qoz: String?
}

// Case 1
Foo {
  Bar(baz: "test").baz
}

// Case 2
Foo {
  Bar(baz: "test").qoz
}

// Case 3
Foo {
  Foo<String>.Emission.finish
}

// Case 3 continued
Foo {
  Foo<String?>.Emission.next(Bar(baz: "test").qoz)
}

// Case 4
Foo {
  .finish // Generic parameter 'T' could not be inferred
}

// Case 5
Foo {
  .next(Bar(baz: "test").qoz)
}

The call sites cannot be disambiguated as "more specific" so case 1-3 all invoke the initializer A implementation! Case 4 doesn't even compile (since it cannot determine the housing of the .finish, but case 5 does seem to pick the right one.

So this practically means that if we were to alter the API to use a sentry type for nil parameters it would mean that could be the only version since both version of case 3 kinda puts a snag in what was ordinarily a pretty droll signature. From my experience with maintaining APIs like AsyncSequence and Publisher it is not nearly as common to produce an optional, and when folks do it is reasonable to expect termination (since that is likely the end of stream). Where that is not the case, which seems like the lesser occurrence, it is ok to have to spell it out precisely.

edit: the examples are understandably a bit thin and real-world methods are likely much more complex.

77

SwiftUI often cheats overload resolution with @_disfavoredOverload; the version below seems to invoke the intended initializers for me (A for 1 & 2, B for cases 3 & 5).

Note that case 4 (omitted below) is equivalent to an Optional-taking closure that contains only return nil; it's not unique to the overload resolution here, just generally possible of a generic enum which contains a case agnostic to its generic parameters.

Note that in the below code, no scenario is customized for Optional<Element>: the behaviors fall out naturally of the same generic type signatures:

  • Emitting optional values out of an endless sequence is fine via initializer A
  • Terminating a sequence of optional values early is fine via initializer B; this still requires specifying .finish as with any other type argument
struct Foo<Element> {
 @_disfavoredOverload
 init(_ emit: () -> Element) {
   print("initializer A")
 }

 init(_ emit: () -> Emission<Element>) {
   print("initializer B")
 }
}

 enum Emission<Element> {
   case next(Element)
   case finish
 }

struct Bar {
 var baz: String
 var qoz: String?
}

// Case 1
_ = Foo {
  Bar(baz: "test").baz // A
}

// Case 2
_ = Foo {
  Bar(baz: "test").qoz // A
}

// Case 3
_ = Foo {
  Emission<String>.finish // B
}

// Case 3 continued
_ = Foo {
  Emission<String?>.next(Bar(baz: "test").qoz) // B
}

// Case 5
_ = Foo {
   next(Bar(baz: "test").qoz) // B
}
2 Likes
78

But if I am understanding the objections correctly:

_ = Foo {
  Bar(baz: "test").qoz // A
}

That is expected by the objection to not compile and force a usage of the B variant. Which won't work.

That is roughly equivalent to writing out the signatures manually:

Observed {
  someBarContainingObject.bar.qoz //terminating when qoz is nil
}

Observed<String?> {
  Optional<String?>.some(  someBarContainingObject.bar.qoz) //not terminating when qoz is nil
}
79

I think the goal would be that this compiles as written, and results in a sequence of String? elements, with nil and non-nil elements intermingled.

2 Likes
80

Right, in either case for constructing the 'empty' observed sequence which produces no values you need to provide the type context somehow.

I will stake out my personal position that I think it would be a huge mistake to implement this API such that Observed { obj.someOptionalProperty } doesn't 'do what it says on the tin' and produce a sequence of value changes for someOptionalProperty—if this sequence will simply stop producing values as soon as someOptionalProperty happens to take on a nil value, I think that would be a huge footgun.

I work in a codebase where this sort of property-based observation is super commonplace, and it's also exceedingly common to have properties which take on optional values only to be made non-optional at some point in the future. To me, it seems like weak references are the exception where you can have a general principle of 'once it's nil, it'll never come back'.

Hm? No, picking the A variant here is exactly right in my eyes—this should produce a sequence which triggers on all value updates of qoz, and produces all values, including nils.

9 Likes
81

Note; the A variant in that example is the "terminate upon nil" in the result.