Some suggestions for Swift Observation

gwendal.roue · 2023-04-15T13:04:47.995Z

Hello,

I do not want to derail the review of SE-0395: Observability, and at the same time I want to share my multi-years experience about observations in a concurrent and transactional world.

This experience is available in the GRDB SQLite toolkit: Database Observation.

The foundations of SE-0395 look pretty solid to me, as well as its SwiftUI support. I'll take them for granted. I'll suggest a different high-level api, though.

I will expose below what I think are the essential use cases of an observation api. I will sometimes take actual GRDB examples, and sometimes imagine some code inspired by SE-0395. I must warn that I'm quite far from an expert in Swift Concurrency, and I plea for reader indulgence: please don't discard my incomplete examples, and maybe try to compensate for my lack of knowledge instead.

That's enough for an introduction.

Fundamentals: Transactions, Invariants

The concept of database transaction has no direct equivalent in the Swift runtime, and that's why SE-0395 describe them as below:

Observable objects can also provide changes grouped into transactions, which coalesce any changes that are made between suspension points.

This definition looks pretty solid to me. It allows a piece of synchronous code, or ExecutorJob to craft related changes together. It makes it possible to observe related changes that are tied together by some invariants. Invariants are very important, so I'll give two simple examples.

The first example describes an invariant on a single class:

struct Player { }

// Invariant: `bestPlayers.count < totalPlayerCount`
@Observable class HallOfFame {
    var totalPlayerCount: Int
    var bestPlayers: [Player]
}

// A transaction makes it possible to observe
// those two related changes as a whole, so that
// no observer of hallOfFame can see a broken invariant.
hallOfFame.totalPlayerCount = players.count
hallOfFame.bestPlayers = Array(players.sorted { ... }.prefix(10))

The second example describes an invariant that ties multiple classes:

// Invariant: `book.author.books` contains `book`
@Observable class Book {
    unowned var author: Author
}

@Observable class Author {
    var books: [Book]
}

// A transaction makes it possible to observe
// those two related changes as a whole, so that
// no observer of book and author can see a
// broken invariant.
book.author = author
author.books.append(book)

Support for transactions and invariants is paramount. An ideal observation api does not optimize for the optimistic trivial cases. Instead, it must support complex applications with complex invariants out of the box.

Practically speaking, this means that observing a single property of an observable object is firmly a second-class use case. We'll even see below that apis that only support the observation of a single property create more problems than they solve.

Transaction Observation

The first observation tool I'll describe is called transaction observation.

The characteristics of a transaction observation are:

A transaction observation tracks a set of values.
Once the observation has started, the observer is notified of absolutely every transactions that modify at least one of the tracked values (we talk about impactful transactions).
The observer can process changes before another transaction had an opportunity to perform further changes.

The very nature of transaction observation creates strong concurrency constraints. In the context of SE-0395, the client application must be notified of a transaction at the very end of the executor job - there's no other location where observer could be notified so that the above constraints are respected.

Transaction observation has a synchronous aspect: the observer is not synchronously notified of individual changes, but it is synchronously notified at the end of the transaction.

Use cases for transaction observation are pretty demanding:

Processing changes before the operating system had any opportunity to put the application in a suspended state. This is guaranteed because of the synchronous aspect of transaction observations.
Application-side "triggers", which means additional modifications that are automatically executed in response to certain modifications.
Performing side effects on every single impactful transaction (accessing the file system, sending network requests, whatever).

GRDB support for transaction observation is the DatabaseRegionObservation type.

One creates an observation with a description of the the tracked database region:

// A GRDB transaction observation that tracks both the author and book database tables
let observation = DatabaseRegionObservation(tracking: Table("book"), Table("author"))

When one starts the observation from a transaction, the observer will be notified of all future impactful transactions, with a full guarantee that no frame will be missed. Because the transaction is started from a transaction, the observer has an extra piece of information, which is the current state of the database:

// [SIMPLIFIED]
// Start a write access (sync or async)
try dbQueue.write { _ in
    // Start the observation
    observation.start(in: dbQueue) {
        // Handle transaction:
        // Access tracked values is needed
        // Perform side effects
    }
}

GRDB api is based on callbacks, and does not provide any async sequence that wraps DatabaseRegionObservation - I'm unsure if it is possible or not, considering the next paragraph.

What's important in this api is that the callback does not provide the tracked values. Instead, the callback can access the tracked values if wanted. This works because transaction observation guarantees that no other transaction had the opportunity to perform further changes: the observer can read the latest changes. And because we're observing transactions instead of individual values, all invariants are preserved. It should be possible to implement transaction observation without ever evaluating the tracked values.

It is not very easy to imagine a SE-0395 equivalent to DatabaseRegionObservation, because we have to specify the tracked region. Below I imagine an api based on result builders - it's not perfect because it evaluates the tracked values, when all we need is a specification of the tracked values.

// MyObject1 and MyObject2 are tied by some invariant
@Observable class MyObject1 { 
    var books
    var authors
}
@Observable class MyObject2 { 
    var otherValue
}

let myObject1 = MyObject1()
let myObject2 = MyObject2()

let observation = TransactionObservation {
    myObject1.books
    myObject1.authors
}

observation.start {
    // Books or authors have been changed.
    // Access values is needed, including myObject2.
    // Perform desired side effects.
}

Alternative that does not evaluate the tracked values (I'm not thrilled either):

let observation = TransactionObservation {
    myObject1.observableProperty(\.books)
    myObject1.observableProperty(\.authors)
}

Note that myObject2 is not tracked by this observation (one could, but the example does not). Yet object2 can freely be accessed from the callback, because invariants are preserved (thanks to the transaction observation).

I have further difficulties imagining the specification of the actor that isolates transactions. It is important, though, because the observation should only be started from this actor. Otherwise, we'll drop transactions. Apologies because I have to left this to the knowledgeable reader.

As a closing note, I need to mention that this section does not address KVO and ObservableObject use cases, which focus on observation of individual properties, in a willSet/didSet fashion. Here, the observation of individual properties is a subset of the possible observations, and individual modifications are synchronously coalesced at the end of the transaction.

I'm curious to know if those "delayed synchronous notifications" address @tcldr's needs.

Value Observation

The second observation tool I'll describe is called value observation.

It's much more relaxed than transaction observation, and is very close to the SE-0395 magic that fuels SwiftUI.

The characteristics of a value observation are:

It tracks one single value (freely built from other values if needed).
It always notify an initial value before eventual changes.
It notifies fresh values on the chosen actor.
It can notify the initial value synchronously (when the observed actor and the observer actor are the same).
It may coalesce subsequent changes together into a single notification.
It may notify consecutive identical values.
It respects strict transaction ordering.
It never forgets the "last frame".

Some of those items are somewhat abstract, so let's breakdown.

Observing a single value does not mean that we can only observe a property. No, we can observe a value that is as complex as desired - just as a SwiftUI view can access as many values it needs in its body, from as many @Observable instances it needs:

// PSEUDO-CODE
let observation = ValueObservation.tracking {
    let value1 = ...
    let value2 = ...
    let sum = value1 + value2
    let value3 = ...
    return (sum, value3)
}

observation.start { (sum, value3) in
    print("Fresh sum: \(sum)")
    print("Fresh value3: \(value3)")
}

A value observation always notify an initial value before eventual changes, for three reasons:

First, it is just so handy, because the observer does not have to duplicate code in order to handle the current value, and future modifications.
Second, the initial evaluation of the value closure tells which values should be tracked (just as SE-0395 knows what a SwiftUI view needs by evaluating its body)
Third, this allows the implementation to guarantee that it won't never ever miss the "last frame".

A value observation notifies fresh values on the chosen actor, and this is how one can display on screen values that are owned by, say, an actor. Obviously the tracked value has be to Sendable when the tracked values do not belong to the same actor as the observer:

// PSEUDO-CODE
actor MyCounter {
    var count = 0
}

let counter = MyCounter()

@MainActor func display(with count: Int) { ... }

// Eventually displays 0
let observation = ValueObservation.tracking { counter.count }
observation.start { count in display(count) }

Task {
    // Eventually displays 1
    await counter.count += 1
}

A value observation can notify the initial value synchronously, when the observer and the tracked value belong to the same actor:

// PSEUDO-CODE
@MainActor class MyCounter {
    var count = 0
}

let counter = MyCounter()

@MainActor func display(with count: Int) { ... }

// Synchronously displays 0
let observation = ValueObservation.tracking { counter.count }
observation.startSynchronously { count in display(count) }

// Eventually displays 1
counter.count += 1

As in the above example, though, further modifications, even if performed synchronously after the observation has started, can be notified later if this fits the implementation strategy.

The value observation may coalesce subsequent changes together into a single notification. This is intended to ease the implementation of value observation. If the user wants to observe absolutely all changes, the user must use a transaction observation, described above.

The value observation may notify consecutive identical values. Again, this is intended to ease the implementation of value observation. The implementor only notifies duplicate values, if it can't be avoided, due to a lack of information, in order to honor the other rules. Users can use deduplication techniques if needed.

The value observation respects strict transaction ordering. For example, the values notified to an observer of a increasing counter must also be increasing. Given the two rules above, if a counter has the subsequent values (1, 2, 3), then the observer may observe (3), (1, 3), (2, 3), (1, 1, 2, 3), etc. But the observer never sees (3, 2), (2, 1), or (3, 1).

The value observation never forgets the "last frame". Even if observed values may be notified with a delay, due to the eventual actor hop, the observer must always to eventually notified of the last one. If a if a counter has the subsequent values (1, 2, 3) and then is not modified for a while, then the observer must eventually see 3. I know this paragraph does not strictly define "last" nor "for a while", but I hope the reader figures what's this rule is about.

GRDB support for transaction observation is the ValueObservation type.

A GRDB ValueObservation can be turned into a Combine publisher, or an asynchronous sequence. Obviously, the asynchronous sequence does not support the initial synchronous notification.

// A GRDB value observation that tracks all authors and all books
let observation = ValueObservation.tracking { db in
    let authors = try Author.fetchAll(db)
    let books = try Book.fetchAll(db)
    return (authors, books)
}

// Use case 1
// A publisher that publishes all values asynchronously on the main thread
let publisher = observation.publisher(in: dbQueue)

// Use case 2
// A publisher that publishes the initial value synchronously,
// on subscription, and all modifications asynchronously on the main thread.
let publisher = observation.publisher(in: dbQueue, scheduling: .immediate)

// Use case 3
// A publisher that publishes all values asynchronously on the chosen dispatch queue
let publisher = observation.publisher(in: dbQueue, scheduling: .async(onQueue: queue))

// Use case 4
// An asynchronous sequence of fresh values
let values = observation.values(in: dbQueue)

Observation of individual properties considered harmful

This last section is a critic of SE-0395 of the values(for:) asynchronous sequences.

The problem with those sequences is that they can not be composed in a way that preserves invariants.

For example, given two sequences on two properties tied by an invariant, there is no way to iterate them together in a way that preserves the invariant. The observer will eventually see values that do not go together.

struct Player { }

// Invariant: `bestPlayers.count < totalPlayerCount`
@Observable class HallOfFame {
    var totalPlayerCount: Int
    var bestPlayers: [Player]
}

let hallOfFame = HallOfFame(...)

// How to merge them together while honoring the invariant?
let counts = hallOfFame.values(for: \.totalPlayerCount)
let bestPlayers = hallOfFame.values(for: \.bestPlayers)

Those sequences are seducing, but they are a lure. Developers will want to use them, and then they will want to compose them, and then they will face broken invariants, and many of them will have difficulties understanding the root cause, or performing the needed refactoring.

That's not what Swift observation should foster. Instead, Swift observation should directly aim at invariant-preserving observations, and make it pretty clear that users who care about invariants should observe related values together, in a single observation.

I have described above two kinds of invariant-preserving observations, transaction observation, and value observations, that do not exactly address the same set of use cases.

Thanks for reading.

tera · 2023-04-15T13:40:37.961Z

Fantastically good write up.

Just one suggestion: we could use open/close brackets to group transaction changes, in case of SwiftUI the corresponding bracket opening closing could be inside SwiftUI itself (e.g.:

    openTransaction()
    actionCallout() // calls user code that can change the state
    closeTransaction()

Naturally transactions could be nested and if transaction is not open - an automatic micro transaction is open/close for individual operations:

openTransaction()
	doOp1()
	openTransaction() // could be nested
		doOp2()
	closeTransaction
closeTransaction()

doOp3() // implicit transaction will be open/closed here

By and large the current swift UI machinery of notifying view about model changes (via ObservableObject model, Published var property, and ObservedObject var in the view) looks good to me. It's the model to model observation via combine that I found substandard (too cumbersome, too manual, very easy to get wrong), so I welcome any changes in this area.

tcldr · 2023-04-15T15:15:08.151Z

gwendal.roue:

I'm curious to know if those "delayed synchronous notifications" address @tcldr's needs.

Yes, I think this would.

AFAICT any dependent property (updated as part of the observation being processed by the observer) will still be updated within the current event loop cycle. So in addition to the mentioned benefit of invariants being maintained, the necessary willSet will fire on dependent properties – synchronising animations, user events, etc.

More generally though (although perhaps because I haven't used a transactional observation system like this and so don't know any better) I'd probably be very happy something along the lines of the Value Observation you describe.

Having said that, I can definitely see the value of it, and it would be interesting to keep that kind of functionality as an open possibility.

And in that case, being greedy, I wonder if the API could be unified in some way maybe, too. Forgive me if I'm missing something that makes this not possible. Variadic Generics could open up some elegant possibilities, too.

gwendal.roue:

A GRDB ValueObservation can be turned into a Combine publisher, or an asynchronous sequence. Obviously, the asynchronous sequence does not support the initial synchronous notification.

And this is definitely the way I think SE-0395 should handle this. The primary mechanism should be something along the lines of the TransactionObservation and ValueObservation you describe, which can then be tapped-off via observation.async or similar to get the asynchronous sequence.

QuinceyMorris · 2023-04-15T18:53:37.418Z

gwendal.roue:

struct Player { }

// Invariant: `bestPlayers.count < totalPlayerCount`
@Observable class HallOfFame {
    var totalPlayerCount: Int
    var bestPlayers: [Player]
}

let hallOfFame = HallOfFame(...)

// How to merge them together while honoring the invariant?
let counts = hallOfFame.values(for: \.totalPlayerCount)
let bestPlayers = hallOfFame.values(for: \.bestPlayers)
Those sequences are seducing, but they are a lure. Developers will want to use them, and then they will want to compose them, and then they will face broken invariants, and many of them will have difficulties understanding the root cause, or performing the needed refactoring.

I see what you're saying here, but I don't agree with your conclusion. For the observer in such a system, there are no broken invariants, unless one of the following two conditions exist:

The observed source exposes its broken invariants generally.
The change notification carries multiple values out of the source to the observer, and the values it carries contain a broken invariant.

An example of #1 would be a scenario where HallOfFame is not an actor, and its properties are updated without any atomicity or other protection. In that case, you've got a more global problem just the observer. Any client of HallOfFame could see a scenario where there are 10 best players, but a total player count of only players, for example.

I don't think this is a problem for an observation system to fix. It's just a problem.

An example of #2 would be where HallOfFame was an actor, but it emitted changes to its properties synchronously across the actor isolation boundary, at moments when its invariants are temporarily broken internally to the actor.

This can't happen with SE-0395, because changes are emitted at suspension points, at which time a properly-written actor has no broken invariants.

The scenario where different change notification streams are composed is something else again, I think. It's not a broken invariant, just a reflection of the fact that values retrieved from mutable objects have different values at different times.

It's not the job of the observer to construct, decode or enforce an actor's invariants, nor to resolve the puzzle of why values change over time. It's not an error for totalPlayerCount to be observed as 3, then as 100 a moment later. By the same principle, it's not an error for totalPlayerCount to be observed as 3 at one moment, then for bestPlayers.count to be 10 a moment later, or even at the same moment.

This kind of "broken invariant", I think, is characteristic of yet another system alongside "observation" and "publication" systems: a "synchronization" system:

An observation system ensures that knowledge of value changes at a source reaches a destination, eventually but promptly enough.
A publication system ensures that changed values at a source are transmitted to a destination reliably, even when they're not/no longer visible at the source.
A synchronization system ensures that values at a source are simultaneously visible at the source by a destination, within a timing window transmitted from the source to the destination.

I think the only thing we don't completely agree on is which of these systems SE-0395 should implement. My only goal here is to avoid possibly redirecting SE-0395 away from an observation system to one of the others, and failing to provide the semantics of any of them.

tcldr · 2023-04-15T19:26:42.908Z

QuinceyMorris:

I think the only thing we don't completely agree on is which of these systems SE-0395 should implement. My only goal here is to avoid possibly redirecting SE-0395 away from an observation system to one of the others, and failing to provide the semantics of any of them.

Using your definitions of 'observation', 'publication' and 'synchronization' systems, under which category would you place:

The ObservationTracking mechanism from SE-0395
The asynchronous sequence ObservedChanges and ObservedValues mechanisms in SE-0395
The existing combine based @Published/ObservableObject mechanism.
And one for luck... KVO.

gwendal.roue · 2023-04-16T06:30:17.558Z

tera:

Just one suggestion: we could use open/close brackets to group transaction changes

SE-0395 defines a transaction as everything between two suspension points, which is a good first definition. If the Language Workgroup happens to be interested in the transaction observations that I described above, then the definition of a transaction would need to be slightly amended. The reason for this is that a transaction observer may perform further changes in response to initial changes (I called that a "trigger"). Those extra changes are wrapped in second transaction (that may itself trigger other transaction observers):

  suspension point                          suspension point
--*-----------------------------------------*--->
  < first transaction >< second transaction >

This means that "transaction" is not an exact synonym for "everything between two suspension points". From this, we may be able to expose transactions to userland, with begin/end pairs, or some kind of withTransaction { ... } api.

Maybe we need to find good reasons for such extra apis in the first place.

As for nested transactions, I don't see yet what purpose they could have.

gwendal.roue · 2023-04-16T06:42:54.751Z

tcldr:

More generally though (although perhaps because I haven't used a transactional observation system like this and so don't know any better) I'd probably be very happy something along the lines of the Value Observation you describe.

Value observation clearly aims at fulfilling the needs of GUI apps, because it is the tool that makes it possible to handle on the main actor changes that were not made on the main actor. But what value observation provides to the main actor is a "fresh value", not the "current value". The real current value may not be synchronously accessible from the main actor in the first place. And even if the current value is synchronously accessible from the main actor, it may already be different (and in this case, it is already scheduled as the next fresh value).

To take an extreme example: if a value observer increments the counter it observes, then it does create an infinite loop. But it is not a reentrant loop that leads to a stack overflow. Instead, each increment schedules the observer for another round, ad lib. And this, whether the counter and the observer belong to the same actor, or not.

And in that case, being greedy, I wonder if the API could be unified in some way maybe, too.

If you're talking about merging transaction and value observation, then no, they can't be merged, not as they were defined. They have fundamentally different behaviors w.r.t concurrency. Only transaction observation can process transactions synchronously. And only value observation can transfer observed value, as long as they're Sendable, from one actor to another.

gwendal.roue · 2023-04-16T06:52:52.984Z

QuinceyMorris:

I see what you're saying here, but I don't agree with your conclusion.

Cool :-)

This can't happen with SE-0395, because changes are emitted at suspension points, at which time a properly-written actor has no broken invariants.

Yes, I follow. An SE-0395 observation of \.bestPlayers and an observation of \.totalPlayerCount don't carry broken invariants.

But I have great concerns for the code that composes them together in order to reconstruct a pair of values (for display on screen, for example, or some more "serious" computations). It is the composer that I don't trust at all.

The only value I trust is the (bestPlayers, totalPlayerCount) pair, built synchronously from a sound HallOfFame instance. Once I observe ONE value (which may be a tuple or a complex struct) that is guaranteed to preserve invariants, I don't need to rely on composition on multiple values in a concurrent world, a problem that is known to be super hard.

That's the reason why I foster value observations that synchronously compose values together into one final observed value. The code that composes the observed value is obviously correct, so I'm 100% sure the observer will process sound values.

It's so simple!

And that's exactly what the body property of a SwiftUI view does: it synchronously composes values together, with full reliance on invariants, in a manner that is obviously correct.

SwiftUI is not a composer of views that can only access a single property. But that's exactly what the SE-0395 sequences foster. I do not see why only SwiftUI should be granted with this essential feature: observing several values, from several observable objects, in one stroke.

By the same principle, it's not an error for totalPlayerCount to be observed as 3 at one moment, then for bestPlayers.count to be 10 [...] at the same moment.

Some relaxed observers don't care about invariants, and for them it's OK to have more best players that the total count. But I think a Swift Observation api should aim first at demanding observers that need to rely on invariants. And for those observers, it is unacceptable to observe the values you describe.

Imagine if your SwiftUI view would render broken invariants! What a laugh! SE-0395 makes SwiftUI able to rely on invariants, and I firmly assert that the rest of us want similar guarantees.

Support for relaxed observers is still there, because a relaxed observer is just a specific case of demanding observer that focuses on partial sets of invariant-tied values, or a composition of such observers.

tcldr · 2023-04-16T10:16:38.237Z

Thanks for putting this all together. I think what's clear to me after reading your write-up plus the thoughts of @QuinceyMorris is that there are many flavours of Observation. I can't claim to be able to list the pros and cons of each, but in the context of SE-0395, it seems to me that, the intended primary purpose seems to be 'fulfilling the needs of GUI apps' as that was the context given by the proposal – but perhaps the aim was more general.

For me, I would like a cohesive solution, the parts of which can be used together seamlessly; using SE-0395's proposed ObservationTracking mechanism, should work well when also using the ObservedChanges/ObservedValues mechanism.

One example of what I mean by this is that: if I have some View directly observing a property on each of ObservableA and ObservableB using the ObservationTracking mechanism, then ObservableA also directly observes some property on ObservableB (a dependent property) using whatever the ObservedChanges/ObservedValues mechanism becomes, they all need to be able to synchronise in the current event loop cycle.

As SE-0395 is currently proposed, when ObservableB is updated, triggering both an observation event on the View and ObservableA, the View will receive its event in the current event loop cycle, while ObservableA will receive its update (via an async sequence) in some future event loop cycle. So ultimately the View will receive updates for this single update across multiple event loop cycles.

diagram734×862 7.6 KB

This is what I mean when I'm talking about invariants. (In this context, at least.) I realise that invariants can still be temporarily broken in the current frame/event loop cycle, but I think that's a lot easier to reason about than invariants broken across multiple frames/event loop cycles.

I do wonder if the attempt to combine both 'fulfilling the needs of GUI apps' and inter-actor observation in the same mechanism is perhaps too ambitious. Not quite satisfying either goal. Perhaps focusing the proposal on a more specific goal would lead to a better overall solution. Then, as part of a future direction, it can be considered whether or not the mechanism can be successfully expanded to achieve inter-actor observation without compromising the existing primary purpose.

gwendal.roue · 2023-04-16T10:37:07.196Z

tcldr:

I do wonder if the attempt to combine both 'fulfilling the needs of GUI apps' and inter-actor observation in the same mechanism is perhaps too ambitious. Not quite satisfying either goal. Perhaps focusing the proposal on a more specific goal would lead to a better overall solution. Then, as part of a future direction, it can be considered whether or not the mechanism can be successfully expanded to achieve inter-actor observation without compromising the existing primary purpose.

I totally understand this point of view, and I suppose it will be the point of view of the Language Workgroup.

At the same time, I pretty well know, from experience, how difficult it is to reach what you call an "ambitious" goal. Especially considering that inter-actor observation is not very different from same-actor observation, as soon as observer notifications are delayed (at the end of the suspension point, or on the next runloop cycle). Coding this right, while preserving useful characteristics, is excruciatingly difficult (not missing important "frames", synchronous start, etc, etc). It clearly belongs to the realm of the standard toolkit in my humble opinion. I'm sure no one thinks the observation tool I have described are trivially composable™. Yet I confidently assert that they address fairly common needs - surely not all needs¹, but fairly frequent ones.

SE-0395 courageously deals with the "ambitious" goal, but for SwiftUI only. This thread is intended to act as a wake-up call to the Language Workgroup and the SE-0395 author. Not only ObservationTracking.withTracking can not be used as a building block, and we actually have nothing to build upon, but the asynchronous sequences of SE-0395 fail to address the reasonable use cases that I have described in the original post above.

¹ I'm sincerely curious about arguments from KVO defenders.

tcldr · 2023-04-16T10:46:47.850Z

gwendal.roue:

At the same time, I pretty well know how difficult it is to reach what you call an "ambitious" goal. Especially considering that inter-actor observation is not very different from same-actor observation, as soon as observer notifications are delayed (at the end of the suspension point, or on the next runloop cycle). Coding this right is excruciatingly difficult, and clearly belongs to the realm of the standard toolkit in my humble opinion. I'm sure no one thinks the observation tool I have described are trivially composable™. Yet I confidently assert that they address fairly common needs - surely not all needs¹, but fairly frequent ones.

I agree this is a great goal. And I think we've discussed elsewhere some of the discoveries of communicating between actors. Better to have a single channel of communication delivering the entire state of an actor, rather than lots of little observations of different properties that inevitably end up breaking invariants, etc.

But I think that's one of my concerns: I feel that communicating intra-actor and inter-actor ends up having quite different semantics. Trying to force them together (as seems might be the goal with async sequences) seems like it could create more problems than it solves.

(Not to say that you couldn't serve both needs with a single type for example, but that the semantics are likely to be different and therefore each might be better placed with their own problem specific API. Basically, accepting that they are different.)

gwendal.roue · 2023-04-16T10:54:46.649Z

tcldr:

But I think that's one of my concerns: I feel that communicating intra-actor and inter-actor ends up having quite different semantics. Trying to force them together (as seems might be the goal with async sequences) seems like it could create more problems than it solves.

The SE-0395 sequences certainly have problems, and I believe they are an anti-pattern. The review thread is the place to mention them (many of us already have).

SE-0395 forces us into a bottom-up review: "can the provided apis fulfill reasonable needs?" This is a difficult exercise, because 1. reasoning from first principles is hard, and requires an abstract mindset 2. when first principles look sound, we do not dare criticizing them, even when they fail at fulfilling basic needs (we think "they look so smart, surely I'm a bad programmer…"). It's a pity, because the SE-0395 authors desire quality feedback.

That's why this thread uses a reversed top-down approach: start from reasonable use cases and the runtime guarantees they need, and infer apis from them. My hope is that it's much easier to judge the relevance of the suggestions.

tera · 2023-04-16T13:32:41.077Z

gwendal.roue:

As for nested transactions, I don't see yet what purpose they could have.

I am thinking of a situation where you have:

func foo() {
    openTransaction()
    change1()
    bar()
    change4()
    closeTransaction()
}

where bar might open it's own transaction for a reason of its own:

func bar() {
    openTransaction()
    change2()
    change3()
    closeTransaction()
}

In this case the outermost (in this case foo's) transaction's close should trigger the observation, and the inner bar's closeTransaction() would just decrement some counter without doing anything.

gwendal.roue · 2023-04-16T13:52:04.986Z

tera:

I am thinking of a situation where you have [a foo function that opens a transaction in which bar is called], where bar might open it's own transaction for a reason of its own.

Reasoning by analogy can be misleading, but… The way I see it, classical databases prevent transaction nesting, in order to clearly mark the point that ends the transaction (as a signal that all writes are completed, duly persisted, and ready for use). Eventually, databases have invented savepoints. Savepoints support nesting, and some people call them "nested transactions" (yeah that's confusing). But savepoints are mainly a support for rollbacking: one can undo a whole savepoint without invalidating a whole transaction. Rollbacking is not a Swift concept. We don't need savepoints.

In the end of the day, there still remains one single point in time that marks the end of writes. We can call this point the "end of the transaction". And this means that transactions don't nest.

In this context, it is a programmer error to nest transactions, and the interaction between foo() and bar() should trap (or raise an error). When a programmer wants to use explicit transaction apis, they need to think hard about transaction boundaries, and make sure they don't nest. Otherwise, we end up with sloppy code where no one knows when observers (in the context of this thread) kick in, and what value they observe. Looking at the implementation of bar doesn't tell the reader what is the content of the transaction, despite the use of explicit transaction apis. That's the recipe for muddy and unclear runtime behavior.

I'm perfectly satisfied with the definition of transactions made by SE-0395 (with one tiny amendment) and I firmly think we should start with it, without exposing any public apis for managing transactions.

tera · 2023-04-16T14:06:43.255Z

gwendal.roue:

In this context, it is a programmer error to nest transactions, and the interaction between foo() and bar() should trap (or raise an error).

I've seen this in many places and don't think there's a problem with this approach, or that it necessarily leads to a sloppy or muddy or unclear behaviour.

func moveMoneyFromAdamToBeth(amount: Int) {
	openTransaction()
	adjustAdamsAcount(-amount)
	adjustBethsAccount(amount)
	closeTransaction()
}

func adjustAdamsAcount(amount: Int) {
	openTransaction()
	adamsCurrentAccount += amount
	adamsTotalMoney += amount
	closeTransaction()
}

func adjustBethsAccount(amount: Int) {
	openTransaction()
	bethsCurrentAccount += amount
	bethsTotalMoney += amount
	closeTransaction()
}

Philippe_Hausler · 2023-04-17T16:55:00.267Z

One of the earlier prototypes of observation did have observable type defined transactions. However that fell apart when trying to make transactions across more than one object. In truth transactionality needs to be defined exteriorly to the object.

So the exterior begin/end transaction is very spiritually similar to the changes(for:) async sequence. The differential is that the begin of the transaction is demarcated at the first willChange after the iteration is initiated and then the end of the transaction is indicated by the available suspension of the isolation actor.

So consider one actor/thread doing changes quickly and then another task (or the same task even) doing the boundaries.

Production         Iteration                    Isolation Actor
willSet A                                       Busy
A changed          call to next, enqueue end 
didSet A           
willSet B          B added to the transaction
B changed                                
didSet B
willSetA           A added to the transaction
A changed                                       Available, end of transaction
didSet A

If the iteration and the isolation is on the same actor then it means that it awaits until the next available suspension. If the change and the iteration and the isolation are the same actor then it will gather up changes until the next suspension. From what I can tell on the proposal the objection is when all three are differing actors; which I agree that is hard to get those invariants right - to me however that seems like a VERY advanced use case and likely is not what folks are going to easily be able to write given ANY system (not just async/await variations.

I can easily see examples of usage for transforming a property into an AsyncSequence. Likewise I can see examples of the same actor for change and iteration. However, I know that the appetite for the closure variation of those is not going to gain traction.

The issue boils down to one key issue; the use of the trailing edge - aka didSet. As much as personally I find the trailing edge leads to more correct designs around state consistency - that more functional perspective has been expressed to me that it should not be the primary machinery in which it works.

The examples here are very close to the withTracking.
If we knew the actor upon things being accessed then we could say

func render() {
  let currentActor = ... 
  withTracking {
    print(book.author)
    print(author.books)
  } onChange: {
    currentActor.sendExecution {
     render()
    }
  }
}

void change() {
  book.author = author
  author.books.append(book)
}

We can't know the current actor exists without the tracking being async. Making the withTracking async would definitely not work for SwiftUI (but id give the exception that perhaps we can make a specialized variation that works with the main actor).

So perhaps the signature of the withTracking is an overload:

@MainActor
func withTracking(_ apply: () -> Void, onChange: @Sendable @MainActor @escaping () -> Void) -> Void
func withTracking(_ apply: @escaping () -> Void) async -> Void

Philippe_Hausler · 2023-04-17T16:56:20.715Z

The only issue I have run across with transactions being bound like that is the A B A B problem; which is solved well with closures defining them.

jeremyabannister · 2023-04-19T19:29:35.562Z

SE-0395: Observability

I wrote up some ideas that I had about observing changes to move-only struct instances, and I'm posting the link here not to insist that anyone give it more consideration than they care to, but just to make sure that its existence is visible here on this more prominent thread for anyone who is interested but would have missed it otherwise.