Hello,
I do not want to derail the review of SE-0395: Observability, and at the same time I want to share my multi-years experience about observations in a concurrent and transactional world.
This experience is available in the GRDB SQLite toolkit: Database Observation.
The foundations of SE-0395 look pretty solid to me, as well as its SwiftUI support. I'll take them for granted. I'll suggest a different high-level api, though.
I will expose below what I think are the essential use cases of an observation api. I will sometimes take actual GRDB examples, and sometimes imagine some code inspired by SE-0395. I must warn that I'm quite far from an expert in Swift Concurrency, and I plea for reader indulgence: please don't discard my incomplete examples, and maybe try to compensate for my lack of knowledge instead.
That's enough for an introduction.
Fundamentals: Transactions, Invariants
The concept of database transaction has no direct equivalent in the Swift runtime, and that's why SE-0395 describe them as below:
Observable objects can also provide changes grouped into transactions, which coalesce any changes that are made between suspension points.
This definition looks pretty solid to me. It allows a piece of synchronous code, or ExecutorJob
to craft related changes together. It makes it possible to observe related changes that are tied together by some invariants. Invariants are very important, so I'll give two simple examples.
The first example describes an invariant on a single class:
struct Player { }
// Invariant: `bestPlayers.count < totalPlayerCount`
@Observable class HallOfFame {
var totalPlayerCount: Int
var bestPlayers: [Player]
}
// A transaction makes it possible to observe
// those two related changes as a whole, so that
// no observer of hallOfFame can see a broken invariant.
hallOfFame.totalPlayerCount = players.count
hallOfFame.bestPlayers = Array(players.sorted { ... }.prefix(10))
The second example describes an invariant that ties multiple classes:
// Invariant: `book.author.books` contains `book`
@Observable class Book {
unowned var author: Author
}
@Observable class Author {
var books: [Book]
}
// A transaction makes it possible to observe
// those two related changes as a whole, so that
// no observer of book and author can see a
// broken invariant.
book.author = author
author.books.append(book)
Support for transactions and invariants is paramount. An ideal observation api does not optimize for the optimistic trivial cases. Instead, it must support complex applications with complex invariants out of the box.
Practically speaking, this means that observing a single property of an observable object is firmly a second-class use case. We'll even see below that apis that only support the observation of a single property create more problems than they solve.
Transaction Observation
The first observation tool I'll describe is called transaction observation.
The characteristics of a transaction observation are:
- A transaction observation tracks a set of values.
- Once the observation has started, the observer is notified of absolutely every transactions that modify at least one of the tracked values (we talk about impactful transactions).
- The observer can process changes before another transaction had an opportunity to perform further changes.
The very nature of transaction observation creates strong concurrency constraints. In the context of SE-0395, the client application must be notified of a transaction at the very end of the executor job - there's no other location where observer could be notified so that the above constraints are respected.
Transaction observation has a synchronous aspect: the observer is not synchronously notified of individual changes, but it is synchronously notified at the end of the transaction.
Use cases for transaction observation are pretty demanding:
- Processing changes before the operating system had any opportunity to put the application in a suspended state. This is guaranteed because of the synchronous aspect of transaction observations.
- Application-side "triggers", which means additional modifications that are automatically executed in response to certain modifications.
- Performing side effects on every single impactful transaction (accessing the file system, sending network requests, whatever).
GRDB support for transaction observation is the DatabaseRegionObservation
type.
One creates an observation with a description of the the tracked database region:
// A GRDB transaction observation that tracks both the author and book database tables
let observation = DatabaseRegionObservation(tracking: Table("book"), Table("author"))
When one starts the observation from a transaction, the observer will be notified of all future impactful transactions, with a full guarantee that no frame will be missed. Because the transaction is started from a transaction, the observer has an extra piece of information, which is the current state of the database:
// [SIMPLIFIED]
// Start a write access (sync or async)
try dbQueue.write { _ in
// Start the observation
observation.start(in: dbQueue) {
// Handle transaction:
// Access tracked values is needed
// Perform side effects
}
}
GRDB api is based on callbacks, and does not provide any async sequence that wraps DatabaseRegionObservation
- I'm unsure if it is possible or not, considering the next paragraph.
What's important in this api is that the callback does not provide the tracked values. Instead, the callback can access the tracked values if wanted. This works because transaction observation guarantees that no other transaction had the opportunity to perform further changes: the observer can read the latest changes. And because we're observing transactions instead of individual values, all invariants are preserved. It should be possible to implement transaction observation without ever evaluating the tracked values.
It is not very easy to imagine a SE-0395 equivalent to DatabaseRegionObservation
, because we have to specify the tracked region. Below I imagine an api based on result builders - it's not perfect because it evaluates the tracked values, when all we need is a specification of the tracked values.
// MyObject1 and MyObject2 are tied by some invariant
@Observable class MyObject1 {
var books
var authors
}
@Observable class MyObject2 {
var otherValue
}
let myObject1 = MyObject1()
let myObject2 = MyObject2()
let observation = TransactionObservation {
myObject1.books
myObject1.authors
}
observation.start {
// Books or authors have been changed.
// Access values is needed, including myObject2.
// Perform desired side effects.
}
Alternative that does not evaluate the tracked values (I'm not thrilled either):
let observation = TransactionObservation {
myObject1.observableProperty(\.books)
myObject1.observableProperty(\.authors)
}
Note that myObject2
is not tracked by this observation (one could, but the example does not). Yet object2
can freely be accessed from the callback, because invariants are preserved (thanks to the transaction observation).
I have further difficulties imagining the specification of the actor that isolates transactions. It is important, though, because the observation should only be started from this actor. Otherwise, we'll drop transactions. Apologies because I have to left this to the knowledgeable reader.
As a closing note, I need to mention that this section does not address KVO and ObservableObject use cases, which focus on observation of individual properties, in a willSet
/didSet
fashion. Here, the observation of individual properties is a subset of the possible observations, and individual modifications are synchronously coalesced at the end of the transaction.
I'm curious to know if those "delayed synchronous notifications" address @tcldr's needs.
Value Observation
The second observation tool I'll describe is called value observation.
It's much more relaxed than transaction observation, and is very close to the SE-0395 magic that fuels SwiftUI.
The characteristics of a value observation are:
- It tracks one single value (freely built from other values if needed).
- It always notify an initial value before eventual changes.
- It notifies fresh values on the chosen actor.
- It can notify the initial value synchronously (when the observed actor and the observer actor are the same).
- It may coalesce subsequent changes together into a single notification.
- It may notify consecutive identical values.
- It respects strict transaction ordering.
- It never forgets the "last frame".
Some of those items are somewhat abstract, so let's breakdown.
Observing a single value does not mean that we can only observe a property. No, we can observe a value that is as complex as desired - just as a SwiftUI view can access as many values it needs in its body
, from as many @Observable
instances it needs:
// PSEUDO-CODE
let observation = ValueObservation.tracking {
let value1 = ...
let value2 = ...
let sum = value1 + value2
let value3 = ...
return (sum, value3)
}
observation.start { (sum, value3) in
print("Fresh sum: \(sum)")
print("Fresh value3: \(value3)")
}
A value observation always notify an initial value before eventual changes, for three reasons:
- First, it is just so handy, because the observer does not have to duplicate code in order to handle the current value, and future modifications.
- Second, the initial evaluation of the value closure tells which values should be tracked (just as SE-0395 knows what a SwiftUI view needs by evaluating its
body
) - Third, this allows the implementation to guarantee that it won't never ever miss the "last frame".
A value observation notifies fresh values on the chosen actor, and this is how one can display on screen values that are owned by, say, an actor. Obviously the tracked value has be to Sendable
when the tracked values do not belong to the same actor as the observer:
// PSEUDO-CODE
actor MyCounter {
var count = 0
}
let counter = MyCounter()
@MainActor func display(with count: Int) { ... }
// Eventually displays 0
let observation = ValueObservation.tracking { counter.count }
observation.start { count in display(count) }
Task {
// Eventually displays 1
await counter.count += 1
}
A value observation can notify the initial value synchronously, when the observer and the tracked value belong to the same actor:
// PSEUDO-CODE
@MainActor class MyCounter {
var count = 0
}
let counter = MyCounter()
@MainActor func display(with count: Int) { ... }
// Synchronously displays 0
let observation = ValueObservation.tracking { counter.count }
observation.startSynchronously { count in display(count) }
// Eventually displays 1
counter.count += 1
As in the above example, though, further modifications, even if performed synchronously after the observation has started, can be notified later if this fits the implementation strategy.
The value observation may coalesce subsequent changes together into a single notification. This is intended to ease the implementation of value observation. If the user wants to observe absolutely all changes, the user must use a transaction observation, described above.
The value observation may notify consecutive identical values. Again, this is intended to ease the implementation of value observation. The implementor only notifies duplicate values, if it can't be avoided, due to a lack of information, in order to honor the other rules. Users can use deduplication techniques if needed.
The value observation respects strict transaction ordering. For example, the values notified to an observer of a increasing counter must also be increasing. Given the two rules above, if a counter has the subsequent values (1, 2, 3), then the observer may observe (3), (1, 3), (2, 3), (1, 1, 2, 3), etc. But the observer never sees (3, 2), (2, 1), or (3, 1).
The value observation never forgets the "last frame". Even if observed values may be notified with a delay, due to the eventual actor hop, the observer must always to eventually notified of the last one. If a if a counter has the subsequent values (1, 2, 3) and then is not modified for a while, then the observer must eventually see 3. I know this paragraph does not strictly define "last" nor "for a while", but I hope the reader figures what's this rule is about.
GRDB support for transaction observation is the ValueObservation
type.
A GRDB ValueObservation
can be turned into a Combine publisher, or an asynchronous sequence. Obviously, the asynchronous sequence does not support the initial synchronous notification.
// A GRDB value observation that tracks all authors and all books
let observation = ValueObservation.tracking { db in
let authors = try Author.fetchAll(db)
let books = try Book.fetchAll(db)
return (authors, books)
}
// Use case 1
// A publisher that publishes all values asynchronously on the main thread
let publisher = observation.publisher(in: dbQueue)
// Use case 2
// A publisher that publishes the initial value synchronously,
// on subscription, and all modifications asynchronously on the main thread.
let publisher = observation.publisher(in: dbQueue, scheduling: .immediate)
// Use case 3
// A publisher that publishes all values asynchronously on the chosen dispatch queue
let publisher = observation.publisher(in: dbQueue, scheduling: .async(onQueue: queue))
// Use case 4
// An asynchronous sequence of fresh values
let values = observation.values(in: dbQueue)
Observation of individual properties considered harmful
This last section is a critic of SE-0395 of the values(for:)
asynchronous sequences.
The problem with those sequences is that they can not be composed in a way that preserves invariants.
For example, given two sequences on two properties tied by an invariant, there is no way to iterate them together in a way that preserves the invariant. The observer will eventually see values that do not go together.
struct Player { }
// Invariant: `bestPlayers.count < totalPlayerCount`
@Observable class HallOfFame {
var totalPlayerCount: Int
var bestPlayers: [Player]
}
let hallOfFame = HallOfFame(...)
// How to merge them together while honoring the invariant?
let counts = hallOfFame.values(for: \.totalPlayerCount)
let bestPlayers = hallOfFame.values(for: \.bestPlayers)
Those sequences are seducing, but they are a lure. Developers will want to use them, and then they will want to compose them, and then they will face broken invariants, and many of them will have difficulties understanding the root cause, or performing the needed refactoring.
That's not what Swift observation should foster. Instead, Swift observation should directly aim at invariant-preserving observations, and make it pretty clear that users who care about invariants should observe related values together, in a single observation.
I have described above two kinds of invariant-preserving observations, transaction observation, and value observations, that do not exactly address the same set of use cases.
Thanks for reading.