[Second review] SE-0395: Observability

[dmt (Dima Galimzianov)]

Are there any plans to provide a transaction concept in this API?

[Fritzt]

Maybe related:

@Observable final class HealthModel {
  private var heartRateErrorHandler: ((Swift.Error) -> Swift.Void)? = nil
}

crashes the Swift compiler on an M1 Max Xcode 15 beta 4

Reported as FB12610280

[Philippe_Hausler]

The closure property crasher is something completely different than the #if macro-misstep.

[Fritzt]

At least one can mitigate it for now using the @ObservationIgnored macro.

[jarod (Jarod Long)]

Sorry to harp on this, but I think it's an important topic to address. Should we expect to not be able to ever use property wrappers with observable types, or will this limitation be able to be lifted in the future? Are there other language features not mentioned in the proposal that are expected to not compose with observability?

Is it fair to assume that these kinds of limitations stem from implementing this feature as a macro rather than part of the language itself? Will it be possible to transition from a macro-based implementation to a language feature in the future if we end up being too constrained by what macros are capable of?

[hborla (Holly Borla)]

@jarod

Is it fair to assume that these kinds of limitations stem from implementing this feature as a macro rather than part of the language itself?

The limitation exists because the property wrapper transform applies after the macro transform. @Observable transforms stored properties into computed properties, so after the macro transform, you're left with a property wrapper applied to a computed property.

@jarod

Will it be possible to transition from a macro-based implementation to a language feature in the future if we end up being too constrained by what macros are capable of?

The built-in property wrapper transform as it exists in the language today is fundamentally not compatible with other custom property transformations, and implementing more bespoke transformations in the compiler that compose different capabilities is not the answer. It's completely reasonable for other macro transformations that add custom setters to originally-stored properties to apply some transformation to the newValue before the macro does whatever it needs to do with that value. I believe the right way for macros to support customizing their transform is by offering property-level macro attributes as customization points that allow the programmer to provide more information to the accessor macro transformation. For example, I could imagine the @Observable API offering something like this (strawman API design, do not bikeshed!):

@propertyWrapper
struct MyWrapper { ... }

@Observable 
class Model {
  @TransformedBy<MyWrapper>
  var value: Int = 100
}

The @TransformedBy attribute is a macro attribute that exists to expose information to the accessor macro applied by @Observable, instructing it to apply the transformation from MyWrapper to the value.

The beauty of this approach is that it is not limited by the property wrapper transform. A macro API can specify customization points defined by protocols that require whatever the macro customization needs, the macro can choose how or if it needs to add additional storage into each instance of the enclosing type, the macro can use the enclosing self instance directly in the property accessors without an unwieldy static subscript to workaround memory exclusivity issues caused by factoring operations on the enclosing self type into a method on the wrapper type.... the list of limitations that prevent property wrappers from being usable in many circumstances goes on. Those issues can all go away with macros, at the cost of a slightly more verbose attribute on your property.

[jarod (Jarod Long)]

@hborla

The limitation exists because the property wrapper transform applies after the macro transform. @Observable transforms stored properties into computed properties, so after the macro transform, you're left with a property wrapper applied to a computed property.

There's still a backing stored property that the Observable macro creates — can the property wrapper not be transferred there?

@hborla

The built-in property wrapper transform as it exists in the language today is fundamentally not compatible with other custom property transformations, and implementing more bespoke transformations in the compiler that compose different capabilities is not the answer. It's completely reasonable for other macro transformations that add custom setters to originally-stored properties to apply some transformation to the newValue before the macro does whatever it needs to do with that value. I believe the right way for macros to support customizing their transform is by offering property-level macro attributes as customization points that allow the programmer to provide more information to the accessor macro transformation. For example, I could imagine the @Observable API offering something like this (strawman API design, do not bikeshed!):

@propertyWrapper
struct MyWrapper { ... }

@Observable 
class Model {
  @TransformedBy<MyWrapper>
  var value: Int = 100
}

The @TransformedBy attribute is a macro attribute that exists to expose information to the accessor macro applied by @Observable, instructing it to apply the transformation from MyWrapper to the value.

The beauty of this approach is that it is not limited by the property wrapper transform. A macro API can specify customization points defined by protocols that require whatever the macro customization needs, the macro can choose how or if it needs to add additional storage into each instance of the enclosing type, the macro can use the enclosing self instance directly in the property accessors without an unwieldy static subscript to workaround memory exclusivity issues caused by factoring operations on the enclosing self type into a method on the wrapper type.... the list of limitations that prevent property wrappers from being usable in many circumstances goes on. Those issues can all go away with macros, at the cost of a slightly more verbose attribute on your property.

I don't disagree that property wrappers are limited (though I would imagine many of their limitations could be mitigated by improving them). I think macros and property wrappers both make sense as features and each have their time and place.

I've felt a little uneasy at the idea of major features like observability being implemented with macros though. It just seems inevitable that they'll be inundated with edge cases where things just don't compose nicely or at all and result in these kinds of problems.

To give a concrete example, the situation I ran into that prompted me to post here was the following.

I have a model object that previously conformed to ObservableObject for use in a SwiftUI view, and I decided to try converting it to use the new Observable macro. It contains a few properties that use a property wrapper which I use for persisting small values in UserDefaults, similar to the SwiftUI AppStorage / SceneStorage wrappers. Its usage looked like this:

@Stored("some-value") var someValue = "" {
    willSet { objectWillChange.send() }
}

After switching to Observable, I got the errors I mentioned before, which were not particularly great and only make sense if you understand the underlying details of the macro implementation. It took some time studying the proposal to figure out a workaround, which is this:

var someValue: String {
    get {
        access(keyPath: \.someValue)
        return _someValue
    }
    set {
        withMutation(keyPath: \.someValue) {
            _someValue = newValue
        }
    }
}

@ObservationIgnored @Stored("some-value") private var _someValue = ""

I'm glad to have the option to work around the problem this way, but that's not code that I want to keep around long-term. So I was really hoping that this was a temporary limitation and that I'd eventually be able to do the obvious thing and just write @Stored("some-value") var someValue = "".

The idea of something like a TransformedBy macro makes sense, but to me it feels like a band-aid fix. From the perspective of someone just using the language without closely following its development and the inner workings of new features, I think it will be very confusing if adding @Observable to a type breaks their property wrappers. Presumably there would be better error handling that points them toward using TransformedBy if that were to exist. Then that would become a weird extra bit of syntax that feels like it should be unnecessary, but you have to remember to wrap your property wrappers with in this specific situation.

On top of that, it's not something that exists today and doesn't seem likely to exist before @Observable ships, so in the mean time we're on our own and have to just hope that it's something that gets added in the coming months or years.

To be clear, I'm not arguing against these kind of marker macros in general — ObservationTracked and ObservationIgnored make perfect sense to me because they're communicating information that's specific to Observable itself. I just think it would be unfortunate if we have to use a marker to forward along information unrelated to it that we already have a syntax for.

Zooming out a bit, I worry about this general approach to building official features. I think macros are a great tool for prototyping features and for solving specific problems at the project level where you don't necessarily need to account for all the edge cases that come up in general use. But it seems really hard and maybe impossible in some cases for big, complicated macros like this one to compose as well as I think they can be reasonably expected to.

It starts to get particularly worrisome when considering other features like this being implemented as macros in the future. Reasoning about multiple layers of code transformations seems like a major headache, and the chances of them all playing nice with each other seem slim. Will they each have to add their own bespoke TransformedBy marker macros?

All that said, I understand that features aren't going to be perfect from the start and that we certainly haven't hit the limit of how far we can take observability in this form. But to go back to one of my earlier questions, I'm interested to know how much we're locking ourselves into this approach. Setting aside whether we agree that this will happen or not, if it turns out to be the case that observability as a macro is too limited, will it be possible to convert it into a language feature in a source / ABI compatible way? My gut feeling is that it would be possible since the macro is just doing syntax transformation, but I'm not sure.

Lastly, I hope I haven't come off as too negative. I only bring up the issue because I like the feature a lot and am excited to use it! Thanks to you and all the proposal authors for all your work on it.

[David_Catmull]

@jarod

There's still a backing stored property that the Observable macro creates — can the property wrapper not be transferred there?

That would require some way for the macro to communicate to the property wrapper about the new backing property, which sounds like a whole new can of worms.

[jarod (Jarod Long)]

@David_Catmull

That would require some way for the macro to communicate to the property wrapper about the new backing property, which sounds like a whole new can of worms.

How so? If I do this:

@Observable class Model {
    @MyWrapper var value = ""
}

Couldn't that be transformed into this? What needs to be communicated to the property wrapper?

class Model: Observable {
    var value: String {
        // ...
    }

    @MyWrapper var _value = ""
}

[David_Catmull]

I think only allowing for the underscore prefix is probably too limiting as a general solution. Some macros might use a different naming system.

[dmt (Dima Galimzianov)]

@dmt

Are there any plans to provide a transaction concept in this API?

This is an important topic that doesn't seem to be discussed at all. The reason why it is important is that the lack of a transaction concept significantly reduces the application scope of didSet observation and values async sequence. Often, it is necessary to update several values in response to some action, and updating them separately makes no sense.
An observer for didSet might read a value upon the update of another value and get an unexpected result.
A trivial transaction system would be to defer observer calls until the end of the transaction. A more complex variant should take into account that transactions can be nested within each other.

[bzamayo (Benjamin Mayo)]

I think the shape of withObservationTracking(_:onChange:) is slightly underspecified. It isn't intuitively obvious that the onChange callback is fired on 'willSet', and there's currently no way to get a 'didSet' callback. I seem to recall an earlier version of this proposal had an options argument that let the caller choose when the change event was fired ... maybe that should be brought back.

Or maybe it is enough to rename the onChange parameter label to willChange, to indicate the leading edge behaviour, and leave other customisation options as future work.

[tera]

@jarod

<... about using macros to implement Observation ...>

My thoughts exactly!

@dmt

This is an important topic that doesn't seem to be discussed at all. The reason why it is important is that the lack of a transaction concept significantly reduces the application scope of didSet observation and values async sequence.

I agree it is important. To implement that I guess you'd need two copies of the state: the current state and the new not yet committed state, then assign current state to be the new state in one go along with sending the observation(s) changed messages. (I designed something very similar before, in my case the two states lived on two different threads.)

[timothycosta]

I agree, and it's also surprising given this comment in the proposal:

Combine's ObservableObject produces changes at the beginning of a change event, so all values are delivered before the new value is set. While this serves SwiftUI well, it is restrictive for non-SwiftUI usage and can be surprising to developers first encountering that behavior.

Despite that comment and the fact that this is a language feature, the proposal still seems focused on SwiftUI. It would be nice to have examples in the proposal showing how this will help non-SwiftUI code as well.

@Published was designed for SwiftUI, but it is still useful in model code when you need to (rather obviously) observe changes to a specific property. Likewise for KVO. As far as I can tell based on the examples, the proposal doesn't really help in this area.

The fact that this gets rid of property wrappers is very welcome if it helps add consistency.

Aside about property wrappers

Property wrappers have lead to a lot of smelly code. E.g.

• Some property is @Published but you need didSet? Use receive(on:)
• SomeObservableObject.someProperty is not @Published? use .objectWillSend + map + removeDuplicates.
• Want a Binding but don't have @ObservedObject, @State or @Binding (i.e., not from a View)? Build a custom binding.

[hborla (Holly Borla)]

@jarod

I don't disagree that property wrappers are limited (though I would imagine many of their limitations could be mitigated by improving them). I think macros and property wrappers both make sense as features and each have their time and place.

There have been many attempts (by me and by others) to improve property wrappers with further bespoke enhancements, and the result is either a pile of unsatisfactory rules or a ton of additional language complexity for little gain. Enclosing-self access is a great example where the property wrapper model is fundamentally extremely prone to memory exclusivity violations when accessing self, and no amount of improvements to property wrappers will solve that problem well. The answer is to allow the transformation to generate code accessing self directly in the computed property accessor. That's exactly what accessor macros were designed for, which are used extensively by the observation macros in this proposal.

@jarod

I've felt a little uneasy at the idea of major features like observability being implemented with macros though. It just seems inevitable that they'll be inundated with edge cases where things just don't compose nicely or at all and result in these kinds of problems.

The main (or only?) category of semantic issues that cause existing code to change behavior unexpectedly when applying a macro are cases where stored properties turn into computed properties. The semantic difference between a stored and computed property manifests in initialization, which SE-0400: Init Accessors is meant to address by generalizing property wrapper initialization behavior, synthesized conformances of Equatable, Hashable, etc, and property wrapper application. I do not consider this to be an inundation of edge cases, and if these limitations are prohibitive for macro authors, we should pursue enhancements to macros (or general language features like init accessors that macro expansions can use) instead of baking specific macro transformations into the compiler with special semantics.

I'm also strongly against the idea that the future of Swift API design heavily privileges standard libraries over all other libraries. I suspect you also want property wrappers to work with other accessor macros that people are building, both in Apple's SDK (such as @Model in SwiftData) and in libraries within the Swift package ecosystem. Teaching a bespoke @Observable transformation about property wrappers does not solve that problem.

[jarod (Jarod Long)]

@hborla

There have been many attempts (by me and by others) to improve property wrappers with further bespoke enhancements, and the result is either a pile of unsatisfactory rules or a ton of additional language complexity for little gain. Enclosing-self access is a great example where the property wrapper model is fundamentally extremely prone to memory exclusivity violations when accessing self, and no amount of improvements to property wrappers will solve that problem well. The answer is to allow the transformation to generate code accessing self directly in the computed property accessor. That's exactly what accessor macros were designed for, which are used extensively by the observation macros in this proposal.

Sure, I don't doubt that there are some hard limits to what property wrappers can do. They aren't going to solve every problem. But I'm skeptical of the idea that they can't be or aren't worth being improved whatsoever. Even if that is the case, I think they're still useful as they exist today.

I'd be very interested in some clarity on what the vision is for macros and property wrappers in the future. The subtext I'm getting seems to be that macros are strictly superior to property wrappers and that all usage of property wrappers can be replaced with macros, so it's not important for new features to continue supporting property wrappers. Is that the idea?

@hborla

I do not consider this to be an inundation of edge cases, and if these limitations are prohibitive for macro authors, we should pursue enhancements to macros (or general language features like init accessors that macro expansions can use) instead of baking specific macro transformations into the compiler with special semantics.

I'm not sure exactly what you mean by baking macro transformations into the compiler. I don't think I was advocating for anything like that. What I was wondering is why the observable macro itself can't transform a stored property that looks like this: @MyWrapper var value = "" into this: @MyWrapper var _value = "". The macro has access to the attributes attached to the original property, so I don't see why they can't be transferred to the generated stored property.

Regarding edge cases, what I was talking about are ones that seem intrinsic to the nature of macros as independent syntax-level transformations. Imagine that we apply a second similarly-complex macro to an @Observable type. What are the odds that multiple layers of complicated, independent syntax transformations like this will just work? If it's impossible for a macro like this on its own to support basic language features like property wrappers, I think it's reasonable to be wary of a world in which many features are implemented as macros.

Again, I do think that macros are a great feature. The point I want to make is that they seem to not be a great fit for official language features that need to be as robust and composable as possible.

@hborla

I suspect you also want property wrappers to work with other accessor macros that people are building, both in Apple's SDK (such as @Model in SwiftData) and in libraries within the Swift package ecosystem. Teaching a bespoke @Observable transformation about property wrappers does not solve that problem.

I agree, but I'm confused — does that not describe the approach you were advocating for with the TransformedBy macro? I also think that teaching every macro about property wrappers would be an unfortunate situation, that is the point I was trying to make.

[hborla (Holly Borla)]

@jarod

I'm not sure exactly what you mean by baking macro transformations into the compiler. I don't think I was advocating for anything like that.

I assumed that you were advocating for a built-in code transformation for @Observable based on these statements above:

@jarod

I've felt a little uneasy at the idea of major features like observability being implemented with macros though.

@jarod

Zooming out a bit, I worry about this general approach to building official features. I think macros are a great tool for prototyping features and for solving specific problems at the project level where you don't necessarily need to account for all the edge cases that come up in general use.

@jarod

if it turns out to be the case that observability as a macro is too limited, will it be possible to convert it into a language feature in a source / ABI compatible way?

Apologies if I misunderstood, but the statements above don't sound like you're advocating for solving these sorts of problems for all macros in a general way. If that is what you're advocating for, then I think we should explore this design space independent of this review thread (I'm mindful of taking over too much of the discussion with my opinions about the future of macros and property wrappers )

[jarod (Jarod Long)]

Sorry for the confusion! There were a lot of things I was trying to communicate that I think got mixed up a bit. I'll try to summarize the core ideas.

I wanted to raise my general concern about using macros to implement official features like observability. I think that we would be better off building these sorts of things into the language. My concerns aren't limited to interactions with property wrappers — that's only the specific issue that I happened to run into during my brief time experimenting with the observable macro. Maybe I'll be proven wrong, but my gut feeling is that this won't be the only situation where the macro breaks down.

Since we'll almost certainly ship the macro-based implementation, I'd like to understand what our options are for addressing its shortcomings in the future. One option might be to transition from a macro to a built-in language feature, which is why I'm wondering if it's possible to do such a thing while maintaining compatibility.

Another option, which I believe you're advocating for, is to continually enhance macros to eliminate whatever limitations we run into. It's hard for me to engage with this idea because I haven't gotten a straight answer about exactly why the observable macro can't transfer property wrappers to the generated stored property, and I don't understand what limitation we're dealing with and whether it's possible to address it. I'm not at all opposed to improving macros, but I still believe that they are an intrinsically crude tool best reserved for specific, specialized use cases, and I don't have a lot of confidence that this path will result in observability (or any future macro-based feature) being as robust as they should be.

I hope that was clear, and I do agree that the general discussion about macros and property wrappers is best taken to a new thread, though I'm not sure that I have the time or energy to do that if I'm being honest. If I can find the time I'll try to start a thread in a few days maybe, but if anyone else is passionate about the subject please feel free to take up the reigns

On the topic of observability specifically though, can we answer the following?

1. What exactly prevents the observable macro from transferring property wrappers to the generated stored property?
2. What is the long-term vision for use cases where we would normally use property wrappers in an observable type? The following code seems to be the best we can do at the moment, and I hope we can agree that it's far from ideal. How can we improve on it in the future?

var someValue: String {
    get {
        access(keyPath: \.someValue)
        return _someValue
    }
    set {
        withMutation(keyPath: \.someValue) {
            _someValue = newValue
        }
    }
}

@ObservationIgnored @Stored("some-value") private var _someValue = ""

[Matt_McLaughlin]

@jarod

I wanted to raise my general concern about using macros to implement official features like observability. I think that we would be better off building these sorts of things into the language.
…
I'm not at all opposed to improving macros, but I still believe that they are an intrinsically crude tool best reserved for specific, specialized use cases, and I don't have a lot of confidence that this path will result in observability (or any future macro-based feature) being as robust as they should be.

This is an (admittedly vague) concern that I share as well. What I’d like to see is:

• Some broad principles (guidelines not a straight jacket) for when a given feature should be implemented as a macro vs a language feature
• When implemented as a macro some examination of the degree to which its macroness is limiting its API ergonomics, composability or performance
• If the ergonomics, composability or performance are limited, a discussion of whether implementing as a macro now will preclude moving to a more ergonomic, more composable, or more performant language feature in the future due to ABI

If the answer is “we looked at it closely and we don’t see any way in which making it a language feature would enhance ergonomics, composability or performance” then box checked and move along. I just don’t feel like we’ve had that particular discussion robustly.

[rvsrvs]

@Ben_Cohen Given that this review was to run through June 12th and we are currently about 6 weeks past that, can you provide some update on the current status of the proposal? I'm having difficulty understanding what is being taken from the thread.