SE-0385: Custom Reflection Metadata

Hello, Swift community!

The review of SE-0385 "Custom Reflection Metadata" begins now and runs through February 7th, 2023.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager by email. When contacting the review manager directly, please keep the proposal link at the top of the message and put "SE-0385" in the subject line.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at:

https://github.com/apple/swift-evolution/blob/main/process.md

Thank you,

Doug Gregor
Review Manager

20 Likes

Is this available in the nighty-main snapshot behind a flag?

I read the proposal with a fair degree of attention, and I have a few thoughts/questions which I’ll write up more thoroughly separately, but to begin with I just want to ask about a central aspect of this that isn’t entirely clear to me. When you say “the compiler will synthesize a call to ___”, when does this call get executed? What causes it to be executed? Where is the resulting instance accessible from? Or is the idea that the initialized instance is immediately discarded and the initializers are therefore expected to be side-effect-ful?

1 Like

The call is invoked through the reflection API. From the proposal:

Instances of metadata types are initialized in the Reflection query to gather the metadata.

This is done in the implementation of Attribute.allInstances. The compiler-synthesized call is emitted into a "generator function" with a record of that generator function in the reflection metadata. The implementation of Attribute.allInstances gathers the metadata and initializes all of the resulting instances.

The only way to retrieve metadata instances is through the reflection query, which is accessible anywhere that imports the library containing the reflection API. In the proposal, that's a dedicated Reflection library, as described in-detail in this pitch.

1 Like

Yes, you can use it via -enable-experimental-feature RuntimeDiscoverableAttrs

3 Likes

Would it be possible to rename (or alias) it to CustomReflectionMetadata to reflect the name of the proposal? Or would that happen when it moves to preview?

We'll remove the flag and rename attribute if the proposal gets accepted.

From my understanding of the proposal, any attribute that can be applied to foo() in the following code can also be applied to bar(_:), right? In other words there’s no way to restrict it to only methods or only global functions.

struct S {
  func foo() {}
}

func bar(_: S) {}

I can’t think of a case in which that’s an issue, but I want to be sure I understand this correctly.

Is it possible to attach an attribute to an initializer? If so, what type would the attachedTo parameter be? Intuitively I would expect () -> T or (T.Type) -> T if the init takes no parameters, but it would be nice to see that spelled out in the proposal.

Yes, there is no way disambiguate between these two cases by function types alone.

1 Like

Just a heads-up that the link is broken on swift.org/swift-evolution (missing a 0 before "385").

-1 from me.

I understand the motivation for test and plug-in types, so the custom metadata attributes make sense to me for types. However, the proposal doesn’t explicitly specify why functions should have attached metadata. In fact, many frameworks usually compartmentalize complex behavior in types instead of functions (such as SwiftUI Views being structs not functions); thus, it isn’t clear why functions would require custom metadata.

There is also insufficient motivation for adding arguments in the metadata attribute @Attribute(arg1: 1), such as for the editor-command metadata. The proposal isn’t convincing that the arguments-in-attribute approach is better than conforming to a protocol that requires these arguments as computed properties:

protocol MyAttributeRequirements {
  var arg1: Int { get }
}
@Attribute // Requires Self : MyAttributeRequirements 
struct MyStruct: MyAttributeRequirements {
  var arg1: Int { 1 }
}

Of course, the protocol approach is verbose, but it’s also a less complex language feature.

Without proper motivation for the aforementioned use cases, the alternative of simply expanding existing language features is a lot more appealing. Particularly, we could expose a reflection API that looks up all types conforming to a specific protocol. Looking up protocol conformances would satisfy 90% of the uses of the proposed feature, while significantly reducing complexity. Now, the proposal argues that custom metadata has the advantage over conformance lookup; it is opt-in and, thus, doesn’t generate needless metadata. However, protocols intended for conformance lookup could refine a marker protocol or have an attribute, which would still result in a simpler feature.

2 Likes

The proposal includes this bullet in the Motivation section, which is meant to describe why this feature may be useful when attached to functions (emphasis mine):

  • Since tests are declared implicitly, there is no way for a user to provide additional details about an individual test or group of tests. It would be useful to have a way to indicate whether a test is enabled, its requirements, or other metadata, for example, so that the testing library could use this information to inform how it executes tests and offer more powerful features.

Going further with this example, you could imagine an individual test being defined as a function with a custom reflection metadata attribute, and then one could pass arguments to that attribute to customize things about the test, such as marking it disabled or adding extra information.

Using protocols as you describe has a downside that you need an instance of the type conforming to the relevant protocol to call its instance-level methods or properties. This can be problematic if the information is needed prior to when any of those instances are created—for example, to decide whether or how those instances should be created. And the arguments to one of these attributes on a type are meant to apply to the type itself, not to any particular instance of that type.

5 Likes

The proposal also has commentary on how property wrappers are currently used for storing metadata and the downsides of this approach:

Regarding the use of property wrappers to represent metadata on properties: We feel that property wrappers are not an ideal tool for reflection metadata because they require an instance of the backing property to be stored for each instance, even though the wrapper is constant per-declaration. Property wrappers that are only used for reflection metadata don’t need to introduce any access indirection of the wrapped value, either. The value itself can simply be stored inline in the type, rather than synthesizing computed properties.

Enabling custom reflection metadata on properties that are discovered via key-path can replace property wrappers that are used as metadata for a property declaration.

5 Likes

Thank you for bringing this up, I forgot to address it in my previous post.

Firstly, I don’t think the current naming scheme is that bad, and it allows users to opt out of tests by changing their methods’ names. The redundancy of just four letters in “test” is not deal breaker in my opinion, and since not every method in a test class actually performs a test, it is not always redundant. Again, I agree that it looks nicer to have test methods opt out (e.g. with the proposed attribute) instead of in (with the “test” prefix), but the complexity of the proposed solution doesn’t convince me.

As for the additional information, what would it be and what features would it enable exactly? I know this isn’t a proposal for a new testing facility, but I think the proposal’s motivation needs to be more concrete.

I’m pretty sure the API exposed in Echo allows users to capture a conforming type’s meta type. From there, a protocol that provides an initializer allows users to transform the meta type instance into a type instance. This same API could be exposed in a Reflection library. Please let me know if I misunderstood your point.

The stored-properties use case is another part of the proposal I had some trouble understanding. I think the text has some motivating examples spread throughout but the actual “Motivation” section mainly focuses on tests. Since this is a novel and quite broad feature, I think it’d be more effective to discuss specific cases.

Coming back to property wrappers, my understanding of the proposal is that property wrappers used to store metadata are inefficient. Are there any examples the authors are considering where the current inefficiency has significant impacts on code size or performance?

Also, I think property wrappers could work by carrying semantic information through the storage type. That is, a wrapper could simply store the wrapped value and expose it as inlined. This should offer direct access to the value of interest. However, the backing storage would also carry type information accessible through general-purpose reflection APIs. I’m not sure if this would work in practice, though, so I’d appreciate the perspectives of compiler engineers.

I have other feedback, but I agree with @filip-sakel and others who'd like more concrete examples of the definition and usage of the custom metadata. Surely there's something between "entire testing framework" and "bare example" that can be shown?

Sure thing, I can add a code example involving property wrapper "metadata" to the Motivation section!

Yes, but it's not just about performance. It's a serious expressivity limitation that you must have an instance in order to retrieve property wrapper metadata. I've seen use cases that will create dummy instances of types just to gather the metadata from stored property wrappers. I'll include this in the code example I'll add to the Motivation section.

EDIT: I remembered a very recent example posted on the forums over here:

This use case in Realm is using a very brittle approach that turns a key-path into an identifier used for database access. This use-case could instead use a custom metadata attribute that provides an identifier for a given key-path that is provided by the programmer, or we could have #function in a metadata attribute grab the name of the function it's attached to (which also works for properties today).

6 Likes

Fluent does this to retrieve fields path. By the way, can property wrappers work together with reflection metadata?

4 Likes
// @EditorCommandRecord is inferred
struct SelectWordCommand: EditorCommand { /* ... */ }

What argument does the inferred @EditorCommandRecord have here?
Do they have default argument?


Can the same @Metadata be attached for multiple times, like this?

@EditorCommandRecord(keyboardShortcut: "j", modifier: .command) 
@EditorCommandRecord(keyboardShortcut: "k", modifier: .command)
struct FooCommand {}

The concrete things Realm would use this for are:

  1. Automatic model class discovery. If the user does not supply a schema when opening a database, we currently call objc_copyClassList() to find all subclasses of RealmSwift.Object (and some other things) and generate a schema from them. objc_copyClassList() has several issues, but the big functional problem is that it eagerly loads all linked dylibs. This hurts app startup time, increases general app memory usage, and can make extensions overshoot their memory budget by itself.

    This proposal gives us a way to get automatic schema discovery without most of the drawbacks, and in a way that can probably be deployed as a non-breaking change to our users (the only thing I could see being a problem is if the Reflection module has a hard requirement on a deployment target instead of requiring @available guards for some reason).

  2. Advanced schema customization. For example, a property @Persisted var foo: Int creates an integer column named "foo" in the underlying database table. We'd like to support something like @Persisted(named: "bar") var foo: Int or @Named("bar") @Persisted var foo: Int to let the user specify that the underlying column should instead be named "bar". Doing this via a property wrapper requires storing the string "bar" on every instance of the object and not just the one instantiated for schema discovery, which is an unacceptable increase in memory usage.

    We currently support doing this by overriding a class method that returns a dictionary of name mappings, which is not a very good API.

I don't think we'd use this proposal for anything else by itself. The problem with using it to avoid instantiating an object, using Mirror to slurp the properties, and then using ivar_getOffset to work around the lack of keypaths on Mirror is that we would still need the @Persisted property wrapper. It does some wacky things for schema discovery that could go away, but it's also just the thing that turns property accesses into database reads. This means that we'd need property declarations to be something like @Discoverable @Persisted var foo: Int, i.e. we'd be making our API worse and requiring users to write more boilerplate just to make our implementation simpler.

It's possible that declaration macros fix this. We could perhaps have a single macro attribute that applies both required attributes to the property (or fully expands into getters and setters and skips the property wrapper entirely).


My opinion on the proposal is still roughly what it was for the pitch: I'm not wild about this specific design, but it is something we'd use if it existed and it'd solve both functional and API design issues we're currently facing.

3 Likes

I think it would be nice to figure out a easy way to mix properties wrapper and reflection metadata. I can't see how right now and it seems that without this the tradeoff may not be worth it.

1 Like