SE-0385: Custom Reflection Metadata

-1 from me.

I understand the motivation for test and plug-in types, so the custom metadata attributes make sense to me for types. However, the proposal doesn’t explicitly specify why functions should have attached metadata. In fact, many frameworks usually compartmentalize complex behavior in types instead of functions (such as SwiftUI Views being structs not functions); thus, it isn’t clear why functions would require custom metadata.

There is also insufficient motivation for adding arguments in the metadata attribute @Attribute(arg1: 1), such as for the editor-command metadata. The proposal isn’t convincing that the arguments-in-attribute approach is better than conforming to a protocol that requires these arguments as computed properties:

protocol MyAttributeRequirements {
  var arg1: Int { get }
}
@Attribute // Requires Self : MyAttributeRequirements 
struct MyStruct: MyAttributeRequirements {
  var arg1: Int { 1 }
}

Of course, the protocol approach is verbose, but it’s also a less complex language feature.

Without proper motivation for the aforementioned use cases, the alternative of simply expanding existing language features is a lot more appealing. Particularly, we could expose a reflection API that looks up all types conforming to a specific protocol. Looking up protocol conformances would satisfy 90% of the uses of the proposed feature, while significantly reducing complexity. Now, the proposal argues that custom metadata has the advantage over conformance lookup; it is opt-in and, thus, doesn’t generate needless metadata. However, protocols intended for conformance lookup could refine a marker protocol or have an attribute, which would still result in a simpler feature.

2 Likes

The proposal includes this bullet in the Motivation section, which is meant to describe why this feature may be useful when attached to functions (emphasis mine):

  • Since tests are declared implicitly, there is no way for a user to provide additional details about an individual test or group of tests. It would be useful to have a way to indicate whether a test is enabled, its requirements, or other metadata, for example, so that the testing library could use this information to inform how it executes tests and offer more powerful features.

Going further with this example, you could imagine an individual test being defined as a function with a custom reflection metadata attribute, and then one could pass arguments to that attribute to customize things about the test, such as marking it disabled or adding extra information.

Using protocols as you describe has a downside that you need an instance of the type conforming to the relevant protocol to call its instance-level methods or properties. This can be problematic if the information is needed prior to when any of those instances are created—for example, to decide whether or how those instances should be created. And the arguments to one of these attributes on a type are meant to apply to the type itself, not to any particular instance of that type.

5 Likes

The proposal also has commentary on how property wrappers are currently used for storing metadata and the downsides of this approach:

Regarding the use of property wrappers to represent metadata on properties: We feel that property wrappers are not an ideal tool for reflection metadata because they require an instance of the backing property to be stored for each instance, even though the wrapper is constant per-declaration. Property wrappers that are only used for reflection metadata don’t need to introduce any access indirection of the wrapped value, either. The value itself can simply be stored inline in the type, rather than synthesizing computed properties.

Enabling custom reflection metadata on properties that are discovered via key-path can replace property wrappers that are used as metadata for a property declaration.

5 Likes

Thank you for bringing this up, I forgot to address it in my previous post.

Firstly, I don’t think the current naming scheme is that bad, and it allows users to opt out of tests by changing their methods’ names. The redundancy of just four letters in “test” is not deal breaker in my opinion, and since not every method in a test class actually performs a test, it is not always redundant. Again, I agree that it looks nicer to have test methods opt out (e.g. with the proposed attribute) instead of in (with the “test” prefix), but the complexity of the proposed solution doesn’t convince me.

As for the additional information, what would it be and what features would it enable exactly? I know this isn’t a proposal for a new testing facility, but I think the proposal’s motivation needs to be more concrete.

I’m pretty sure the API exposed in Echo allows users to capture a conforming type’s meta type. From there, a protocol that provides an initializer allows users to transform the meta type instance into a type instance. This same API could be exposed in a Reflection library. Please let me know if I misunderstood your point.

The stored-properties use case is another part of the proposal I had some trouble understanding. I think the text has some motivating examples spread throughout but the actual “Motivation” section mainly focuses on tests. Since this is a novel and quite broad feature, I think it’d be more effective to discuss specific cases.

Coming back to property wrappers, my understanding of the proposal is that property wrappers used to store metadata are inefficient. Are there any examples the authors are considering where the current inefficiency has significant impacts on code size or performance?

Also, I think property wrappers could work by carrying semantic information through the storage type. That is, a wrapper could simply store the wrapped value and expose it as inlined. This should offer direct access to the value of interest. However, the backing storage would also carry type information accessible through general-purpose reflection APIs. I’m not sure if this would work in practice, though, so I’d appreciate the perspectives of compiler engineers.

I have other feedback, but I agree with @filip-sakel and others who'd like more concrete examples of the definition and usage of the custom metadata. Surely there's something between "entire testing framework" and "bare example" that can be shown?

Sure thing, I can add a code example involving property wrapper "metadata" to the Motivation section!

Yes, but it's not just about performance. It's a serious expressivity limitation that you must have an instance in order to retrieve property wrapper metadata. I've seen use cases that will create dummy instances of types just to gather the metadata from stored property wrappers. I'll include this in the code example I'll add to the Motivation section.

EDIT: I remembered a very recent example posted on the forums over here:

This use case in Realm is using a very brittle approach that turns a key-path into an identifier used for database access. This use-case could instead use a custom metadata attribute that provides an identifier for a given key-path that is provided by the programmer, or we could have #function in a metadata attribute grab the name of the function it's attached to (which also works for properties today).

6 Likes

Fluent does this to retrieve fields path. By the way, can property wrappers work together with reflection metadata?

4 Likes
// @EditorCommandRecord is inferred
struct SelectWordCommand: EditorCommand { /* ... */ }

What argument does the inferred @EditorCommandRecord have here?
Do they have default argument?


Can the same @Metadata be attached for multiple times, like this?

@EditorCommandRecord(keyboardShortcut: "j", modifier: .command) 
@EditorCommandRecord(keyboardShortcut: "k", modifier: .command)
struct FooCommand {}

The concrete things Realm would use this for are:

  1. Automatic model class discovery. If the user does not supply a schema when opening a database, we currently call objc_copyClassList() to find all subclasses of RealmSwift.Object (and some other things) and generate a schema from them. objc_copyClassList() has several issues, but the big functional problem is that it eagerly loads all linked dylibs. This hurts app startup time, increases general app memory usage, and can make extensions overshoot their memory budget by itself.

    This proposal gives us a way to get automatic schema discovery without most of the drawbacks, and in a way that can probably be deployed as a non-breaking change to our users (the only thing I could see being a problem is if the Reflection module has a hard requirement on a deployment target instead of requiring @available guards for some reason).

  2. Advanced schema customization. For example, a property @Persisted var foo: Int creates an integer column named "foo" in the underlying database table. We'd like to support something like @Persisted(named: "bar") var foo: Int or @Named("bar") @Persisted var foo: Int to let the user specify that the underlying column should instead be named "bar". Doing this via a property wrapper requires storing the string "bar" on every instance of the object and not just the one instantiated for schema discovery, which is an unacceptable increase in memory usage.

    We currently support doing this by overriding a class method that returns a dictionary of name mappings, which is not a very good API.

I don't think we'd use this proposal for anything else by itself. The problem with using it to avoid instantiating an object, using Mirror to slurp the properties, and then using ivar_getOffset to work around the lack of keypaths on Mirror is that we would still need the @Persisted property wrapper. It does some wacky things for schema discovery that could go away, but it's also just the thing that turns property accesses into database reads. This means that we'd need property declarations to be something like @Discoverable @Persisted var foo: Int, i.e. we'd be making our API worse and requiring users to write more boilerplate just to make our implementation simpler.

It's possible that declaration macros fix this. We could perhaps have a single macro attribute that applies both required attributes to the property (or fully expands into getters and setters and skips the property wrapper entirely).


My opinion on the proposal is still roughly what it was for the pitch: I'm not wild about this specific design, but it is something we'd use if it existed and it'd solve both functional and API design issues we're currently facing.

3 Likes

I think it would be nice to figure out a easy way to mix properties wrapper and reflection metadata. I can't see how right now and it seems that without this the tradeoff may not be worth it.

1 Like

Would something like this be possible? The metadata would be applied to the property name

@propertyWrapper
struct Field<Value>: MetadataReflectable {
    
   @reflectionMetadata
    struct Metadata {
        
        let key: String
    
        ...
        
    }
    
    var wrappedValue: Value
    
    let metadata: Metadata
    
    init(wrappedValue: Value, key: String) {
        self.metadata = Metadata(key: key)
        self.wrappedValue = wrappedValue
    }
    
}

struct Model {
    
    @Field(key: "name") var name: String = "Lorem Ipsum"
    
}


Inferred type would have whatever arguments are passed to the protocol if there are any custom arguments.

No, currently proposal allows only one attribute of the same type per declaration.

1 Like

I'm sorry, but confused.
IIUC, the protocol EditorCommand requires conforming type to be @EditorCommandRecord with any arguments.

@EditorCommandRecord
protocol EditorCommand { /* ... */ }

However, here, @EditorCommandRecord is inferred even though there are no explicit @EditorCommandRecord(keyboardShortcut:modifier:) attribute. Then, how does the inferred @EditorCommandRecord work?

// @EditorCommandRecord is inferred
struct SelectWordCommand: EditorCommand { /* ... */ }

In other words; does @EditorCommandRecord struct FooCommand {} work?

In other words; does @EditorCommandRecord struct FooCommand {} work?

Yes, it does. Placing attribute on a protocol doesn't make it a requirement, it just means that all of the types that declare a conformance at their declaration (i.e. struct X: EditorCommand) would get the attribute inferred from the protocol as-if they have been declared as @EditorCommandRecord struct X: EditorCommand { ... })

1 Like

Clarification on this one - attributes places on protocols cannot have any custom arguments, we have that specified in the proposal as Reflection metadata attributes applied to protocols cannot have additional attribute arguments; attribute arguments must be explicitly written on the conforming type.

1 Like

If I recall your pitch feedback correctly, your stance was that the "discover all types that conform to a protocol" would be better as a general reflection facility that works for all types, rather than as opt-in functionality via attribute. (Please correct me if I misremembered!)

For the property metadata use case, there still needs to be a way to store the metadata separately from the enclosing instance itself that is discoverable at runtime through a key-path. This proposal provides exactly that, but I understand you're not fond of the design. Is there an alternative design for this functionality that you have in mind? A while back there was a pitch for "per-declaration storage for property wrappers" which may work for the Realm use case, but there are also downsides to that approach:

  • It's a much larger addition to the language than this proposal. I also think that future enhancements to property wrappers should leverage attached macros where possible, because macros will enable a lot of property-wrapper-like behavior that is not expressible with property wrappers today.
  • Shared property wrapper storage only works for stored properties with an attached property wrapper. It does not work for computed properties, nor does it work for plain stored properties that don't need access indirection.
  • Shared property wrapper storage doesn't provide custom reflection metadata for functions.
1 Like

I think passing arbitrary values, such as integers or strings, through generics would be a much better future direction than either the proposed solution or shared property wrappers. It would be a general solution that could also be used for multi dimensional arrays and would combat the problem of duplicating information for every type instance. This is an example of such a wrapper:

@propertyWrapper
struct Clamping<let range: ClosedRange<Int>> {
  var _storage: Int

  init(
    wrappedValue: Int, between range: ClosedRange<Int>
  ) where Self.range == range { … }

  var wrappedValue: Int {
   get { _storage }
   set {
     _storage = newValue.clamp(Self.range) // hypothetical method
   }
  }
}

I understand this feature would be even more complex than shared property wrappers. However, without a concrete justification of how duplicating data in instances is so inefficient, I’m not convinced we should invent a specialized feature for wrappers. Admittedly the proposal is broader than property wrappers, but some of the proposed features seem to target only the property-wrapper case, essentially creating that specialized feature.

Edit: I realized my feedback is getting quite negative. I agree that some of the problems outlined in the Motivation section need to be solved. Particularly, having XCTest-like discovery would be great! I think I’d be more inclined to agree with the proposal if it narrowed down on a specific use case and demonstrated how custom metadata is superior to other approaches. We followed this approach for features like opaque types where “some” was proven to be useful even with initially limited functionality, and later expanded it. As I made clear in my previous posts, I’m not convinced we need a whole new type to represent metadata. I think a much more straightforward approach would be to expand on the reflection infrastructure the proposal is based upon. For example, we could focus on selectively emitting metadata for protocols whose conformances can be looked up with a new attribute for protocols. This would follow the precedent of controlling metadata emission, set by the recently pitched Reflectable protocol.

I added this to Other Swift Flags and tried it on both the main and 5.8 development snapshots from Jan 27th to build:

@runtimeMetadata
struct Flag
{
	init<T>(attachedTo: T.Type) {}
}

This gives Runtime discoverable attributes are an experimental feature.
I tried runtimeDiscoverableAttrs and runtime-discoverable-attrs too.

What am I doing wrong?

With regards to the pitch. If this can be used to annotate fields to make them discoverable for creating a database schema, then I am very much in favour of it. It does look that way from the other comments. :)
I hope I get it to work to give more substantial feedback, if any.

Try -Xfrontend -enable-experimental-feature -Xfrontend RuntimeDiscoverableAttrs

3 Likes

I think this open pull request is necessary to fully experiment with the part of accessing the metadata. Is that right @xedin?