SE-0385: Custom Reflection Metadata

Thank you for bringing this up, I forgot to address it in my previous post.

Firstly, I don’t think the current naming scheme is that bad, and it allows users to opt out of tests by changing their methods’ names. The redundancy of just four letters in “test” is not deal breaker in my opinion, and since not every method in a test class actually performs a test, it is not always redundant. Again, I agree that it looks nicer to have test methods opt out (e.g. with the proposed attribute) instead of in (with the “test” prefix), but the complexity of the proposed solution doesn’t convince me.

As for the additional information, what would it be and what features would it enable exactly? I know this isn’t a proposal for a new testing facility, but I think the proposal’s motivation needs to be more concrete.

I’m pretty sure the API exposed in Echo allows users to capture a conforming type’s meta type. From there, a protocol that provides an initializer allows users to transform the meta type instance into a type instance. This same API could be exposed in a Reflection library. Please let me know if I misunderstood your point.

The stored-properties use case is another part of the proposal I had some trouble understanding. I think the text has some motivating examples spread throughout but the actual “Motivation” section mainly focuses on tests. Since this is a novel and quite broad feature, I think it’d be more effective to discuss specific cases.

Coming back to property wrappers, my understanding of the proposal is that property wrappers used to store metadata are inefficient. Are there any examples the authors are considering where the current inefficiency has significant impacts on code size or performance?

Also, I think property wrappers could work by carrying semantic information through the storage type. That is, a wrapper could simply store the wrapped value and expose it as inlined. This should offer direct access to the value of interest. However, the backing storage would also carry type information accessible through general-purpose reflection APIs. I’m not sure if this would work in practice, though, so I’d appreciate the perspectives of compiler engineers.

I have other feedback, but I agree with @filip-sakel and others who'd like more concrete examples of the definition and usage of the custom metadata. Surely there's something between "entire testing framework" and "bare example" that can be shown?

Sure thing, I can add a code example involving property wrapper "metadata" to the Motivation section!

Yes, but it's not just about performance. It's a serious expressivity limitation that you must have an instance in order to retrieve property wrapper metadata. I've seen use cases that will create dummy instances of types just to gather the metadata from stored property wrappers. I'll include this in the code example I'll add to the Motivation section.

EDIT: I remembered a very recent example posted on the forums over here:

This use case in Realm is using a very brittle approach that turns a key-path into an identifier used for database access. This use-case could instead use a custom metadata attribute that provides an identifier for a given key-path that is provided by the programmer, or we could have #function in a metadata attribute grab the name of the function it's attached to (which also works for properties today).

6 Likes

Fluent does this to retrieve fields path. By the way, can property wrappers work together with reflection metadata?

4 Likes
// @EditorCommandRecord is inferred
struct SelectWordCommand: EditorCommand { /* ... */ }

What argument does the inferred @EditorCommandRecord have here?
Do they have default argument?


Can the same @Metadata be attached for multiple times, like this?

@EditorCommandRecord(keyboardShortcut: "j", modifier: .command) 
@EditorCommandRecord(keyboardShortcut: "k", modifier: .command)
struct FooCommand {}

The concrete things Realm would use this for are:

  1. Automatic model class discovery. If the user does not supply a schema when opening a database, we currently call objc_copyClassList() to find all subclasses of RealmSwift.Object (and some other things) and generate a schema from them. objc_copyClassList() has several issues, but the big functional problem is that it eagerly loads all linked dylibs. This hurts app startup time, increases general app memory usage, and can make extensions overshoot their memory budget by itself.

    This proposal gives us a way to get automatic schema discovery without most of the drawbacks, and in a way that can probably be deployed as a non-breaking change to our users (the only thing I could see being a problem is if the Reflection module has a hard requirement on a deployment target instead of requiring @available guards for some reason).

  2. Advanced schema customization. For example, a property @Persisted var foo: Int creates an integer column named "foo" in the underlying database table. We'd like to support something like @Persisted(named: "bar") var foo: Int or @Named("bar") @Persisted var foo: Int to let the user specify that the underlying column should instead be named "bar". Doing this via a property wrapper requires storing the string "bar" on every instance of the object and not just the one instantiated for schema discovery, which is an unacceptable increase in memory usage.

    We currently support doing this by overriding a class method that returns a dictionary of name mappings, which is not a very good API.

I don't think we'd use this proposal for anything else by itself. The problem with using it to avoid instantiating an object, using Mirror to slurp the properties, and then using ivar_getOffset to work around the lack of keypaths on Mirror is that we would still need the @Persisted property wrapper. It does some wacky things for schema discovery that could go away, but it's also just the thing that turns property accesses into database reads. This means that we'd need property declarations to be something like @Discoverable @Persisted var foo: Int, i.e. we'd be making our API worse and requiring users to write more boilerplate just to make our implementation simpler.

It's possible that declaration macros fix this. We could perhaps have a single macro attribute that applies both required attributes to the property (or fully expands into getters and setters and skips the property wrapper entirely).


My opinion on the proposal is still roughly what it was for the pitch: I'm not wild about this specific design, but it is something we'd use if it existed and it'd solve both functional and API design issues we're currently facing.

3 Likes

I think it would be nice to figure out a easy way to mix properties wrapper and reflection metadata. I can't see how right now and it seems that without this the tradeoff may not be worth it.

1 Like

Would something like this be possible? The metadata would be applied to the property name

@propertyWrapper
struct Field<Value>: MetadataReflectable {
    
   @reflectionMetadata
    struct Metadata {
        
        let key: String
    
        ...
        
    }
    
    var wrappedValue: Value
    
    let metadata: Metadata
    
    init(wrappedValue: Value, key: String) {
        self.metadata = Metadata(key: key)
        self.wrappedValue = wrappedValue
    }
    
}

struct Model {
    
    @Field(key: "name") var name: String = "Lorem Ipsum"
    
}


Inferred type would have whatever arguments are passed to the protocol if there are any custom arguments.

No, currently proposal allows only one attribute of the same type per declaration.

1 Like

I'm sorry, but confused.
IIUC, the protocol EditorCommand requires conforming type to be @EditorCommandRecord with any arguments.

@EditorCommandRecord
protocol EditorCommand { /* ... */ }

However, here, @EditorCommandRecord is inferred even though there are no explicit @EditorCommandRecord(keyboardShortcut:modifier:) attribute. Then, how does the inferred @EditorCommandRecord work?

// @EditorCommandRecord is inferred
struct SelectWordCommand: EditorCommand { /* ... */ }

In other words; does @EditorCommandRecord struct FooCommand {} work?

In other words; does @EditorCommandRecord struct FooCommand {} work?

Yes, it does. Placing attribute on a protocol doesn't make it a requirement, it just means that all of the types that declare a conformance at their declaration (i.e. struct X: EditorCommand) would get the attribute inferred from the protocol as-if they have been declared as @EditorCommandRecord struct X: EditorCommand { ... })

1 Like

Clarification on this one - attributes places on protocols cannot have any custom arguments, we have that specified in the proposal as Reflection metadata attributes applied to protocols cannot have additional attribute arguments; attribute arguments must be explicitly written on the conforming type.

1 Like

If I recall your pitch feedback correctly, your stance was that the "discover all types that conform to a protocol" would be better as a general reflection facility that works for all types, rather than as opt-in functionality via attribute. (Please correct me if I misremembered!)

For the property metadata use case, there still needs to be a way to store the metadata separately from the enclosing instance itself that is discoverable at runtime through a key-path. This proposal provides exactly that, but I understand you're not fond of the design. Is there an alternative design for this functionality that you have in mind? A while back there was a pitch for "per-declaration storage for property wrappers" which may work for the Realm use case, but there are also downsides to that approach:

  • It's a much larger addition to the language than this proposal. I also think that future enhancements to property wrappers should leverage attached macros where possible, because macros will enable a lot of property-wrapper-like behavior that is not expressible with property wrappers today.
  • Shared property wrapper storage only works for stored properties with an attached property wrapper. It does not work for computed properties, nor does it work for plain stored properties that don't need access indirection.
  • Shared property wrapper storage doesn't provide custom reflection metadata for functions.
1 Like

I think passing arbitrary values, such as integers or strings, through generics would be a much better future direction than either the proposed solution or shared property wrappers. It would be a general solution that could also be used for multi dimensional arrays and would combat the problem of duplicating information for every type instance. This is an example of such a wrapper:

@propertyWrapper
struct Clamping<let range: ClosedRange<Int>> {
  var _storage: Int

  init(
    wrappedValue: Int, between range: ClosedRange<Int>
  ) where Self.range == range { … }

  var wrappedValue: Int {
   get { _storage }
   set {
     _storage = newValue.clamp(Self.range) // hypothetical method
   }
  }
}

I understand this feature would be even more complex than shared property wrappers. However, without a concrete justification of how duplicating data in instances is so inefficient, I’m not convinced we should invent a specialized feature for wrappers. Admittedly the proposal is broader than property wrappers, but some of the proposed features seem to target only the property-wrapper case, essentially creating that specialized feature.

Edit: I realized my feedback is getting quite negative. I agree that some of the problems outlined in the Motivation section need to be solved. Particularly, having XCTest-like discovery would be great! I think I’d be more inclined to agree with the proposal if it narrowed down on a specific use case and demonstrated how custom metadata is superior to other approaches. We followed this approach for features like opaque types where “some” was proven to be useful even with initially limited functionality, and later expanded it. As I made clear in my previous posts, I’m not convinced we need a whole new type to represent metadata. I think a much more straightforward approach would be to expand on the reflection infrastructure the proposal is based upon. For example, we could focus on selectively emitting metadata for protocols whose conformances can be looked up with a new attribute for protocols. This would follow the precedent of controlling metadata emission, set by the recently pitched Reflectable protocol.

I added this to Other Swift Flags and tried it on both the main and 5.8 development snapshots from Jan 27th to build:

@runtimeMetadata
struct Flag
{
	init<T>(attachedTo: T.Type) {}
}

This gives Runtime discoverable attributes are an experimental feature.
I tried runtimeDiscoverableAttrs and runtime-discoverable-attrs too.

What am I doing wrong?

With regards to the pitch. If this can be used to annotate fields to make them discoverable for creating a database schema, then I am very much in favour of it. It does look that way from the other comments. :)
I hope I get it to work to give more substantial feedback, if any.

Try -Xfrontend -enable-experimental-feature -Xfrontend RuntimeDiscoverableAttrs

3 Likes

I think this open pull request is necessary to fully experiment with the part of accessing the metadata. Is that right @xedin?

Right, I am building a toolchain, will post a link once it becomes available.

2 Likes

https://ci.swift.org/job/swift-PR-toolchain-macos/521/artifact/branch-main/swift-PR-63168-521-osx.tar.gz

3 Likes

Thank you xedin for building this toolchain.

I got it to work and played with it a bit.
I tried to get some code snippets to work for building a DB schema.

adhoc code:
import Foundation
import Reflection

@available(macOS 9999, *)
@runtimeMetadata
struct Flag
{
	var content: Int = 11
	init<T, V>(attachedTo path: KeyPath<T, V>, custom: Int)
	{
		self.content = custom
	}
}

@runtimeMetadata
struct PrimaryKey
{
	init<T, Result>(attachedTo: (T.Type) -> Result)
	{
	}
}

@available(macOS 9999, *)
struct Test
{
	let question: String = "to everything"

	@Flag(custom: 42)
	var answer: Int = 45

	@PrimaryKey
	static func computeStateless()
	{
		print("computeStateless")
	}
}

available(macOS 9999, *)
func reflect(_ type: Any.Type)
{
	let source = Type(type)

	if source.swiftType is Flag.Type
	{
		print("-- flagged --")
		dump(source)
	}

	print("--reflect \(type)--")
	print(source.swiftType)

	print("± fields ±")
	for field in source.fields
	{
		print(field.name)
		print(field.type)
		print(field.isVar)
		print(field.keyPath)
		print("-")
	}

	print("± cases ±")
	for a in source.cases
	{
		dump(a)
	}

	print("± generic ±")
	for arg in source.genericArguments
	{
		dump(arg)
	}

	print("± function ±")
//	for arg in source.functionParameters
//	{
//		dump(arg)
//	}

	print("± other: ±")
	print("")
}

in main:

if #available(macOS 9999, *)
{
	reflect(Test.self)

	print("-- flags --")
	for instance in Attribute.allInstances(of: Flag.self)
	{
		print(instance.content)
	}

	print("-- keys --")
	for instance in Attribute.allInstances(of: PrimaryKey.self)
	{
		print(instance)
	}
}

Maybe I haven't fully grasped yet how this is best used. I kinda expected to be able to iterate over the type with its fields and then for each field (or function or...) be able to extract the Flag instance.

I do like the new reflection module and this proposal. They will come in handy for e.g. a plugin system.
For a DB schema I don't think having a central registry is ideal. Different schema's are possible. A similar application would be to selectively save fields to disk (@SaveToDisk). Though perhaps my comment is too focused on a narrow use case, sorry about that.