SE-0385: Custom Reflection Metadata

Would something like this be possible? The metadata would be applied to the property name

@propertyWrapper
struct Field<Value>: MetadataReflectable {
    
   @reflectionMetadata
    struct Metadata {
        
        let key: String
    
        ...
        
    }
    
    var wrappedValue: Value
    
    let metadata: Metadata
    
    init(wrappedValue: Value, key: String) {
        self.metadata = Metadata(key: key)
        self.wrappedValue = wrappedValue
    }
    
}

struct Model {
    
    @Field(key: "name") var name: String = "Lorem Ipsum"
    
}


Inferred type would have whatever arguments are passed to the protocol if there are any custom arguments.

No, currently proposal allows only one attribute of the same type per declaration.

1 Like

I'm sorry, but confused.
IIUC, the protocol EditorCommand requires conforming type to be @EditorCommandRecord with any arguments.

@EditorCommandRecord
protocol EditorCommand { /* ... */ }

However, here, @EditorCommandRecord is inferred even though there are no explicit @EditorCommandRecord(keyboardShortcut:modifier:) attribute. Then, how does the inferred @EditorCommandRecord work?

// @EditorCommandRecord is inferred
struct SelectWordCommand: EditorCommand { /* ... */ }

In other words; does @EditorCommandRecord struct FooCommand {} work?

In other words; does @EditorCommandRecord struct FooCommand {} work?

Yes, it does. Placing attribute on a protocol doesn't make it a requirement, it just means that all of the types that declare a conformance at their declaration (i.e. struct X: EditorCommand) would get the attribute inferred from the protocol as-if they have been declared as @EditorCommandRecord struct X: EditorCommand { ... })

1 Like

Clarification on this one - attributes places on protocols cannot have any custom arguments, we have that specified in the proposal as Reflection metadata attributes applied to protocols cannot have additional attribute arguments; attribute arguments must be explicitly written on the conforming type.

1 Like

If I recall your pitch feedback correctly, your stance was that the "discover all types that conform to a protocol" would be better as a general reflection facility that works for all types, rather than as opt-in functionality via attribute. (Please correct me if I misremembered!)

For the property metadata use case, there still needs to be a way to store the metadata separately from the enclosing instance itself that is discoverable at runtime through a key-path. This proposal provides exactly that, but I understand you're not fond of the design. Is there an alternative design for this functionality that you have in mind? A while back there was a pitch for "per-declaration storage for property wrappers" which may work for the Realm use case, but there are also downsides to that approach:

  • It's a much larger addition to the language than this proposal. I also think that future enhancements to property wrappers should leverage attached macros where possible, because macros will enable a lot of property-wrapper-like behavior that is not expressible with property wrappers today.
  • Shared property wrapper storage only works for stored properties with an attached property wrapper. It does not work for computed properties, nor does it work for plain stored properties that don't need access indirection.
  • Shared property wrapper storage doesn't provide custom reflection metadata for functions.
1 Like

I think passing arbitrary values, such as integers or strings, through generics would be a much better future direction than either the proposed solution or shared property wrappers. It would be a general solution that could also be used for multi dimensional arrays and would combat the problem of duplicating information for every type instance. This is an example of such a wrapper:

@propertyWrapper
struct Clamping<let range: ClosedRange<Int>> {
  var _storage: Int

  init(
    wrappedValue: Int, between range: ClosedRange<Int>
  ) where Self.range == range { … }

  var wrappedValue: Int {
   get { _storage }
   set {
     _storage = newValue.clamp(Self.range) // hypothetical method
   }
  }
}

I understand this feature would be even more complex than shared property wrappers. However, without a concrete justification of how duplicating data in instances is so inefficient, I’m not convinced we should invent a specialized feature for wrappers. Admittedly the proposal is broader than property wrappers, but some of the proposed features seem to target only the property-wrapper case, essentially creating that specialized feature.

Edit: I realized my feedback is getting quite negative. I agree that some of the problems outlined in the Motivation section need to be solved. Particularly, having XCTest-like discovery would be great! I think I’d be more inclined to agree with the proposal if it narrowed down on a specific use case and demonstrated how custom metadata is superior to other approaches. We followed this approach for features like opaque types where “some” was proven to be useful even with initially limited functionality, and later expanded it. As I made clear in my previous posts, I’m not convinced we need a whole new type to represent metadata. I think a much more straightforward approach would be to expand on the reflection infrastructure the proposal is based upon. For example, we could focus on selectively emitting metadata for protocols whose conformances can be looked up with a new attribute for protocols. This would follow the precedent of controlling metadata emission, set by the recently pitched Reflectable protocol.

I added this to Other Swift Flags and tried it on both the main and 5.8 development snapshots from Jan 27th to build:

@runtimeMetadata
struct Flag
{
	init<T>(attachedTo: T.Type) {}
}

This gives Runtime discoverable attributes are an experimental feature.
I tried runtimeDiscoverableAttrs and runtime-discoverable-attrs too.

What am I doing wrong?

With regards to the pitch. If this can be used to annotate fields to make them discoverable for creating a database schema, then I am very much in favour of it. It does look that way from the other comments. :)
I hope I get it to work to give more substantial feedback, if any.

Try -Xfrontend -enable-experimental-feature -Xfrontend RuntimeDiscoverableAttrs

3 Likes

I think this open pull request is necessary to fully experiment with the part of accessing the metadata. Is that right @xedin?

Right, I am building a toolchain, will post a link once it becomes available.

2 Likes

https://ci.swift.org/job/swift-PR-toolchain-macos/521/artifact/branch-main/swift-PR-63168-521-osx.tar.gz

3 Likes

Thank you xedin for building this toolchain.

I got it to work and played with it a bit.
I tried to get some code snippets to work for building a DB schema.

adhoc code:
import Foundation
import Reflection

@available(macOS 9999, *)
@runtimeMetadata
struct Flag
{
	var content: Int = 11
	init<T, V>(attachedTo path: KeyPath<T, V>, custom: Int)
	{
		self.content = custom
	}
}

@runtimeMetadata
struct PrimaryKey
{
	init<T, Result>(attachedTo: (T.Type) -> Result)
	{
	}
}

@available(macOS 9999, *)
struct Test
{
	let question: String = "to everything"

	@Flag(custom: 42)
	var answer: Int = 45

	@PrimaryKey
	static func computeStateless()
	{
		print("computeStateless")
	}
}

available(macOS 9999, *)
func reflect(_ type: Any.Type)
{
	let source = Type(type)

	if source.swiftType is Flag.Type
	{
		print("-- flagged --")
		dump(source)
	}

	print("--reflect \(type)--")
	print(source.swiftType)

	print("± fields ±")
	for field in source.fields
	{
		print(field.name)
		print(field.type)
		print(field.isVar)
		print(field.keyPath)
		print("-")
	}

	print("± cases ±")
	for a in source.cases
	{
		dump(a)
	}

	print("± generic ±")
	for arg in source.genericArguments
	{
		dump(arg)
	}

	print("± function ±")
//	for arg in source.functionParameters
//	{
//		dump(arg)
//	}

	print("± other: ±")
	print("")
}

in main:

if #available(macOS 9999, *)
{
	reflect(Test.self)

	print("-- flags --")
	for instance in Attribute.allInstances(of: Flag.self)
	{
		print(instance.content)
	}

	print("-- keys --")
	for instance in Attribute.allInstances(of: PrimaryKey.self)
	{
		print(instance)
	}
}

Maybe I haven't fully grasped yet how this is best used. I kinda expected to be able to iterate over the type with its fields and then for each field (or function or...) be able to extract the Flag instance.

I do like the new reflection module and this proposal. They will come in handy for e.g. a plugin system.
For a DB schema I don't think having a central registry is ideal. Different schema's are possible. A similar application would be to selectively save fields to disk (@SaveToDisk). Though perhaps my comment is too focused on a narrow use case, sorry about that.

I had to operate over all the instances to retrieve metadata of a specific type, which doesn't seem ideal. Will the Reflection library provide methods to do more fine-grained query of custom metadata? @Alejandro


@main
public struct HelloCustomReflectionMetadata {
    
    public static func main() {
        
        if #available(macOS 9999, *) {
            
            let allFieldsInStruct = fields(in: Struct.self)
            
            print(allFieldsInStruct.count) // 3
            
            let field = field(for: \Struct.f1)
            print(field!.key) // "field1"
        }
        
    }
    
}

@available(macOS 9999, *)
func fields<T>(in type: T.Type) -> [Field] {
    Attribute.allInstances(of: Field.self).filter {
        $0.keyPath is PartialKeyPath<T>
    }
}
 
@available(macOS 9999, *)
func field<T, U>(for keyPath: KeyPath<T, U>) -> Field? {
    Attribute.allInstances(of: Field.self).first {
        $0.keyPath == keyPath
    }
}

@runtimeMetadata
struct Field {
    let key: String
    let keyPath: AnyKeyPath
    init<T, U>(attachedTo keyPath: KeyPath<T, U>, _ key: String) {
        self.key = key
        self.keyPath = keyPath
    }
}

struct Struct {
    
    @Field("field1")
    let f1: Int
    
    @Field("field2")
    let f2: Int
    
    @Field("field3")
    let f3: Int
}

I think this is going to be covered by an extension of newly proposed Reflection module.

3 Likes

The cost of redundant metadata would significantly affect a system that I am building. We avoid using property wrappers for this specific reason.

2 Likes

Thank you for mentioning your use case, but could you elaborate more on your project and any alternatives you considered?

This is sublime and a gift of the gods.

Thank you for putting this proposal and prototype together, this is something I really wanted for my Swift bindings for Godot.

Being able to attach metadata to types and functions is great (I come from the land where metadata on properties and method arguments is common, so if anything, I hope that this directionally keeps growing, as opposed to our fellow that wants to limit it). In platforms like .NET, we have used this metadata for all sorts of interesting uses, like Web serving frameworks, to modeling the Apple APIs and describing their semantics, so this proposal gets a +100 from me.

One thing that was not clear to me is how I would retrieve the metadata for narrower cases. I might not want to trigger the initialization of every possible instance of my metadata across the program, sometimes I might just want to lookup at the instances attached to a specific type or function. Something like "Does this function have the @oneWay attribute on it?" or "Get me all the attributes attach to this particular function" or just "Get me all the methods that have this attribute on this type", which are narrower uses than what seems to be in the proposal.

10 Likes

I have a couple of pull requests open to amend the proposal but first I wanted to share them here for discussion:

Please take a look and let us know what you think!

2 Likes

+1, excited for this one.

I would also add, that I think @xedin's first PR about magic literals should definitely be accepted - unlike with Type declarations, I don't see other way of getting the names of the functions and properties.

However, I have one question. When should we run the Attribute .allInstances(of:) function? I assume, that we should re-run this function every time a .dylib/.so/.dll is loaded or unloaded. I can imagine some C __attribute__ that we could use, but I think, that this should be somehow part of the design - possibly with some API that allows us to iterate loaded modules and their related instances of metadata.