[Pitch] Custom Metadata Attributes

[Joe_Groff]

Maybe that's not the problem this proposal is immediately trying to solve, but yeah, it seems to me like when we introduce a compile-time evaluation model, that we could have a query function that's compiler-evaluable and can produce a collection of metadata values from annotated declarations at compile time. We could probably do similar things with a lot of our reflection APIs too.

[John_McCall]

Yeah, we could do that here. Of course, preserving the information at runtime doesn't actually stop us from also making it available statically, so I think what you're arguing is that it could also be useful to have that kind of API available for static annotations, under the constraint that it's module-local. And that maybe there are use cases which would be totally satisfied by that capability and don't need dynamic discovery.

[ktoso (Konrad 'ktoso' Malawski 🐟🏴‍☠️)]

@John_McCall

They’d currently only be useful for external source tools, simply because there’s nothing in the language which can ask arbitrary static questions about declarations. That’s a very likely future direction for procedural macros, though.

Yeah, that's right. There's not really a way to query for such things at compile time yet (there is for source generators, using swift syntax one could say tho). With with the raise of macros and constant evaluation there might soon be, and since we're introducing new ways for custom attributes, it kind of hit me as similar to the Java annotations story, where the two are not completely different, but just vary in "scope" they're visible in.

@John_McCall

Maybe this confusion is coming from the lack of “runtime” in the thread title.

Yeah that's probably the case tbh

@Joe_Groff

Maybe that's not the problem this proposal is immediately trying to solve, but yeah, it seems to me like when we introduce a compile-time evaluation model, that we could have a query function that's compiler-evaluable and can produce a collection of metadata values from annotated declarations at compile time. We could probably do similar things with a lot of our reflection APIs too.

@John_McCall

[...] I think what you're arguing is that it could also be useful to have that kind of API available for static annotations, under the constraint that it's module-local. And that maybe there are use cases which would be totally satisfied by that capability and don't need dynamic discovery.

Right, that's what I was getting at. For example some lambda / server / serverless functions framework could prefer to discover "the functions (or actors!) I should expose" at compile time rather than via runtime reflection. Though it's hard too judge how very or little benefit this would give, but in JVM land those runtime "scan for impls" frameworks (Spring comes to mind, which used to have a painful "scanning for java beans" startup period before it could actually start working) always take quite a hit on startup time and then more indirections... Perhaps this would not be true for us, but thought it was worth considering.

Those mechanisms are definitely outside of the scope of this proposal... but just a thought to see if those "compile time only" could be part of the same way we declare such attributes, rather than doing something separate again once we get to it

[AliSoftware (Olivier Halligon)]

I am quite confused as to why the metadata generation itself has to be executed only at runtime.

Couldn't the custom attributes attached to declarations be resolved at compile time, and the query evaluation would be the only runtime feature? Which, to me, boils down mostly to: getting better/richer reflection APIs than what we currently have with Mirror.

---

For example (pseudo code, not at all deeply thought out, but just to give the gist):

// new metadata keywords to tell the compiler of this metadata storage's existence
metadata Test: () -> Void {
  name: String
}

class MyTest {
  // attribute Metadata.Test is generated *at compile time* as metadata attached the function.
  // the compiler knows Test is valid custom metadata that is allowed to be attached to ()->Void types
  // The compiler can statically store the metadata in the binary alongside the generated code
  @Test(name: "Foo")
  func sayHello() {
  }
}

func listTests {
  // Runtime call to some new reflection API that swift would provide
  let tests: [Metadata.Test] = NewReflectionAPI.listAll(Metadata.Test)
  for test in tests {
    print(test.$name) // access metadata
    let fn: () -> Void = test.instance
    fn() // execute the annotated method
  }
}

Again, this is just pseudo code.
But the idea feels closer to existing solutions we already have (and which works and solve the problems mentioned in Motivation section) when using code generation tools like Sourcery.

I have to admit I haven't caught up with all other pending swift evolution proposals, maybe the new macro system might even be able to already cover that?

In any case, the metadata would be generated at compile time; and maybe we could even imagine the lists of all annotated objects be also generated at compile time (would break if using dynamically loaded code though, so maybe not) instead of having to use some reflection API executed at runtime.

Wouldn't that still solve the use cases mentioned in the motivation section?

[curt (Curt Clifton)]

@Alejandro

@allevato

Then, if I get fresh data every time getAllInstances is called, let's say the first call returns [A, B] and then a dylib is loaded later that adds C. Presumably a subsequent call to getAllInstances would return [A, B, C]. Do I need to do my own bookkeeping so that I only process C the next time around?

Yes, the API would return [A, B, C] and if you only wanted to consume C you'd need to do bookkeeping to figure what you've already seen.

That’s helpful. Thanks.

It seems like we’d also need a notification mechanism to know that the API will return a different result.

To wit, suppose I have two modules, one that I control and one that I don’t control but have empowered to load plug-ins. If the other module loads a new plug-in, how do I know to run my introspection/registration code again?

[filip-sakel]

I think that through compiler evaluation we can unite a lot of features that are currently discussed separately. Both upthread and in the macro threads, forms of static introspection/reflection were discussed. Besides the static guarantee and optimization, I don’t see how that’s different from metadata-based runtime reflections.

This feature also doesn’t seem that motivating. Marking types with metadata attributes is very similar to protocol conformances, so I don’t see why we don’t introduce a function to fetch all types conforming to a protocol (like what Echo offers). Now, local declarations would have no metadata mechanism, but that’s arguably okay. As previously mentioned, property wrappers are very similar to metadata markers, and general reflection could be used to look up properties. These properties could then be filtered for the property-wrapping storage type. This still leaves functions, which would no way to be marked; for functions either the name could be looked up (again through reflection), or we could introduce the much requested function wrappers. I know the current test method naming was specifically discussed in the proposal, but I haven’t personally faced any problems with the “test”-prefixed methods and didn’t find the pitch’s arguments convincing.

All in all, I think we should focus our efforts on a comprehensive reflection system. This opens the door to declarative metaprogramming where the compiler can make smart decisions. For example, the user can declare they’re looking for types conforming to a test protocol. Then it’s up to the compiler whether to constant fold and potentially remove the type’s metadata, or to explicitly use runtime reflections (perhaps for better code size).

[tgoyne (Thomas Goyne)]

After pondering this proposal for a while I think that while we would use things from it, I'm overall not a fan. I think it feels too much like a stopgap that could be made unnecessary by fleshing out other features for something which introduces a major new concept to the language.

We do really need a Swift-native mechanism for runtime type enumeration, but I think the "obvious" form for that is something which returns all types which conform to a given protocol, rather than introducing a new way to mark types. This could maybe require that the protocol be marked with an attribute to opt-in to generating the required metadata to do this.

The two main things we need from this for properties are:

1. A way to get a keypath for each property on an object (via an actual public API). This presumably would be part of any improved reflection API, but it also seems like it'd be easy to just add this to Mirror (albeit it'd have to be in the form of an AnyKeyPath there).
2. A way to have strings associated with a property which don't have to be stored on every instance of the type. The shared storage idea would have been perfect for us.

For the test method discovery use-case, property wrappers which apply to methods (i.e. function decorators?) would be a very powerful feature which would address that along with a reflection feature to get a list of methods. That does admittedly also seem like a very large feature to add.

[filip-sakel]

@tgoyne

This presumably would be part of any improved reflection API,

This was discussed here.

[David_Sweeris]

@hborla

We could also choose to flatten the array in the runtime query so it just produces [T]. The downside is you won't know whether there were any unavailable metadata attributes at runtime, but maybe that's okay.

We could also spell it allAvailableInstances and deal with unavailable instances somewhere else.

[mfilonen2]

I think that metadata attributes feature should be derived from macro system that is currently under construction, as it is in Rust, and not as a runtime feature as Swift generally prefers compile-time over runtime checks.

[Nickolas_Pohilets (Nickolas Pokhylets)]

One of the ways to attach to a generic type, would be to wrap type constructor into a function:

@runtimeMetadata
struct Flag {
    var archetype: (Any.Type) -> Any.Type
    init(attachedTo archetype: @escaping (Any.Type) -> Any.Type) {
        self.archetype = archetype
    }
}

@Flag
struct Container<A> { }

let allFlags = Attribute.getAllInstances(of: Flag.self)
let types = allFlags.compactMap { $0?.archetype(String.self) }
print(types) // [ ..., Container<String>, ... ]

[bbrk24]

I don't think this is viable. At least, I don't like the arbitrary choice that has to be be made for types like

@Flag
struct MyType<T: AsyncSequence> { ... }

This is the same problem I brought up in the Reflection thread @filip-sakel linked above:

@bbrk24

I was thinking of [Void: Void], but I guess there's not always going to be a good equivalent -- what about AsyncLineSequence for example? Would I have to say something like AsyncLineSequence<TaskGroup<UInt8>>? There is no AnyAsyncSequence<T> type, and the AsyncSequence protocol doesn't currently support the generic syntax.

Further, it may not even always be possible to fulfill the archetype:

protocol NotImplemented {}

@Flag
struct CantUseMe<T: NotImplemented> {}

[Nickolas_Pohilets (Nickolas Pokhylets)]

I think it should be possible to use meta types constrainted to protocols in the signature of the init(attachedTo:).

struct Flag {
    init(attachedTo archetype: @escaping (AsyncSequence.Type) -> MyProtocol.Type)
}

@Flag
struct MyType<T: AsyncSequence>: MyProtocol { ... }

It can be challenging through to express everything that can go into where clause.

I think with generalized existentials, it should be possible to capture where clause by combining all generic arguments into a dummy tuple type:

@Flag
struct Zipper<A: Sequence, B: Sequence> where A.Element == B.Element {
    typealias Element = A.Element
}

struct Flag {
    init(attachedTo archetype: @escaping (any <A: Sequence, B: Sequence> (A, B).Type where A.Element == B.Element) -> Sequence.Type)
}

But even this does not allow to capture the fact that Element of the resulting sequence type is the same as element of A and B.