[Pitch] Custom Metadata Attributes

Of course they're not fleeing, the vast majority of them are just users and so have no influence over how those features are implemented. The alternative to not using attributes for Unity is... not using Unity (or at least not using it in a supported manner). But those runtime attributes are a huge drain on Unity's runtime performance, and like all attributes, lead to poor developer experience.

I don't see how this would change. Similar to @davedelong's example, you would just express the key difference with a different attribute. Your example is also pretty poor given Codable has many long standing UX issues that could just be fixed rather than needing a whole runtime system to make usable. It seems like your problem could be solved by improvements to Codable, improvements to property wrappers, or another compile time feature. It doesn't seem like this problem needs runtime involvement at all.

This doesn't really make sense to me. If you insist on having all of the metadata exposed, which is not a given, you can do so in a structured way at compile time. There's nothing requiring compile time constructs to be as awkward as Codable.

And in fact, composition issues can be mitigated by moving metadata around, since it then becomes possible to define away the sharp edges of the composition in the first place. Besides which, I was asking how the pitch's actual feature composes with other actual Swift features, which is important to define, so it wasn't a general question with a general answer. There is also the question of how these things interact at runtime, but that's a problem for attribute implementors, as there's little a general runtime system can do to limit those issues.

Would there be a way to see the backing storage? Is visibility impacted by the order of attributes? It seems like people would want to change behavior based on whether a property is wrapped by another type.

2 Likes

It does, but not to the same extent really. I can't decide to make up a:

/// A lot of docs here
func oldImpl() {}
@sameDocsAs(oldImpl) func newImpl() {} 

// or a compile-time serialization system, that needs user assistance:
// these can be kind of done with a property wrapper but feels off
// @Tag(1) var field: String
// @Tag(2) var age: Int

in today's Swift. The attribute here being just an example, but I can imagine other cases - there's examples in the Java ecosystem we can look at if necessary (typical examples are things like OpenAPI/Swagger source generators). There are forms of custom attributes in Swift today, but they all have some meaning to Swift itself. One notable custom attribute type for example is global actors, but they carry specific meaning to the typechecker and runtime.

The attributes pitched here seem to be more usable by library/tool developers though and be it runtime, or compile time, don't really matter to Swift in any way. Thus, allowing the use in various other cases where today's "custom attributes" that exist in Swift don't facilitate.

I'm not aware of a way to make arbitrary custom attributes by library developers in Swift today, unless I'm missing something. There's type wrappers, property wrappers, global actors, but they all have specific meanings to Swift really, not just as a way to associate some meta-information with a field/type/method. Sendable is also an @Attribute but we can't declare such a thing ourselves as library authors, right? :thinking:

That said, for myself and server and library use cases I'm thinking about, it's not a big deal; but I'm surprised that while "opening up attributes to anyone" we didn't think about ways to not store runtime metadata while at it -- using the same mechanism, as Java does, there's no separate different way to declare "runtime" vs "class" vs "source" retention, it's just a custom annotation, with a scope defined.

It might also be worth checking what for source retention is being used in the real world; A quick github search reveals a lot of use, but I've not dug very deep into them. I can do so if that'd be helpful.


On that same thread... I wonder, IF, we had such source retention, couldn't @available and friends eventually become "normal" attributes, rather their own weird custom thing...? But perhaps that ship has sailed already, and they're way too custom with their own little grammar even.

Consolidating these under the same rules as normal swift would be nice, but perhaps impossible by now :thinking:


The re-iterate though; I don't think this is a show stopper nor my primary question here; just something that hit me as being not quite there, when comparing our many ad-hoc compile time attributes plus these runtime attributes with the Java annotations which of handle both cases, with the retention policy instead.

3 Likes

Currently what's implemented is the type of the instance that this is on. I.e.

struct A {
  @CustomRuntimeAttribute
  var x: String
}

In this case I can actually provide you a value of (String.self, CustomRuntimeAttribute?), so the API could be edited to look like [(Any.Type, T?)] (or something wrapped in a struct with a proper name).

Yes, querying in the other direction is something a lot of folks have been asking for and I imagine we'll want to alter the implementation a bit to allow for efficient querying the other way. I would love if this interacted with the proposed API in the reflection pitch to allow for say Type.attributes or Field.attributes to get all the attributes on a specific type or a specific field.

These attribute initializers are only called when you request them via the getAllInstances API. Also, only the specific attribute's initializers are called. For example, if you have 2 custom runtime attributes @A and @B and you ask for @B 's instances, we'll only initialize those attributes whose type is B and not any of the A ones.

The getAllInstances method is provided for you via the proposed Reflection module ([Pitch] Reflection). You are expected to implement your own attachedTo: inits to constrain what type of declarations your attribute can be applied to.

So we do both. On platforms like Linux and Windows, we need to scan each section every time for every attribute. On platforms like Darwin, we can scan each section once, cache the results, and for new images loaded at runtime we'll be notified of new sections and only have to scan those and cache the results.

Yes, the API would return [A, B, C] and if you only wanted to consume C you'd need to do bookkeeping to figure what you've already seen.

6 Likes

I guess this answers my question

You folks know way more than me but are we sure it wouldn't be nice to design this metadata attributes with some plan to be able to access them at compile time with some comptime system? From the libraries I know in java that used pervasive annotations they switched to use them at compile time, so feels a bit backwards that Swift is not thinking about that when introducing custom attributes.

I like @ktoso overall thoughts on this.

We can certainly add static metadata attributes. They’d currently only be useful for external source tools, simply because there’s nothing in the language which can ask arbitrary static questions about declarations. That’s a very likely future direction for procedural macros, though.

Maybe this confusion is coming from the lack of “runtime” in the thread title.

8 Likes

As I mentioned upthread, there does seem to be some degree of functionality here that could be consumed at compile time; a module ought to be able to ask itself for the full set of metadata records for its own declarations and get an answer at compile time.

6 Likes

How would it get that answer at compile time, though? Just an assurance that we can constant-fold some particular function call?

2 Likes

Maybe that's not the problem this proposal is immediately trying to solve, but yeah, it seems to me like when we introduce a compile-time evaluation model, that we could have a query function that's compiler-evaluable and can produce a collection of metadata values from annotated declarations at compile time. We could probably do similar things with a lot of our reflection APIs too.

5 Likes

Yeah, we could do that here. Of course, preserving the information at runtime doesn't actually stop us from also making it available statically, so I think what you're arguing is that it could also be useful to have that kind of API available for static annotations, under the constraint that it's module-local. And that maybe there are use cases which would be totally satisfied by that capability and don't need dynamic discovery.

11 Likes

Yeah, that's right. There's not really a way to query for such things at compile time yet (there is for source generators, using swift syntax one could say tho). With with the raise of macros and constant evaluation there might soon be, and since we're introducing new ways for custom attributes, it kind of hit me as similar to the Java annotations story, where the two are not completely different, but just vary in "scope" they're visible in.

Yeah that's probably the case tbh :slight_smile:

Right, that's what I was getting at. For example some lambda / server / serverless functions framework could prefer to discover "the functions (or actors!) I should expose" at compile time rather than via runtime reflection. Though it's hard too judge how very or little benefit this would give, but in JVM land those runtime "scan for impls" frameworks (Spring comes to mind, which used to have a painful "scanning for java beans" startup period before it could actually start working) always take quite a hit on startup time and then more indirections... Perhaps this would not be true for us, but thought it was worth considering.

Those mechanisms are definitely outside of the scope of this proposal... but just a thought to see if those "compile time only" could be part of the same way we declare such attributes, rather than doing something separate again once we get to it :slight_smile:

7 Likes

I am quite confused as to why the metadata generation itself has to be executed only at runtime.

Couldn't the custom attributes attached to declarations be resolved at compile time, and the query evaluation would be the only runtime feature? Which, to me, boils down mostly to: getting better/richer reflection APIs than what we currently have with Mirror.


For example (pseudo code, not at all deeply thought out, but just to give the gist):

// new metadata keywords to tell the compiler of this metadata storage's existence
metadata Test: () -> Void {
  name: String
}

class MyTest {
  // attribute Metadata.Test is generated *at compile time* as metadata attached the function.
  // the compiler knows Test is valid custom metadata that is allowed to be attached to ()->Void types
  // The compiler can statically store the metadata in the binary alongside the generated code
  @Test(name: "Foo")
  func sayHello() {
  }
}

func listTests {
  // Runtime call to some new reflection API that swift would provide
  let tests: [Metadata.Test] = NewReflectionAPI.listAll(Metadata.Test)
  for test in tests {
    print(test.$name) // access metadata
    let fn: () -> Void = test.instance
    fn() // execute the annotated method
  }
}

Again, this is just pseudo code.
But the idea feels closer to existing solutions we already have (and which works and solve the problems mentioned in Motivation section) when using code generation tools like Sourcery.

I have to admit I haven't caught up with all other pending swift evolution proposals, maybe the new macro system might even be able to already cover that?

In any case, the metadata would be generated at compile time; and maybe we could even imagine the lists of all annotated objects be also generated at compile time (would break if using dynamically loaded code though, so maybe not) instead of having to use some reflection API executed at runtime.

Wouldn't that still solve the use cases mentioned in the motivation section?

2 Likes

That’s helpful. Thanks.

It seems like we’d also need a notification mechanism to know that the API will return a different result.

To wit, suppose I have two modules, one that I control and one that I don’t control but have empowered to load plug-ins. If the other module loads a new plug-in, how do I know to run my introspection/registration code again?

I think that through compiler evaluation we can unite a lot of features that are currently discussed separately. Both upthread and in the macro threads, forms of static introspection/reflection were discussed. Besides the static guarantee and optimization, I don’t see how that’s different from metadata-based runtime reflections.

This feature also doesn’t seem that motivating. Marking types with metadata attributes is very similar to protocol conformances, so I don’t see why we don’t introduce a function to fetch all types conforming to a protocol (like what Echo offers). Now, local declarations would have no metadata mechanism, but that’s arguably okay. As previously mentioned, property wrappers are very similar to metadata markers, and general reflection could be used to look up properties. These properties could then be filtered for the property-wrapping storage type. This still leaves functions, which would no way to be marked; for functions either the name could be looked up (again through reflection), or we could introduce the much requested function wrappers. I know the current test method naming was specifically discussed in the proposal, but I haven’t personally faced any problems with the “test”-prefixed methods and didn’t find the pitch’s arguments convincing.

All in all, I think we should focus our efforts on a comprehensive reflection system. This opens the door to declarative metaprogramming where the compiler can make smart decisions. For example, the user can declare they’re looking for types conforming to a test protocol. Then it’s up to the compiler whether to constant fold and potentially remove the type’s metadata, or to explicitly use runtime reflections (perhaps for better code size).

2 Likes

After pondering this proposal for a while I think that while we would use things from it, I'm overall not a fan. I think it feels too much like a stopgap that could be made unnecessary by fleshing out other features for something which introduces a major new concept to the language.

We do really need a Swift-native mechanism for runtime type enumeration, but I think the "obvious" form for that is something which returns all types which conform to a given protocol, rather than introducing a new way to mark types. This could maybe require that the protocol be marked with an attribute to opt-in to generating the required metadata to do this.

The two main things we need from this for properties are:

  1. A way to get a keypath for each property on an object (via an actual public API). This presumably would be part of any improved reflection API, but it also seems like it'd be easy to just add this to Mirror (albeit it'd have to be in the form of an AnyKeyPath there).
  2. A way to have strings associated with a property which don't have to be stored on every instance of the type. The shared storage idea would have been perfect for us.

For the test method discovery use-case, property wrappers which apply to methods (i.e. function decorators?) would be a very powerful feature which would address that along with a reflection feature to get a list of methods. That does admittedly also seem like a very large feature to add.

5 Likes

This was discussed here.

1 Like

We could also spell it allAvailableInstances and deal with unavailable instances somewhere else.

I think that metadata attributes feature should be derived from macro system that is currently under construction, as it is in Rust, and not as a runtime feature as Swift generally prefers compile-time over runtime checks.

One of the ways to attach to a generic type, would be to wrap type constructor into a function:

@runtimeMetadata
struct Flag {
    var archetype: (Any.Type) -> Any.Type
    init(attachedTo archetype: @escaping (Any.Type) -> Any.Type) {
        self.archetype = archetype
    }
}

@Flag
struct Container<A> { }

let allFlags = Attribute.getAllInstances(of: Flag.self)
let types = allFlags.compactMap { $0?.archetype(String.self) }
print(types) // [ ..., Container<String>, ... ]

I don't think this is viable. At least, I don't like the arbitrary choice that has to be be made for types like

@Flag
struct MyType<T: AsyncSequence> { ... }

This is the same problem I brought up in the Reflection thread @filip-sakel linked above:

Further, it may not even always be possible to fulfill the archetype:

protocol NotImplemented {}

@Flag
struct CantUseMe<T: NotImplemented> {}

I think it should be possible to use meta types constrainted to protocols in the signature of the init(attachedTo:).

struct Flag {
    init(attachedTo archetype: @escaping (AsyncSequence.Type) -> MyProtocol.Type)
}

@Flag
struct MyType<T: AsyncSequence>: MyProtocol { ... }

It can be challenging through to express everything that can go into where clause.

I think with generalized existentials, it should be possible to capture where clause by combining all generic arguments into a dummy tuple type:

@Flag
struct Zipper<A: Sequence, B: Sequence> where A.Element == B.Element {
    typealias Element = A.Element
}

struct Flag {
    init(attachedTo archetype: @escaping (any <A: Sequence, B: Sequence> (A, B).Type where A.Element == B.Element) -> Sequence.Type)
}

But even this does not allow to capture the fact that Element of the resulting sequence type is the same as element of A and B.