[Pitch] Custom Metadata Attributes

Hello everyone!

The following is a pitch for adding an ability to declared a runtime discoverable attributes in the Swift language. We welcome any feedback or questions about this pitch!


Introduction

In Swift, declarations are annotated with attributes to opt into both builtin language features (e.g. @available) and library functionality (e.g. @RegexComponentBuilder). However, custom attributes defined by libraries are limited - libraries can currently define custom attributes for property wrappers, result builders, and global actors. This proposal introduces the ability to attach library-defined runtime metadata to declarations using custom attributes, which can then be queried by the library to opt client code into library functionality.

Previous Swift Forum discussions

Motivation

There are some problem domains in which it can be beneficial for a library author to let a client annotate within their own code certain declarations that the library should be made aware of, since requiring the client call an explicit API instead would be too onerous, repetitive, or easy to forget.

One classic example is testing: there is a common pattern in unit testing libraries where users define a type that extends one of the library's types, they annotate some of its methods as tests, and the library locates and runs all tests automatically, after initializing a distinct instance of the containing type for each. There is no official mechanism in Swift to implement this test discovery pattern today, however XCTest—the language's current de-facto standard test library—has longstanding workarounds:

  • On Apple platforms, XCTest relies on the Objective-C runtime to enumerate all subclasses of a known base class and their methods, and it considers all instance methods with a supported signature and name prefixed with "test" a test method.
  • On other platforms, XCTest is typically used from a package, and the Swift Package Manager has special logic to introspect build-time indexer data, locate test methods, and explicitly pass the list of discovered tests to XCTest to run.

XCTest's current approach has some drawbacks and limitations:

  • Users must adhere to a strict naming convention by prefixing all test methods with the word "test". This prefix can be redundant since all tests include it, and may surprise users if they accidentally use that prefix on a non-test method since the behavior is implicit.
  • Since tests are declared implicitly, there is no way for a user to provide additional details about an individual test or group of tests. It would be useful to have a way to indicate whether a test is enabled, its requirements, or other metadata, for example, so that the testing library could use this information to inform how it executes tests and offer more powerful features.
  • The lack of a built-in runtime discovery mechanism means that related tools (such as Swift Package Manager) require specialized discovery logic for each test library they support. This makes adding support for alternate test libraries to those tools very difficult and increases their implementation complexity.

The general pattern of registering code to be discovered by a framework is common across Swift programs. For example, a program that uses a plugin architecture commonly uses a protocol for the interface of the plugin, which is then implemented on concrete types in clients. This pattern imposes error-prone registration boilerplate, where clients must explicitly supply a list of concrete plugin types or explicitly register individual plugin types to be used by the framework before the framework needs them.

Java and C# programming languages both have a feature which is called “Java Annotations” and “C# attributes” respectively. It allows to add attributes on methods, variables, parameters, and, in case of Java, packages that are accessible to Java/C# compiler and at runtime via reflection APIs. We propose a similar addition to Swift called custom metadata attributes.

Proposed solution

  • A new builtin attribute @runtimeMetadata that can be applied to structs and classes.
  • Types annotated with this builtin attribute can be used as custom attributes on declarations that can be used as values.
    • The custom attribute can have additional arguments; the custom attribute application will turn into an initializer call on the attribute type, passing in the declaration value as the first argument.
  • A reflection API that can gather all declarations with a given custom attribute attached.

Detailed design

Declaring runtime metadata attributes

Runtime metadata custom attributes are declared by attaching the built-in @runtimeMetadata attribute to a struct or class:

@runtimeMetadata
struct Example { ... }

A runtime metadata type must have an initializer of the form init(attachedTo:). The type of the attachedTo parameter dictates which types of declarations the custom attribute can be applied to, as described in the following section.

Applications of runtime metadata types

Runtime metadata attributes can be applied to any declaration that can be used as a first-class value in Swift, including:

  • Types
  • Global functions
  • Static and instance methods
  • Instance properties

Runtime metadata types opt into which kinds of declarations are supported using initializers. For an application of a runtime metadata attribute to be well-formed, the runtime metadata type must declare an initializer that accepts the appropriate value as the first argument. Applications of a runtime metadata type to a type will synthesize an initializer call with the attribute arguments, and the declaration value passed as the first initializer argument:

  • Types will pass a metatype
  • Global functions will pass an unapplied function reference
  • Static and instance methods will pass an unapplied method reference
  • Instance properties will pass a key-path
@runtimeMetadata
struct Flag {
   // Initializer that accepts metatype of a nominal type
   init<T>(attachedTo: T.Type) {
     // ...
   }
   
   // Initializer that accepts an unapplied reference to a global function
   init<Args, Result>(attachedTo: (Args) -> Result) {
     // ...
   }
   
   // Initializer that accepts an unapplied reference to a method (static &
   // instance).
   init<T, Args, Result>(attachedTo: (T) -> (Args) -> Result) {
     // ...
   }
   
   // Initializer that accepts a reference to an instance property
   init<T, V>(attachedTo: KeyPath<T, V>, custom: Int) {
     // ...
   }
}

// Compiler is going to synthesize following generator call
// -> Flag.init(attachedTo: doSomething)
@Flag func doSomething(_: Int, other: String) {}


// Compiler is going to synthesize following generator call
// -> Flag.init(attachedTo: Test.self)
@Flag
struct Test {
  // Compiler is going to synthesize following generator call
  // -> Flag.init(attachedTo: Test.computeStateless)
  @Flag static func computeStateless() {}

  // Compiler is going to synthesize following generator call
  // -> Flag.init(attachedTo: Test.compute(values:))
  @Flag func compute(values: [Int]) {}
  
  // Compiler is going to synthesize following generator call
  // -> Flag.init(attachedTo: \Test.answer, custom: 42)
  @Flag(custom: 42) var answer: Int = 42
}

A given declaration can have multiple runtime metadata attributes as long as a given runtime metadata type only appears once:

@Flag @Ignore func ignored() { 🟢
  ...
}

@Flag @Flag func specialFunction() { 🔴
      ^ error: duplicate runtime discoverable attribute
  ...
}

Runtime metadata attributes must be applied at the primary declaration of a type; applying the attribute to a type in an extension is prohibited to prevent the same type from having multiple runtime metadata annotations of the same type.

@Flag extension MyType [where ...] { 🔴
 ^ error: cannot associate runtime attribute @Flag with MyType in extension
}

Inference of metadata attributes

A metadata attribute can be applied to a protocol:

@EditorCommandRecord
protocol EditorCommand { ... }

Conceptually, the runtime metadata attribute is applied to the generic Self type that represents the concrete conforming type. When a protocol conformance is written at the primary declaration of a concrete type, the runtime metadata attribute is inferred:

// @EditorCommandRecord is inferred
struct SelectWordCommand: EditorCommand { ... }

If the protocol conformance is written in an extension on the conforming type, attribute inference is prohibited.

// @EditorCommandRecord is not inferred
extension SelectWordCommand : EditorCommand { 🔴
   ...
}

Runtime metadata attributes applied to protocols cannot have additional attribute arguments; attribute arguments must be explicitly written on the conforming type.

Accessing metadata through reflection

With the introduction of the new Reflection module, we feel a natural place to reflectively retrieve these attributes is there.

public struct Attribute {
  public static func getAllInstances<T>(of: T.Type) -> [T?]
}

This API will retrieve all of the instances of your runtime attribute across all modules. Instances of metadata types are initialized in the runtime query to gather the metadata.

API Availability

Custom metadata attributes can be attached to declarations with limited availability. The runtime query for an individual instance of the metadata attribute type will be gated on a matching availability condition, returning nil if the availability condition is not met at runtime. For example:

@available(macOS, introduced: 12)
@Flag
struct NewType { ... }

The runtime query that produces the Flag instance attached to NewType will effectively execute the following code:

if #available(macOS 12, *) {
  return Flat(attachedTo: NewType.self)
} else {
  return nil
}

Alternatives considered

Alternative attribute names

  • @runtimeMetadata
  • @dynamicMetadata
  • @metadata
  • @runtimeAnnotation
  • @runtimeAttribute

Bespoke @test attribute

A previous Swift Evolution discussion suggested adding a built-in @test attribute to the language. However, registration is a general code pattern that is also used outside of testing, so allowing libraries to declare their own domain-specific attributes is a more general approach that supports a wider set of use cases.

39 Likes

Having used attribute-based patterns in other languages (Java, Kotlin), this seems rather antithetical to Swift's common approach of compile time verification. Attribute-based programming suffers in readability, reasonability, debugability, and documentability. It's an obvious solution to the motivating problems, and well precedented in other languages, but these patterns are also kind of hated in those languages, especially by newer programmers. Is there no other solution to these problems?

Aside from my fundamental concerns, the proposal seems rather under specified in regards to how these attributes interact with other attributes and attribute-like constructs. What do these things see when attached to @MainActor or @SomePropertyWrapper? How does ordering impact things?

There's also no notion of how it applies to async constructs. There doesn't seem to be any initializers that deal with them. I would hope we wouldn't introduce any new features that don't work with async, especially while existing features, like property wrappers or dynamic member lookup, don't.

Additionally, I'm wary of adding yet another attribute type when the tooling for other attribute types is still so poor. Is there anything, generally, that can be done to give these sorts of things a better experience out of the box? I know that sort of thing isn't supposed to be part of the proposals, but the situation is getting dire and I'd rather not add yet another Swift feature that has no compiler guidance for implementation.

(As a general aside, I'd really like to see Swift's existing feature set fleshed out, especially concurrency, rather than introducing a bunch of new features.)

11 Likes

The attribute is attached to the declaration not type of thereof, so ordering of other attributes doesn't impact it in any way.

So what would this attribute see when attached to multiple wrappers? Would @WrapperA @WrapperB @Metadata var property: String be WrapperA<WrapperB<String>> or just String?

1 Like

It sees property not its backing storage so it would still be var property: String that it sees.

1 Like

This is something which could potentially be very useful for Realm and possibly eliminate our dependency on the obj-c runtime while also improving the API in a few places. I'll try to find time this week to write up my ideas on where we might use this and try to evaluate if it actually would work for us.

6 Likes

This looks exciting, @xedin!!

The API for accessing attributes seems a little thin — are you planning to expand on what information we can access? It would be helpful to know things like the enclosing type of an instance property/method, the attributed symbol's name, etc. In addition, could we query in the other direction? Given a key path, could we query for a list of attributes?

(I'm thinking through how this capability could influence the design of libraries like swift-argument-parser.)

1 Like

@Alejandro this is a question for you :) We are storing enough information do that in the metadata section, so I'd be inclined so say yes.

I have not personally used the attribute features in other languages. Could you please elaborate on why this form of attribute-based programming suffers from readability, reasonability, etc, especially in ways that Swift's current custom attribute features do not?

I think using reflection-like metadata paired with a runtime query for runtime registration and discovery is the fundamental solution to building registration into the language. I could imagine other spellings for the metadata itself in the language, but I'm not sure how else these patterns would be supported. If anyone else has other ideas, I'd love to hear them!

The general feature composes with async; I believe the only thing blocking this feature from actually being used with async is enabling async key-paths and unapplied method references.

Just so I'm clear - what specifically about the programming experience could use improvement when working with custom attributes? Are you talking about specific limitations in code completion, cursor info (i.e. QuickHelp), etc? Much of the SourceKit functionality is powered by the Swift type checker, so we can certainly incorporate compiler guidance for proposal implementors.

2 Likes

I've used field tags in Golang, and this looks somewhat similar. Field Tags in Go are used to attach metadata specifically to properties in structs. This metadata can then be used to inform different subsystems on how to interact with that property. For example, you can say "when decoding from JSON, this "bar" property should actually use the foo key", or "when extracting from a URL, this comes from the query parameter baz", and so on.

I echo some of the other concerns that these look a lot like property wrappers, and I think that has the potential to be quite confusing.

From a developer point-of-view, I'm wondering:

  • Can I restrict which types are allowed to use one of these attribute annotations, perhaps by adding generic constraints to the types involved?
  • Can I include payload information in these types, so that I could conceivably do things like @JSONKey("foo") on a property declaration?
  • Do I have a way to say "this attribute can only be used on properties", or even "This attribute can only be used on properties that are held by a Decodable type"?
  • When are these attachedTo: methods executed? If it's at load time, what are the potential impacts on process start-up time? If it's at first access, how do you handle a request to getAllInstances if a specific type hasn't been touched yet?
  • Am I expected to implement this getAllInstances method myself? Am I implementing the attachedTo: methods myself? Or are these all synthesized by the compiler? If they're synthesized, how do I add type restrictions? If I implement them myself, how much responsibility do I have around concurrent access to attachedTo: vs getAllInstances, etc?
9 Likes

Yes, you can do that via overloading init(attachedTo:) associated with runtime attribute type.

Yes you can do that as well, with parameter that go after attachedTo:, like in the example with custom: parameter.

Yes, if you only declare init(attachedTo: KeyPath<...>) it would indicate that this attribute type could only be associated with properties.

They are not executed unless your program requests an instance of the attribute via Reflection API.

9 Likes

EDIT: jinx @xedin! :slightly_smiling_face:

I'm going to answer these out of order.

init(attachedTo:) is something you implement yourself as the author of one of these @runtimeAttribute types. getAllInstances is part of the Reflection library, for which you do not need to provide an implementation.

Yes, you would do this by writing generic requirements on your init(attachedTo:).

Yes, init(attachedTo:) can take additional arguments, and you can do whatever you want with those in your runtime attribute type, e.g. store them to be discovered later in the runtime query.

Yes, you do this by only providing init(attachedTo:) that takes a KeyPath. You can restrict the base type of the key-path using a generic requirement.

They are executed as part of the runtime query, i.e. when you call getAllInstances.

5 Likes

This proposal is potentially very powerful -- facilitating meta-programming at runtime. Many issues with that have been surfaced by successful implementations, and it would be helpful to have a proposal that matched the detail in their designs, including alternatives that match known successful examples.

Today in Swift the use-cases mentioned could be addressed with code generation, and macros are promising to bring something similar into the compiler.

A very helpful first step could be what in Java are compile-time annotations, together with compile-time annotation processing.

A key feature of Java annotation declarations is that the annotation parameters are compile-time constants, so the compiler can resolve the annotation attributes early and use the annotations later.

I believe macro processing is the compile-time processing facility of interest now in Swift.

So perhaps this proposal could be to specify compile-time annotations resolved before macro processing, and the ability to target macros to annotated elements.

Otherwise, one constraint or goal I would add for Swift runtime reflection is to maintain type safety. Java proxies created decades of pain through type casting, but Spring used AspectJ to get compile-time type-safety for aspects on other code, avoiding an entire category of bugs and confusion. (Macros while ungainly also can offer the same guarantees.)

Swift's current attribute do suffer from those issues, but due to their limited scope, it's much more manageable. Debugging property wrapper interactions with local variables is still pretty much impossible, so in that regard this proposal can't make things much worse. But many of the issues come from the dynamic nature of this feature, as we can see in other languages.

  • Readability: relatively minor, especially if you don't care what the attributes are actually doing, but determining the logic of a function when it has multiple attributes and a non-trivial body can be rather difficult, especially at a glance.
  • Reasonability: reasoning about multiple dynamic attributes feels like it has polynomial complexity. Once you get beyond one or two it becomes very difficult to determine what the actual behavior will be. This is especially true if any of the attributes interact with each other behind the scenes, as we see with frameworks like Spring, where various attributes contribute to a backend runtime system in different ways.
  • Debugability: essentially impossible, especially for closed source attributes where you can't place the debugger in the wrapper's implementation. This is true of the existing attributes, but give their more rigid behavior, debugging those isn't as important (though SwiftUI's wrappers would hugely benefit from debugging). For fully dynamic attributes like these, it makes it impossible to debug dynamic runtime behavior, which is the core of the feature. If we can't determine why something is triggered when wrapping one expression but not another, it will be very painful. Some sort of debug integration would be nice.
  • Documentability: this is similar to the composability issues that are visible in the areas already mentioned, but for documentation. Given that wrappers can have huge runtime behavior impacts, documentation is key. At the least, I think this means that generated quick help and the inline doc comment generator need to be updated to include summaries or at least links to the documentation for the attributes. Same for the current attributes, but again, those have much lower impact than arbitrary runtime behavior.

You make it sound like this is some core problem the language has to solve and it isn't. Why couldn't Swift offer more limited but statically safer features built specifically to solve particular problems rather than arbitrary runtime access? Why is Swift suddenly abandoning on of its earliest core principles and differentiators from Obj-C? What makes Swift's version of this capability better than what we see in other languages?

In any case, I'm not an Apple language designer, so no, I don't have any particular suggestions for solving the problem as you've defined it. (It does seem a bit circular though.)

That's good, but I believe similar things were said about concurrency and property wrappers, yet the blocks were never removed. I hope this feature can ship and interact with all of Swift's existing constructs.

All of them. Builders are the worst but all of the attributes generally suffer from a lack of autocomplete, sometimes in the attribute itself but almost always in the required members. The members themselves have no inline documentation inherited, so even if it did autocomplete there's no help there. And these members don't show up in Swift's documentation either, nor do most of the attributes themselves. There are also no fixits for most of them, like @dynamicMemberLookup, so you have to manually add their implementations.

3 Likes

People have been asking for runtime registration (or global constructors, as an indirect way of implementing it) pretty much since day one of Swift, so I'm excited to see this move forward.

It isn't clear to me after reading the pitch where the synthesized initializer calls for these attributes get executed. Is it expected to be constant-evaluable, so the metadata can be emitted as a static initialization? Is it injected into main? Does it gasp introduce a static constructor? Does it run lazily when someone asks to getAllInstances?

What should the behavior be when code is dlopen-ed (or whatever the host platform's equivalent for dynamically loading code is)? For some of the possible answers above, it seems like we might need to also provide a way to notify interested code when newly annotated members have been introduced into the process.

Thinking about @Jon_Shier's concerns, I think there is some subset of situations where the attribute and the full set of its users and consumers can be statically determined—for instance, if the @runtimeMetadata type is internal to a module, then it can only ever be applied to other declarations in the same module, and if we're building with WMO, we could reduce the list of getAllInstances to a static array. It may also be interesting to be able to do so for publicly-provided attributes provided by a framework, but intended to be used by a single module, such as the main module for an app. If we provided a getAllInstancesForModule then that query could generally be reduced to a static literal array for WMO'ed modules.

The proposal states that:

Types and key paths have natural identities, but function values do not, so they might be too weak of a type if consumers of the attribute want to do anything more with the function than call it. The "curried" representation of unapplied method references is additionally extremely inefficient, unsound for mutating methods, and should be abolished. Using key paths also restricts what kinds of property declarations can be annotated, as others noted. I wonder if some kind of richer Declaration type that has identity and can also be used to invoke the method, ask for its name, and do other more reflective things would make sense.

19 Likes

This seems very interesting, powerful and needed. But I also was surprised on seeing this direction since adding attributes to everything doesn’t seem that nice in other languages. You often endup with a sort of weird undiscoversble DSL.

My concern is about how to use those, I’ve only seen mention of runtime discovery. Aren’t there any plans on allowing us to access this information somehow at compile time? Most of the things I use metadata are to tell Sourcery how to generate code. I would love to have a compiler solution for that.

2 Likes

It seems like we cannot attach a custom metadata attribute to a generic type, because a generic type is really a type constructor, so there is no metatype object for it. For example this is illegal:

@Flag struct Container<A> { }

Is my understanding correct?

5 Likes

I am delighted to see this coming to the language at last!

Flagging test methods is a great use case. The other thing I’ve had most experience with that (1) is a use case for this and (2) does not fit tidily in Swift’s current API is ORM frameworks like Hibernate, where compile-time metadata is a godsend.

I wonder whether this feature could also replace Codable’s rather awkward and repetitive way of specifying custom attribute names.

Before:

struct Doodad: Codable {
    var id: String
    var width: Int
    var height: Int
    var text: String
    var flavor: String
    var fungible: Bool
    var createdAt: Date?

    enum CodingKeys: String, CodingKey {
        case id  // SO MUCH REPETITION...
        case width
        case height
        case text
        case flavor
        case fungible
        case createdAt = "created_at"  // ...just to change one name!
    }
}

After:

struct Doodad: Codable {
    var id: String
    var width: Int
    var height: Int
    var text: String
    var flavor: String
    var fungible: Bool

    @Coding(key: "created_at")  // aaaahhhh
    var createdAt: Date?
}

This example pushes back against @Jon_Shier’s intense dislike of the feature:

I…don't think it's true that those features are “kind of hated.” Or at least…we’ve talked to different people. I don’t see JUnit or Hibernate users fleeing from annotations. Unity developers certainly lean hard on C# attributes and AFAIK like that arrangement just fine.

(It is true that Java's use @NotNull and friends are widely disliked. Maybe that’s what you’re thinking of? But that dislike is because poor Java is stuck having null in the first place, and its almost-but-not-quite-like-gradually-typed-optionals nullability annotations are an awkward retrofit onto its older type system, not because they're an idea that would otherwise be good except they’re expressed as annotations. Or maybe you’re thinking of Spring? Its problems similarly are often exposed at annotation sites, but are not caused by annotations.)


In any case, the codable example above puts concerns like these in perspective:

The alternative to metadata attributes attached to code is not that the metadata doesn’t exist anywhere and thus there is no complexity! No, the alternative is having that same metadata expressed in more awkward forms elsewhere, in places such as:

  • ad-hoc data structures (like Codable) that tend be redundant and awkward, or
  • config files that are harder to discover and have less static verification.

In either case, Jon’s concerns about how attributes behave when composed is not mitigated by moving that metadata elsewhere.


I do agree that with Jon that we should try for as much static verification of custom metadata as possible. It would be good to sketch out a hypothetical ORM example, and think about what developer errors type constraints might be able to prevent.

13 Likes

This seems intensely broken:

Is there no way around it? It seems like it would lead to some really surprising bugs.

7 Likes

I presume that “attribute inference is prohibited” means that EditorCommand conformance cannot be applied in an extension unless the core type already has @EditorCommandRecord, rather than that @EditorCommandRecord will silently not be applied.

1 Like