[Pitch] Custom metadata attributes for types, and runtime access to custom sections

Evolution Pitches
1

This revises my pitch from last week, Runtime enumeration of custom sections, after the feedback there: it says what I want and how I would want it to work, rather than what a library can do today, and it drops the observation API. It also takes up SE-0385, which the Language Workgroup returned in 2023 asking for a design built on macros and constant initialization. Both now exist.

Swift has no way for a library to find the types that opted into it. Every library that needs this works around the gap. Swift Argument Parser takes a hand-maintained subcommands: array. SwiftLint regenerates a 254-entry BuiltInRules.swift with a tool whenever a rule is added. Quick walks the Objective-C class list and cannot on Linux. Swift Testing wrote a 358-line per-platform walker to find its own tests, and it is the one library whose section the runtime helps with on Linux. I built the same thing for my own framework, in three packages, and met the costs every other library pays: a record format the runtime cannot read, function pointers standing in for metatypes, a walker per platform, and re-reading image files from disk on Linux.

SE-0492 made the writing side a language feature. This pitch is the reading side. What it is like to use:

A library declares a key and applies it to types, directly or from its own macro:

@customMetadata public enum LintRule {}

@LintRule struct ArrayInitRule: Rule { ... }
@LintRule extension TrailingWhitespaceRule {}   // what a framework macro would emit

and asks for the types:

for case let rule as any Rule.Type in CustomMetadata.types(marked: LintRule.self) {
    registry.register(rule)
}

No registry, no code generation, no code at load time. A type registers by existing, in any module, on every platform, and everything else the library knows about it stays where it is today, on the type.

A library that already keeps its own records in a @section keeps them and drops its walker:

for section in customSectionContents(named: pluginSection) {   // the name as written in @section
    for record in section.bindMemory(to: PluginRecord.self) { register(record) }
}

It works on Apple platforms, Linux, Windows and Wasm, and on every deployment target the toolchain supports. On Linux it is the first way for a library to see other images' sections without re-reading their files from disk.

The full proposal, with the design, the restrictions and the alternatives, is here: swift-evolution/proposals/NNNN-custom-type-metadata.md at olaf/custom-type-metadata · OlafAndreas/swift-evolution · GitHub. It is written as two parts that may be reviewed separately. The section access is implemented, as a draft pull request to swiftlang/swift ([Runtime][IRGen][Sema] Emit and expose the SE-0492 custom section index by OlafAndreas · Pull Request #93419 · swiftlang/swift · GitHub): it builds and its tests pass on macOS and Linux, Windows and Wasm emission are checked by cross-compiling and linking, and swift-discovery runs on it with a ten-line change in place of its 315-line walker (GitHub - OlafAndreas/swift-discovery at olaf/stdlib-sections · GitHub). The attribute is not implemented, and an implementer is sought.

I would like to hear whether this shape fits the libraries you maintain, and what you would need from it.

Direction and design by me, built on SE-0385 by Pavel Yaskevich, Holly Borla, Alejandro Alonso and Stuart Montgomery, on SE-0492 by Kuba Mracek, and on the feedback in the earlier thread; prose by AI.

[Pitch] Runtime enumeration of custom sections
2

This is actually a very interesting example to cite in this space, and I'm glad you did. Because the naïve approach of "iterate over all the registered types with some 'command' attribute and register them" is the wrong approach. Since commands form hierarchies, you need to be able to specify which commands are contained by which other commands.

How would that work in the proposed design? A few ideas immediately come to mind:

  1. Subcommands by convention could be nested types in their parent command
  2. The attribute attached to a subcommand points back to the metatype of its parent command
  3. The attribute is attached to a subcommand and a property of that type is assigned the metatype of the parent command

But each of these options falls apart somewhere currently. The language today doesn't provide clean APIs to walk lexical type contexts to clean that information; an implementor would have to use hacks like stringifying types and breaking the names apart. For #2 and #3, they're awkward because the relationship is flipped: a subcommand must say "I belong to this parent command" instead of a parent command saying "these are my subcommands". Moreover, for #2 it doesn't appear that there's a way to attach type-specific metadata to the proposed attribute—it's just a "tag" that lets it appear in a list iterated at runtime. #3 is perhaps the most feasible solution, but it implies that there's a common protocol that these types would also conform to (so that the argument parser can cast them to something common and read that property out), which begs the question, "is the attribute redundant if the protocol conformance is present?"

Thinking this through makes me feel like what's being proposed is just a slightly different shape of "what if I could iterate over the types that conform to a protocol". And that makes me think that designing this around attributes on type declarations isn't the primary way we should surface this. Types already carry a large amount of metadata/descriptors, and if this just amounts to sort-of-a-conformance, then we should just expose an API to iterate the loaded conformances of a protocol instead—that information is already available in the runtime.

But if we want to provide custom metadata records, instead of tagging types and iterating over metatypes, I think I'd rather have a way of defining a type that's populated by constant values and then being able to iterate over that data.

3 Likes
3

Thank you for your remarks. I will start with the last part of your reply, why not just iterate over all types conforming to a protocol. The idea is tempting, for the reasons you give, but it has caveats:

  • A type may conform to the protocol and still not be meant for discovery: a test helper, a base implementation, a type that conforms because it needs the behaviour. A marker protocol separates the two, but the compiler only warns when another module restates a conformance the type's module already declared, and there is no API to enumerate it. The proposal's key is refused with an error that travels in the module interface, and its query exists.
  • A library may want to say about a type that it requires exactly one conforming provider.
  • A project may want to say about a type that it is the one provider for that requirement.
  • And I want those two guarded at compile time where possible, at link time otherwise, and at run time across images.

A conformance query can express none of this; it returns whatever conforms. An explicit application is a declaration about one type, and the compiler can turn each one into a symbol: a strong definition per application and an undefined reference per demand, so a duplicate or a missing provider fails the link, as a second @main or a missing one does today, and the runtime checks across images at query time. Conformances are too common, and too much about behaviour, to carry that. None of this is in the pitched version; it is under Future directions, uniqueness and existence constraints, and a branch of my mediator approximates the link-time half with today's macros. That is my own case, a mediator with exactly one handler per request, and it is why I want the record to be a compiler-known declaration rather than a conformance.

Two more things keep a conformance query from being the primary surface. The runtime indexes conformances from type to protocol, so the reverse query scans every conformance record in every image, while the attribute section holds only records written to be found. And it has been tried at scale: Swift Testing's original discovery walked the type metadata table for conformers of a __TestContainer protocol; its 2025 move to records in swift5_tests (#893) lists the reasons, and the legacy path was removed this June (#1013). A public conformance query may still be worth having, and Alternatives says it could be a future direction. It is a different feature, not a substitute for opt-in.

Hierarchy. You are right. The proposal's Motivation cites the subcommands array as a hand-maintained list, and it also carries structure, which a flat set does not. The attribute answers one question, which types opted into this key; the parent, like every other fact about a type, stays on the type and is read after discovery (Part II). Your second option, an argument naming the parent, is the attribute arguments I come to below. Within one module the array stays the natural form, and nothing here takes it away; the flipped direction earns its place when a subcommand lives in a module the parent's author does not control and cannot be added to the parent's array.

Custom metadata records. Yes, and the proposal treats them as the next step rather than the alternative. Attached to a type, they are Part II's first future direction, attribute arguments: a key with stored properties of constant-expressible types, filled at the application site and returned with the type. The compiler writes that record itself, the current record reserves a field for it, and Alternatives says why this version stops at the marker. Standing alone, they are the linker set SE-0492 sketched, @LinkerSetEntry-style macros over a section API; Part I is that API, reading the data as raw bytes today with the limits listed under Motivation, and the macros become possible as libraries on top of it. That form also waits on something outside this proposal: SE-0492's constants admit no user-defined types, and Swift 6.4 rejects a struct value in a @section global with "not supported in a literal expression", so a type populated by constant values cannot be placed in a section yet. SE-0492 lists generalizing constants as its own future direction.

That is the spirit of the pitch. It is not "we need bricks to build the house" but "we need clay to make bricks": Part I is the clay, the marker attribute is the first brick, attribute arguments and linker-set macros are the next ones, and the house, a rule pipeline or a command tree, stays the library's to build.

If you have a concrete rule table or command tree in mind, I would like to hear what its record would need to carry beyond the type.

4

Such a thing could be useful, but we'd need to extend it to methods as well imho -- types are not fine granularity enough for the most interesting cases, and today we're doing things somewhat ad hoc -- like swift testing or distributed do; I would be interesting in finding ways to express such metadata on methods as well, but of course locating the metadata by "method handle" becomes more tricky then.

If we did such a thing for types though, I do think it should have a path to do the same style of access for methods, and not something completely different.

5

I agree that types are the coarse end, and the proposal gives functions only a line. The path I see: a function has no context descriptor, but the runtime already has the shape that stands in for one, the accessible-function record distributed actors use, with a name, the mangled type, an optional generic environment and the entry point. I would borrow that shape, not the table, whose one reader expects distributed records: a function record in the attribute's own section, with a kind in the reserved flags field, found by the same walk, availability check and deduplication, so an extension rather than a second mechanism. What that version would still have to design is the typed call through the handle, which is where SE-0385 lost information for functions; the generic environment and the mangled name, labels included, give it a better start.

In the meantime a library can do it on Part I today: SE-0492 constants admit function references and captureless closures, Swift 6.4 accepts both in a @section global, and Swift Testing's record is exactly that. So yes: the path to functions should be visible before Part II's record is final, so that the reserved fields are shaped by it rather than hoped to fit. I will draw it out in the future direction in the next revision. If that record shape is the handle you would want for methods, I would like to know.

6

Accessible function records are indeed what I have in mind here. They're not actually tied to distributed at all, they've just not yet been used by other features. There's a flag that marks a record to be a distributed function accessor; One could flag other records otherwise.

It would be good to explore how to get the mangled name for a specific call target. For example perhaps we need a #mangle(targetMethod: thing("", 12)) (naming TBD) which is easier to spell from a call site before actually calling such method -- you'd just forward parameters to it:

func example(s: String, _: Any) {} // we want metadata about this one
func example<O>(o: O, _: Any) {}

func call<G>(s: String, g: G) {
  let _id = #mangle(targetFunc: example(o: s, g)) // get mangled name of target
  let m = _somehowGetMetadata(_id) // TBD but effectively mangled name based

  guard validate(m) else { return } // validate target call based on metadata
  example(s: s, g) // actually make the call
}

which instead of calling the specific target method identified with a normal call-like looking description like this, would mangle the identifier? By which we'd then do a lookup of accessible function and potentially associated metadata :thinking: I expect to be looking into something like this in the near future, so if there's others exploring the space of metadata lookup like this it might be good to already keep functions in mind and make sure we could converge in the near future.

Edit: I just remembered swift-evolution/proposals/0479-method-and-initializer-keypaths.md at main · swiftlang/swift-evolution · GitHub which was returned for revision back then. It would be a reasonable way to spell this.

7

Good, then I had the table too narrow: the lookup is by name, and the Distributed library only asserts the flag on a hit. That shortens the path. IRGen already emits such a record, distributed bit clear, for any SIL function marked [runtime_accessible], a flag nothing sets from source today. A marked function or method would set it, with a flag for its kind, and Part II's attribute record would point at that record instead of a context descriptor, the kind in its reserved flags. Enumeration then walks the attribute section as for types and hands back the accessible-function record as the handle.

#mangle is the piece neither side has. #selector and #_hasSymbol already resolve a declaration reference at compile time, but pick an overload only with an as cast; your call-shaped argument lets the arguments pick it. From the name, your lookup goes name to record, as swift_findAccessibleFunction does, and then record to its attributes. That reverse edge is what Part II would add, a scan-and-cache over the attribute section, and the design question to settle before the record is final.

So I will write the function case into the future direction with the existing table and a kind flag, as you describe, and keep functions in view from here on, so that when you take this up there is one record for the two efforts to converge on.

8

I just remembered method key paths which were pitched a while back but returned for revision: swift-evolution/proposals/0479-method-and-initializer-keypaths.md at main · swiftlang/swift-evolution · GitHub

Reviving these might be useful for this purpose.

9

I'm glad the work we've done on test discovery is useful to you. However, I again want to emphasize that the way Swift Testing does things is not meant to be generalizable to all kinds of metadata. We have constraints and future plans that are specific to the testing domain.

1 Like
10

Thank you, and point taken. I have leaned on Swift Testing in this thread more than is fair, as the library that has done this in production, and I can see how that reads as treating its design as a template. That is not the intent. The proposal does not touch swift5_tests, its records or its kinds, asks nothing of Swift Testing, and should not stand in the way of your plans for it; the Alternatives section says as much, following what you wrote in the earlier thread. What I take from Swift Testing is that the need is real, not its record or its section.

11

Method key paths would fit well, and I think a library could use them on Part I alone: a key path is not a constant, but a @convention(c) closure that produces one is, so a record of an id and such a closure can sit in a @section, Part I enumerates it from every image, and a table keyed by the key path can then be built on the consumer's side. I tried this with two modules on the SE-0479 implementation, with plain methods and with a generic method whose type arguments the key path fixes, and the key paths compared equal across the modules.

What that route does not seem able to carry is a name the runtime can look up, or the generic method itself. Both are what the accessible-function record has, so I would like the next revision to write the function case against it, and to have the runtime ignore attribute records whose flags bits it does not define. The step from a key path to that record is, as far as I can see, in neither SE-0479 nor this proposal yet; that may be where the two meet.

12

You shouldn't even take that from Swift Testing. The need that Swift Testing has is, fundamentally, specific to Swift Testing. That doesn't mean that other code doesn't need @section or corresponding discovery mechanisms, but it also doesn't mean other code does.

13

Point taken, and I will leave it out of this from here. For what it is worth, the need came to me from my own framework before I had looked at Swift Testing at all; it just seemed to me that we were in the same ballpark.

14

A concrete example of where this would be useful is RealityKit's registerComponent: registerComponent() | Apple Developer Documentation

It's a very annoying manual step that is always forgotten about when iterating on a new project.

I can see lots of useful use cases for this feature.

15

Yes, Part II of the pitch would make that possible: the component type opts in where it is declared, and the framework could collect every such type when it starts. Whether RealityKit would do that is Apple's decision, as it is for any framework owner. What the pitch provides is the mechanism.