SE-0385: Custom Reflection Metadata

Evolution Proposal Reviews
61

The ordering doesn't matter for reflection attributes and property wrappers because they do not change the type or name of the declaration they are attached to but for macros I'm not sure, if they are allowed to do that (change name/type etc.) then the compiler has to make sure that the macro attributes are run before all other attributes are considered.

62

I'd like to bring attention and solicit some feedback on a potentially surprising behavior when attributes are attached to methods:

@runtimeMetadata
struct Flag {

  init<T>(attachedTo: (T) -> Void) {}
}

struct Test {
  @Flag func method(_: Int) {}
}

This example is (currently) accepted by the compiler because T is allowed to match (Test, Int) as a tuple while synthesizing a generator function for func method. This means that the call to Flag.init in the generator would have the following signature - ((Test, Int)) -> Void.

This is allowed because SE-0110 allows tuple "splatting" of closure parameters and that's how method references for custom reflection metadata attributes are implemented to be able to flatten type from (Test) -> (Int) -> Void which doesn't work for mutating methods.

This might not be an issue beyond some toy examples because there would be no way to operate on the first argument that represents a "value of type" and adding a protocol requirement like T: Actor would result in rejection of method by the compiler. Nevertheless, I think it's worth discussing whether code like that should be accepted by the compiler.

4 Likes
63

Oh, that's sort of wacky. I could definitely see myself being tripped up by it as my initial thought was "wait, why shouldn't that work?" and I had to reread it a few times, but as you say it's hard to imagine what real scenario where this could be an actual problem. You'd have to be casting T to some known type (or set of types) inside init(attachedTo:), and then you'd get a runtime failure of some sort instead of a compile-time one.

It doesn't seem important to forbid but I would fall on the side of rejecting it.

1 Like
64

Honestly, I could see that being used intentionally before variadic generics land.

65

Chiming in to say that we've (@xedin and myself) looked into distributed actors and methods being able to be discovered using such runtime metadata and I'm very excited for this capability. We made sure it'll work and I think it could unlock the ability to build exciting frameworks that "register" all distributed actors with a specific actor system as "endpoints" or "public api" etc. Really looking forward to giving it a spin eventually :+1:

5 Likes
66

Since that's a weirdness I've bumped into let me give my 2c on it :slight_smile:

It seems pretty accidental and surprising so while discussing it with @xedin thought it might be worth banning to prevent confusion like that. @bbrk24 mentions it might be a workaround until variadic generics land, but I'd be worried about popularizing this workaround -- firstly because I don't think it saves boilerplate: one would still have to likely match over the T to figure out what kind of function we got there -- so there'd still be n matches and paths trying to handle each function arity. Rather than this way, it seems like the way to handle this is to declare many initializers, with 0-n generic arguments I suppose. Though the real solution would be variadic generics of course, and since they're being worked on I think it's not worth introducing this weird workaround as it'd be immediately replaced by the proper way (I hope?).

Anyway, just my few cents as someone who hit this and was very confused why something was compiling that I thought "clearly" shouldn't :-)

1 Like
67

I added an example of property wrapper metadata to the motivation section in [SE-0385] Add another use case to the motivation section. by hborla Ā· Pull Request #1944 Ā· apple/swift-evolution Ā· GitHub

3 Likes
68

Excited to see this feature proposed. It's nice that there are a number of use cases for which this design would be the best solution as compared to alternativesā€”agree that this information would be very nice to have in the proposal text itself. The ideal Motivation section should not just motivate the core of the feature generally but also give a sense of why the boundaries are drawn where they areā€”in this case, for example, why one should be able to attach metadata to all the different kinds of declarations supported. Based on the discussion thus far I'm quite convinced that the authors have thought this through very well, but I think the tenor of the discussion is fair that the proposal text itself doesn't convey all of that thought.

As a point of procedure, I'm not sure what to make of the fact that the proposal includes extensions to a Reflection library which has been pitched but not yet proposed: should those parts be considered subset out of any approval pending the base Reflection proposal having gone through the Evolution process? Should it just be reviewed as part of that latter proposal? Otherwise, it would seem that acceptance here presupposes that the overall Reflection library should take the shape in which it's been pitched before review has occurred.

A few specific points:

  • Agree with @AlexanderM that using #function for type name is quite odd: Isn't static type information retrievable via the generic parameter anyway? Is there another sensible alternative behavior here that doesn't reuse #function for, well, not-a-function?

  • Interesting that (T, { Args... }) -> Return rather than (T) -> ({ Args... }) -> Return is the relevant parameter type. Totally understand that this is for alignment to the inout case which cannot be supported otherwise and agree with the choice, but I wonder if that is part of what's causing the issue that arises due to SE-0110. In any case, I agree that it is better to special-case the handling to prohibit that unusual implicit splatting behavior here.

  • I have concerns about the ad hoc rules around inferring metadata from protocol requirements when conformance is declared on the primary declaration but not extensions, except same-module ones that are unconditionally unavailable, as well as the ad hoc rules around same-module extensions of types: Given recent experience with similar ad hoc rules on property wrappers leading to confusion, I think we should seriously scrutinize and have a high bar for deviations from rules we already have. Could at least 80% of these use cases be preserved by adopting the same rules we have for synthesized Equatable and similar protocolsā€”namely, primary declarations and extensions only within the same file? This way, it would not be necessary to have a special-case rule for unconditional availability in order to prevent users from creating same-module different-file extensions.

2 Likes
69

We couldn't fine one but open to suggestions.

And mutating functions as well. The issue is indeed due to the fact that we need to thunk via a closure to be able to flatten argument list.

We could switch to same file, but that's not the reason why unconditionally available extensions are allowed - they provide a way for users to opt-out from use of the reflection metadata flag.

70

This feature cannot be fully subsumed my macros. Storing custom metadata values separately and initializing the metadata values lazily are both a major part of the motivation for this approach to improve performance for these sorts of code patterns. Any approach that puts all values in a global array, initializes metadata on app launch, etc, will pay the cost of eager metadata initialization, which does not serve the performance goals of this proposal.

Macros will not have the power to add things to dedicated metadata sections without some other language primitive for expressing that. That said, I could imagine an alternative approach where we have that "put this value in a metadata section" primitive, and the ad-hoc metadata value construction is performed by a macro expansion.

3 Likes
71

I don't follow. Firstly, globals in Swift are lazily initialised, so I'm not sure why this would cause any additional work to be performed at app launch. From TSPL:

Global constants and variables are always computed lazily, in a similar manner to Lazy Stored Properties. Unlike lazy stored properties, global constants and variables donā€™t need to be marked with the lazy modifier.

Moreover, macros have complete control over how they surface their generated data. I'm not suggesting all metadata from all attributes get clumped in to a single global array as a language feature; a macro-driven approach would be decentralised, where each macro visits annotated declarations at compile-time (instead of at run-time), and can generate whichever data structures/global variables/functions/computed properties/etc it needs to capture that data.

I also don't think the compile-time approach I'm suggesting would lead to worse performance than the proposed run-time approach.

Consider something like an ORM library. If you use reflection metadata in any part of the object<->row mapping, you need to generate that mapping at runtime using reflection, and that imposes a launch cost. But that mapping is never going to change unless the application is recompiled, so it could instead be evaluated at compile-time by a macro and emitted as static data, not using reflection at all, and with less (or no) launch cost. The same applies to test discovery - why use reflection to search for specially-marked tests at runtime (again, with associated runtime cost) when a macro could just generate a constant list of those specially-marked declarations?

In other words, I'm suggesting that we should prefer static reflection (via macros or other compiler plug-ins) over runtime reflection, and given that we are just now developing the capabilities to analyse and generate code at compile-time, we should see how far we can push those new capabilities in order to understand the full scope of problems which cannot be served by anything other than integration with the language runtime.

Again I'm not sure that I follow. The proposed design involves calling an initialiser for every declaration annotated with a property, and constructing values on-demand:

the custom attribute application will turn into an initializer call on the attribute type, passing in the declaration value as the first argument.

I'm not sure which data you're suggesting could be emitted in a special section. The proposal also doesn't mention anything about that.

Moreover, the kind of metadata I'm talking about is functional - it is a part of the logic of the application. If we had macros generating, say, a static ORM mapping or test database, I think it's perfectly reasonable that that information goes with other static data in the binary's data segment. Why would it be necessary to separate data generated in this way from other static data?

1 Like
72

I meant "eager metadata initialization" more generally, e.g. initializing all Test(...) or Persisted(...) instances at the same time. This proposal allows for finer-grained queries in the future which may only initialize the subset of custom metadata values that are accessed.

Ultimately, I think weā€™re talking about the same basic idea. What I suggested here

is a constant initialization feature that generalizes this proposal. This proposal is a specific form of constant data, where the constant data is always a list of generator functions that construct instances of a type using init(attachedTo:). There are probably other use cases where the actual metadata is constant, and that value could be emitted directly rather than going through a generator function.

Emitting constant data from annotated declarations is exactly what this proposal is doing. The constant data is a list of pointers to generator functions that initialize the library-defined values. The constant data section is also organized by attribute type, so a query for a specific attribute type does not need to search through records from other attribute types.

The "runtime reflection" piece of this proposal refers to the fact that these values are accessible at runtime, e.g. a test runner program can retrieve these values at runtime. The list of annotated declarations is generated at compile time to be discovered efficiently at runtime.

The reason I'm suggesting a separate named section (still within static data) is because I think the data should be collected record-by-record, rather than in a literal array generated by a macro. The named section then acts as the "container" for the individual records.

More generally, macros are designed to expand into code that you can express yourself in regular Swift code. Macros deliberately are not equipped with extra magic that cannot be explained with regular Swift code. So, if we have a general constant initialization feature that allows you to annotate a constant-initialized variable to specify the name of a section it should be emitted in, which becomes an individual record in that section, a macro could then generate those declarations as peers to the annotated declaration. This peer-macro capability is already included in the design for attached macros.

2 Likes
73

I see. I'd like to suggest that running the generator functions is actually not a valuable feature (it may even be an anti-feature), and those functions really should not include expensive processing that makes it valuable to filter them with a predicate. In other words, I think limiting metadata to compile-time constants would be perfectly sufficient. But it's an important thing to clarify.

For example, I tried to use an attribute with a local, and the compiler crashed. I presume this won't be supported?

@runtimeMetadata
struct Attr {
  	var v: Int
	init<T, V>(attachedTo path: KeyPath<T, V>, _ v: Int) {
	  self.v = v
	}
}

func local(value: Int) {
    struct Test {
        @Attr(value)  // <-- 'value' is not a global. Impossible to reflect.
        var field = "Some field"
    }
}

And if we're expecting complex processing, that may include mutable state, meaning that the order in which we call these generator functions becomes an observable property of the system which should be defined.

Also, we'd need to consider actor isolation. My understanding is that eventually all globals must be isolated to a global actor. So if we're allowing data to be passed in, and it must be global, it will be isolated to some global actor and the attribute initializer will either have to be async or isolated to the same actor. That will also need to get propagated to the Attribute.allInstances function somehow, won't it?

Purely constant metadata doesn't have any of these questions. Macro authors could limit parameters to literals, or require a specific global actor, and that would probably cover 99% of real-world uses.

Sorry, I'm still not getting what you mean by this. Does this still apply if we don't have the generator functions at all?

74

Only if they're mutable or non-Sendable.

75

Iā€™d be very interested to see if we could use this for benchmark discovery too - seems it might solve the issue for our package to dynamically discover them nicely!

But what is the status of the pitch, pending review?

1 Like
76

How can someone know all the experimental features their compiler can handle :thinking:

77
78

I think it would be a good idea to have a page on the website listing the official features and what compiler they became available in. We haven't gotten around to that yet. However, I'm a lot less sure it's a good idea to keep a list of experimental features. They are, after all, experimental and totally subject to change before release, assuming they are ever released at all. Pavel's link is probably the best way to track that ā€” it being a link to the source code kind of implicitly gives you the right idea about whether you should be using them.

4 Likes
79

Thanks a lot @xedin

80

I'm very much in favor of this functionality, if not necessarily the proposal as written (I haven't read through it thoroughly enough to judge it one way or the other). But I have long wanted the ability to annotate types, methods, and properties, and then be able to enumerate those annotated types at runtime (to access properties, call methods, or instantiate types).

The use case foremost in my mind is server-side programming. The Java Spring Framework has a set of annotations (declared like interfaces in Java) for configuration a web server:

@Controller
@RequestMapping("/users")
public class MainController {
	@RequestMapping("current", method=GET)
	public String getCurrentUser(HttpServletRequest inReq, Model inModel) {
		User user = ā€¦;
		inModel.addAttribute("user", user);
		return "userPage";  // returns the name of the .jsp file to render
	}

	@RequestMapping(method=POST)
	public String login(HttpServletRequest inReq, Model inModel) {
		String login = inModel.get("login");
		String password = inModel.get("password");
		let user = // fetch user record from login & password
		inModel.addAttribute("user", user);
		return "userPage";  // returns the name of the .jsp file to render
	}
	ā‹®
}

At run time, Java uses its introspection capabilities to find all classes that are annotated with @Controller, instantiates one of each, and then looks for all methods with each of those annotated with @RequestMapping, and builds up a table of routes. In this case, it would use getCurrentUser() to handle GET requests to /users/current, and login() for POST requests to /users/login.

Over on the Vapor discord, we chatted about a way to do this with macros using what I called ā€œmarkerā€ macros. Someone else had the same idea a couple days earlier and has fairly complete working example. But it generates the route registration code at build time (not unreasonable), not at run time.

Also, I prefer the spelling @annotation to @reflectionMetadata.