Pitch #3: Opt-in Reflection metadata

Evolution Pitches
22

I feel like this pitch is missing a few steps: "what metadata does Swift emit today, what purpose is it used for, what can we omit tomorrow, and what do we have to keep?"

I share some of the concerns of people who are relying on printed representations today, particularly around enums, but I think gating on Swift 6 probably makes that safe enough. Part of updating to Swift 6 will be remembering to add Reflectable where it matters. The one place where that wouldn't be good enough is if a client is relying on a library's type's printed representation, but it's not something the library wanted to commit to.

I want to suggest that if full debugging is enabled (i.e. not "line tables only") the full metadata should be emitted in a debug section no matter what mode you're in (EDIT: unless you're in the "on" mode and it's already in the binary). That way, it generally won't be shipped with the binary, but you don't lose any debugging capabilities you would have otherwise had. That might also simplify the flag story: it could just be "off", "opt-in", "on".

16 Likes
23

Thank you all for the feedback folks!

I think I have a plan in mind for how safety issues can be addressed:

  1. Deprecate the existing API that might be using reflection metadata in release builds with @available(swift, deprecated: 5.8, message: "Argument should conform to Reflectable") but keep it around for compatibility with apps built with older stdlib versions.

  2. Introduce the overloaded variance of that API with a generic requirement on conformance to Reflectable protocol.

  3. The compiler will autosynthesize conformance to Reflectable for all types, if reflection metadata mode is "Fully available".

  4. Allow force cast to Reflectable as! Reflectable to silence the warning. (Reflection metadata is still not available)

In that case, during migration to Swift 6, if such an API is used, a developer will get a compile-time warning pointing out the need to add the conformance to Reflectable if the developer wants reflection capabilities.

A developer will be able to silence the warning by force-casting as! Reflectable or by enabling reflection metadata in full.

3 Likes
24

I wonder if the polarity is right here. Maybe it would make sense if:

  • All nominal types default to a lazy internal implicit conformance to Reflectable. The type conforms to Reflectable, but only if something in the module requires that conformance. Outside the module, it is not known to conform.
  • The implicit conformance can be explicitly disabled.
  • The implicit conformance can also be explicitly declared, allowing it to be public.
4 Likes
25

I can imagine wanting to set the default for a whole module, but I definitely don't want to have to track down every type in a module that values code size or secrecy and ensure that I've written !Reflectable or whatever. I do see that we're in a unique situation because at least basic reflection capabilities are currently provided to every type by default, though.

2 Likes
27

All nominal types default to a lazy internal implicit conformance to Reflectable

Would the compiler be able to statically determine all such cases to add implicit conformance?

Apart from these, it doesn't seem too different from what is being proposed,
However, the distinction between internal and public conformances might be more complicated to comprehend.
Jordan's point also makes a lot of sense because disabling might not be a good solution, and we discussed in the beginning that it should be an opt-in rather than an opt-out mechanism to control the emission of reflection symbols.

28

We updated the proposal doc and the implementation. A few key points:

  • Introduced Reflectable Casts (as! Reflectable, as? Reflectable, is Reflectable)
  • Opt-in mode is set by default starting from Swift 6.
  • Synthesized conformance to Reflectable to all declarations if reflection metadata is enabled in full.
  • Implicit conversion to Reflectable is forbidden.
  • Better diagnostics.

Proposal - [Proposal] Opt-In Reflection metadata by maxovtsin ¡ Pull Request #1203 ¡ apple/swift-evolution ¡ GitHub
Implementation - Swift Opt-In Reflection metadata by maxovtsin ¡ Pull Request #34199 ¡ apple/swift ¡ GitHub

1 Like
29

I'm glad to see continued progress here, but my previous feedback still stands:

  • What metadata does Swift 5 currently include for all types that Swift 6 will make opt-in? The existing flags are not documented and not very well known, so this is knowledge you can't expect reviewers to have. (In particular, I would hope that the names of non-Reflectable types would not appear in the final binary, and this proposal does not tell me if that is the case.)

  • What stdlib APIs (and perhaps Foundation APIs, as part of corelibs) will behave differently on non-Reflectable types? In particular, if a type is used with NSCoding, even as a generic parameter, it must be findable by name later, which would be a migration hazard going from Swift 5 to Swift 6 that could result in the loss of user data.

  • What happens when I compile in release mode with full debug info? Does my debugging experience suffer? (more than it does today)

I also think not supporting the Reflectable casts on older OSs makes this tricky to adopt on the consumer side, but maybe it's okay because you're not proposing to add bounds to existing stdlib APIs, which will have some sensible fallback (like "no children") for non-Reflectable types on new and old OSs. Still, I know our dynamic cast system is hookable, and it may be that on older OSs you can use something like "has no name" as a proxy for non-Reflectable.

A thought I've just had now: what reflection metadata is generated for imported types? Do we have any hope of controlling that?

9 Likes
30

Apologies for not addressing these concerns earlier.
Let me try to do that now, and if the explanation makes sense I'll update the proposal accordingly.

What metadata does Swift 5 currently include for all types that Swift 6 will make opt-in? The existing flags are not documented and not very well known, so this is knowledge you can't expect reviewers to have.

I perhaps need to add more details to the proposal explaining different kinds of metadata and what information reflection metadata contains.
But in general, there are two levels of metadata:

  1. Core metadata, such as the type metadata record, nominal type descriptor, etc.
  2. Reflection metadata which contains information about fields' types and their names. (Data from swift5_fieldmd section of a binary)

Core metadata will be emitted in full and not affected by this proposal, while Reflection metadata will be emitted only for types that conform to Reflectable protocol or for debug builds.

(In particular, I would hope that the names of non-Reflectable types would not appear in the final binary, and this proposal does not tell me if that is the case.)

Type names are kept in "nominal type descriptor" which isn't a part of this proposal.

What stdlib APIs (and perhaps Foundation APIs, as part of corelibs) will behave differently on non-Reflectable types? In particular, if a type is used with NSCoding, even as a generic parameter, it must be findable by name later, which would be a migration hazard going from Swift 5 to Swift 6 that could result in the loss of user data.

We decided not to include changes in stdlib in the proposal, because all current API that consumes reflection are kinda for debug purposes and developers shouldn't rely on the output of those APIs. (But might be proposed separately)
Foundation APIs, as far as I am aware, it doesn't know about Swift's reflection and won't be affected.

What happens when I compile in release mode with full debug info? Does my debugging experience suffer? (more than it does today)

By full debug info did you mean an arg from -g family?
I didn't consider that case, and probably it makes sense to keep reflection emission enabled for at least ASTTypes and DwarfTypes debug options.

I also think not supporting the Reflectable casts on older OSs makes this tricky to adopt on the consumer side, but maybe it's okay because you're not proposing to add bounds to existing stdlib APIs, which will have some sensible fallback (like "no children") for non-Reflectable types on new and old OSs. Still, I know our dynamic cast system is hookable, and it may be that on older OSs you can use something like "has no name" as a proxy for non-Reflectable.

We considered backporting the Reflectable casts, but wouldn't want to introduce a compatibility library only for that case. I also don't think this feature will be critical since not many libraries consume reflection nowadays.

A thought I've just had now: what reflection metadata is generated for imported types? Do we have any hope of controlling that?

No, that topic was raised already in the thread and it doesn't seem reasonable to generate reflection for imported types.
I'll mention explicitly in the proposal that conformance to Reflectable is allowed only at the type declaration level, not at the extension level.

3 Likes
31

Thanks for clarifying! The motivation to remove type names is for secrecy reasons, so that someone can’t search the binary for e.g. “PasswordValidationState” to find out how that state is represented, or “ShinyNewFeatureConfig” to confirm that an app developer is working on a new feature. Maybe that’s just more “walls and ladders” obfuscation, but it still seems relevant.

It sounds like this proposal is planning to change the behavior for field metadata only. I don’t know if we’ll get other types of reflection in the future (invoking methods, listing computed properties along with stored ones like ObjC does, etc), but I think calling the capability (and the protocol) Reflectable makes sense. The proposal, however, would feel a lot more approachable if it were in terms of field metadata, with other kinds of reflection mentioned in Future Directions.

So what other reflection do we have today?

  • Custom mirrors. This is opt-in already.
  • Getting the name of a type. This is currently supported by all types, and it sounds like that won’t change for the time being.
  • Looking up a type by name. Ditto.
  • Getting the name of an enum case. Will this be affected by this proposal?
  • Dynamic casts. A big enough deal that they deserve their own proposal, and it makes sense that they’re not covered by Reflectable.
  • probably anything else that’s in a custom section, if we haven’t hit them all already

Why is this list important? For any sort of resource-constrained environments where we’d like to have no metadata at all if it’s never used, but where it’s really hard to prove that. (For instance, if there are no dynamic casts to a protocol type, then the conformance metadata for that protocol only needs to be present if it’s actually used, after optimizations.)

I’m not saying we need a switch for every single thing, and of course I hope our optimization continues to improve. But this is why I’m pressing on this: the current API surface of Mirror does not represent everything the runtime does with “metadata” that could be considered “reflection”.

1 Like
32

The motivation to remove type names is for secrecy reasons, so that someone can’t search the binary for e.g. “PasswordValidationState” to find out how that state is represented, or “ShinyNewFeatureConfig” to confirm that an app developer is working on a new feature. Maybe that’s just more “walls and ladders” obfuscation, but it still seems relevant.

Secrecy isn't our primary goal, even though it might be improved. Hiding/removing strings from binary might be pretty challenging since many features depend on it and will require changes in Core Metadata which isn't a part of this proposal.

It sounds like this proposal is planning to change the behavior for field metadata only. I don’t know if we’ll get other types of reflection in the future (invoking methods, listing computed properties along with stored ones like ObjC does, etc), but I think calling the capability (and the protocol) Reflectable makes sense. The proposal, however, would feel a lot more approachable if it were in terms of field metadata, with other kinds of reflection mentioned in Future Directions.

This is a good point, the proposal currently affects only field metadata. I will emphasize that in the document and add in the "Future Directions" section that all reflection metadata added in the future might also be covered by the proposal.

Why is this list important? For any sort of resource-constrained environments where we’d like to have no metadata at all if it’s never used, but where it’s really hard to prove that. (For instance, if there are no dynamic casts to a protocol type, then the conformance metadata for that protocol only needs to be present if it’s actually used, after optimizations.)

I’m not saying we need a switch for every single thing, and of course I hope our optimization continues to improve. But this is why I’m pressing on this: the current API surface of Mirror does not represent everything the runtime does with “metadata” that could be considered “reflection”.

All emitted metadata might be considered as Reflection, but we try to distinguish between required Core and optional Reflection Metadata. Some metadata you mentioned in the list we consider as Core metadata and to handle it, the compiler needs to use a different approach rather not emitting it. There are a bunch of optimizations to improve dead-stripability of such metadata if provenly not used.
To limit the scope of this proposal we concentrated only on Reflection metadata.

1 Like
33

Realm Swift currently does the following thing: we call objc_copyClassList(), filter the classes to ones inheriting from a base class we define, then use Mirror(reflecting:) to read the property names and types for each of those subclasses.

It appears that with this proposal what we're doing almost would still work with no changes for our users, and no longer require that users build their app with full reflection metadata enabled. The problem is that while subclasses inherit reflectability, it sounds like we won't be able to mark our base class as Reflectable due to it being defined in obj-c (and has to be to work around FB7201126), and thus we can only declare protocol conformances in extensions.

We could require users to explicitly mark each of their subclasses as Reflectable, but that's clunky and error prone, especially if we can't check at runtime for that specifically due casts not being backdeployable (which otherwise doesn't sound like a problem for us).

34

I don't see an easy workaround for your use case. Would you be able to expose a Swift class that would inherit from your ObjC base class and conform to Reflectable at the same time?

(Since RealmSwift is consumed by Swift code, it should be fine, but probably would break source compatibility)

2 Likes
35

The base class used to be defined in Swift, but because of FB7201126 we need it to not be a resilient type even when our Swift library is compiled in evolution mode. Defining it in obj-c instead was the only way I could find to do that.

36

Can you attach a link to that bug? (I assumed that it was an openradar issue, but it wasn't)

37

Ah, sorry, I just assumed you were an Apple employee. Classes with resilient base classes have to defer their metadata initialization to runtime due to that the layout of the base class may be different than what it was at compile time. This initialization is done lazily when the class is first used. The bug was that objc_copyClassList() is supposed to eagerly perform all pending lazy initialization, but failed to do so for this newly added case and so it didn't include any classes with resilient base classes which have not yet been instantiated in the current process. This was fixed in iOS 14.5, but given our current deployment target of iOS 11 we're still a few years away from being able to require that.

38

While working on adding Sendable annotations to things I remembered that __attribute__((swift_attr("@Sendable"))) is a thing, and adding __attribute__((swift_attr("Reflectable"))) is the obvious way to solve the problem of not being able to mark an obj-c class as Reflectable.

39

I believe this feature might be added later in case the proposal is accepted.