Hm, I think this is going a bit too far in mimicking actual reflection (also true of CaseIterable to be honest). We all agree that a real reflection API is needed (to replace Mirror), and it would almost certainly swallow up these protocols; so why add another ad-hoc thing?
Put another way - I suppose conformance (or not) to StoredPropertyIterable would become meaningless once we have reflection. It will also be possible to inspect the stored properties of types which don't declare conformance, and this protocol would be part of the language that never gets used again.
I love that this proposal is being discussed. But I agree with @Karl that we may need a Reflection Manifesto to get a general idea where we want to go before with reflection before evaluating proposals like this one.
For example, if we end up having runtime access to custom attributes, it would make sense for them to be available on the reflection API of properties, and I’m not sure how that would interact with this proposal.
It is not entirely true that this protocol's functionality will be obsolete. This protocol does not just provide what reflection would provide, it also serves as a customization point. Users can override the default allKeyPaths property and expose their own key paths if they choose to. Of course, we need a use case for that (there are use cases in ML which I can follow up on later).
That said, I agree that it would be nice to see a reflection manifesto first. I'm not in a hurry pushing this :)
Another interesting aspect of Mirror is that it allows for both indexed and key-value interfaces; you can iterate through the child nodes of the mirrored structure sequentially or also call them up by name, for things like structs or dictionaries where that makes sense. Maybe these key path-based protocols could also have API for getting a key path by name, and maybe returning the set of all valid names.
The end goal, as I see it, is to have runtime access to type metadata, property metadata, event custom attributes, which seem highly desirable. In that scenario, this protocol would simply allow for a type to "customise" its stored properties, similar to CustomReflectable for Mirror. In that case, shouldn't we pre-entively name this protocol CustomPropertyIterable?
There is value to explicitly conforming to a protocol like this, as a signal to the compiler and to the reader that the dynamic metadata for the type is actively being used.
Without considering how it fits into the larger reflection story, this feature is particularly interesting to me because it enables somewhat-automatic struct-of-arrays transposition of types. Currently, I manually apply this to a few types in my code, and a proposal like this would simplify things greatly.
I could imagine a hypothetical array type (e.g. ispc's soa) that deconstructs structures when they're added and then allows access to individual components by key-path:
struct Point : KeyPathIterable {
let x : Float
let y : Float
}
let array : StructOfArrays<Point> = [Point(x: 0.5, y: 1.0), Point(x: 2.0, y: 0.5)] // internally stored as Point.x, Point.x, Point.y, Point.y
print(array[0, \.x]) // 0.5
That would also require some sort of KeyPathConstructible protocol to init types out of the array (or be restricted to POD types), but it's an interesting possibility.
In that context, I think recursivelyAllKeyPaths might usefully take a depth parameter so you can specify whether e.g. StructOfArrays<Line> where Line is { var start: Point; var end: Point } should be deconstructed into two Points or four Floats.
Similar to what @Joe_Groff said, I would rather see this implemented as a new feature to complement/replace Mirror. The protocol could exist to allow users to customize this behavior, but I'm opposed to adding the ability to the type checker to generate code for iterating over stored properties when equivalent metadata is already emitted elsewhere in IRGen.
We could even re-implement mirror in terms of this protocol, removing the special runtime support we have for mirror.
I think @Slava_Pestov was saying that this protocol should exist to customise stored properties, the same way CustomReflectable allows types to customise Mirror reflection. That way you can use another API (ie, Mirror or a new more strongly typed reflection API) to access those properties. You'll be able to get all stored properties of all types, and types can customise them with this protocol. That's why I think it would be named CustomPropertyIterable.
I'm not sure I agree that customisation makes sense for something like this (pretending that certain properties are stored/not stored). I don't think it generalises.
In the general case, if somebody wants a subset of my type's properties, they need to ask a more-specific question rather than relying on me implicitly understanding what they actually meant to ask. There are lots of ways to model this:
Some kind of MLUpdatable protocol which accepts some weights and lets the type decide how it updates itself.
Some kind of custom attribute, which you can use to filter the properties you discover via reflection.
I think customization makes sense as a way to allow types to describe their logical schema, independent of their underlying implementation. For the core collections like String, Array, and Dictionary, there's little value to accessing their physical properties, and for most purposes you want to think of them as being composed of their keyed or indexed elements. Likewise, a struct could have redundant fields used internally for caching, or do manual size optimization to pack logically independent data into a single field, and allowing the struct to customize the schema it reflects could hide those details from reflective interfaces. This seems to me, if anything, more composable, because you could then take advantage of automatic compiler synthesis and/or default implementations built on top of StoredPropertyIterable that derive ==, hash, coding, etc. from the logical schema, instead of being consigned to manually reimplementing all of those things.
I'm excited that @rxwei and @dan-zheng are working on this highly-requested feature.
To @Slava_Pestov's point about having the feature to complement / replace Mirror, I wonder if we have a "low-level" feature that's about describing the stored properties via a collection of key paths, and a "high-level" feature that's about customizing the conceptual view of the properties. For the low-level feature, I've been imagining it as API on MemoryLayout to really drive home that it's low-level:
extension MemoryLayout {
static var storedProperties: _StoredPropertiesCollection<T>?
}
This information is what's available in the metadata. It would allow us to reimplement much of Mirror in Swift (rather than the C++ code in the runtime), and enable lots of cool experimentation in the area of runtime reflection.
The "higher-level" API would probably be more like KeyPathIterable, and would provide a customization point for instance-specific keypaths (like collections) or types that want to provide a different set of type-level key paths. Reflection-ish APIs like Mirror, which is meant to show a user-centric rather than a machine-centric view of the type, would check for KeyPathIterable conformance first, falling back to MemoryLayout.storedProperties to handle the general case.
As far as "opt-in reflection", I've always been in favor of that so that people can choose between secrecy and flexibility. Joe's convinced me (in the past) that a reasonable compromise position would be "types but no names" in the metadata, with debug info and/or opt-in features providing the names.
To be clear I wasn't advocating a redesign of the interface of this feature to look like Mirror.
I like the new protocol and API as well, my point is that instead of synthesizing an implementation in the type checker, a global default could be provided in a protocol extension that looks at reflection metadata. This will eliminate the type checker component of this proposal, making it a purely additive standard library/runtime change.
"Customizable key path schemas" is an idea that's worthy of exploration.
But a high-level API like KeyPathIterable is better implemented using runtime metadata instead of type-checker code synthesis.
We should explore the relationship with language reflection, including existing APIs like Mirror.
@Douglas_Gregor sketched out the design for a low-level reflection feature:
My interpretation of @Douglas_Gregor's design is: why don't we directly expose a (more-or-less) direct bridge from C++ metadata (FieldType as mentioned, and maybe other stuff) in Swift?
As I understand, Mirror is a direct user of C++ runtime metadata (see "How Mirror Works"). But if there was a low-level metadata API in Swift, then Mirror and custom reflection APIs like KeyPathIterable could be implemented using it.
I wonder if this direction makes sense? Some questions:
What (how much) runtime metadata would be useful to expose in Swift? Exposing FieldType via something like MemoryLayout<T>.storedProperties seems to enable Mirror and KeyPathIterable.
Is the scope of this direction appropriate? Is "exposing runtime metadata in Swift" too broad as a goal? Or maybe the direction could be even broader?
I'm sure runtime experts have thought about this direction before, feedback would be appreciated!
Re-focusing the discussion on KeyPathIterable: I feel that KeyPathIterable is useful as a standalone language feature and may be suitable for evolution!
An evolution proposal for KeyPathIterable needn't be blocked by a fully-general runtime metadata bridge in Swift. A short path to readying KeyPathIterable for evolution would be to implement var allKeyPaths as a protocol extension using runtime metadata instead of type-checker code synthesis (as people have advised in this thread). Then, if a general runtime metadata bridge is later added, var allKeyPaths (and Mirror) could be reimplemented using that bridge.