[Pitch] Reflection

Evolution Pitches
36

I'm really excited about more introspection capabilities in the language, but after reading the proposal, I think it's missing an important feature, and I don't quite understand another.

What I really want to do is be able to find all types in a module (or linked into the app) that conform to a given protocol, and then instantiate instances of those. Example (forgive me playing fast and loose with Swift syntax):

protocol ImageFilterPlugin {
	init(param1: Float, param2: Float)
    func process(image: Image) -> Image
}

…

struct MyImageFilterPlugin : ImageFilterPlugin {
    
    init(param1: Float, param2: Float) {
    }
    
    func process(image: Image) -> Image {
        <process and return an image>
    }
}

…

func
instantiateImageFilter(filterNamed: String)
	-> ImageFilterPlugin
{
	let filterType = Reflection.type<ImageFilterPlugin>(named: "MyImageFilterPlugin")
	let filter = filterType(param1: x, param2: y)
	return filter
}

func
listImageFilters()
{
	for filterType in Reflection.types(conformingTo: ImageFilterType.self) {
		print("Filter: \(filterType.name)")
	}
}

It would be even better if I could iterate over all the methods of a type, find the initializers, and choose one to call. Even better would be a way to annotate things to make them findable (the way Java does). I could annotate any type, property, method, function parameter, and act based on those annotations (search for, construct a call to, etc.).

The thing I'm not clear on in your examples: They show (e.g.) instantiating Types, but it's not clear to me how to instantiate an instance of that type.

8 Likes
37

Overall nice first step, great to see this!

I'd personally want to see a "vision" going along with this initial part, because the long term here is alluded to, but not really explored much. I'm a bit worried about offering such "partial" API, missing features to discover methods etc, even if just for registering them for later execution etc.

There is also overlap between reflection APIs and macros "compile time reflection", and it would be good to explore this overlap if we can at least provide similar (or shared!) APIs -- it would be a bit annoying if the runtime reflection API shape ends up completely different from the compile time one.

Specifically, I'm looking at this having worked with Scala reflection and macros before; where the two APIs were attempting to surface some common interfaces. Scala 2's reflect API offered an abstract API scala.reflect.api what had a subset of things that are accessible both from a macro "universe" as well as a runtime "universe". This may or may not be desirable for Swift, but it might be worth an invesigation -- especially as we're developing macros and reflection right now at the same time.

A particularly interesting aspect of macros is that they are based on the same API used also for Scala’s runtime reflection, provided in package scala.reflect.api . This enables the sharing of generic code between macros and implementations that utilize runtime reflection.

Similarly, Scala 3 also allows macros to access to the full reflection API from the macro runtime via the quotes.reflect module: Reflection | Macros in Scala 3 | Scala Documentation

I bring this up here now, since the initial expression macros pitch explicitly didn't cover any Type interactions yet, so there is room to make sure we make the best possible API happen.

For example, would it make sense to be able to get a Reflection.Type from a MacroEvaluationContext, by querying the compiler to resolve a given AST nodes type? Access to types will most definitely be necessary in more advanced macros, so it'd be interested to see if we can marry the two rather than each have their own almost-the-same "Type".

18 Likes
38

Yeah, this is a great point. We're effectively exposing Swift's type system as an API for users, and it would be best if there could be a single type API that works for runtime reflection, macros, and Swift-based tools.

However, what would this mean for our design? Does the representation of a type need to be something that can be used with several different sources of information at once, i.e., can you get a type from reflection information (say, from a T.self in the running program) and pass that type to some static reflection facility (say, that looks at the binary for a library that hasn't been loaded) or ask the compiler for more information about it from the corresponding source code?

I think the best way to come at this problem is to separately design the type representation we'd want for macros and/or a compiler (I assume they'd be the same, but one never knows), and then do some compare/contrast to see what unification would look like. I suppose that's on me to peek further down the road of providing type information to macros.

Doug

17 Likes
39

I agree partial type is maybe not the best name for this thing. I think type constructor would be a good name if that's the only thing this did (which is pretty much the only thing it does right now). I imagine in the future if we wanted to inspect "generic requirements of a generic type", then this would probably happen on partial types and wouldn't make sense with the name type constructor. Generic type is an interesting name because it fits nicely with the use case above, but it gets a little weird because the type doesn't necessarily need to be generic to have a "GenericType" (e.g. Int has a "GenericType"). That being said, I'm all ears for a better name :slightly_smiling_face:

Keypath introspection is definitely a much requested featured and I imagine we'll eventually want to be able to do that (either directly on keypaths, or some new facility in this Reflection module). I've held off on designing anything that space so far and only focused on what's being proposed.

The second library on that list happens to be one I made a couple of years ago :slightly_smiling_face: That library is intended to be a low level interface to the Swift runtime which allows for higher level APIs like the ones being proposed. The first library has some similarities (Property vs Field), but it looks like it was mostly used to get/set properties via string name (which one can do here by comparing Field.name and using Field.keyPath) and construct instances of types at runtime (using unsafe bits that doesn't always work).

-disable-reflection-metadata removes the reflective capabilities of Field and Case, so every other API available on Type and PartialType would still work because it uses information required by the runtime (not strippable).

Yeah, Field.offset should probably return nil in that case. We can for sure add all of the MemoryLayout APIs and more onto Reflection.Type, but it does feel redundant because you can get the same info from 2 separate facilities.

Yeah, those placeholder types are probably better to use than [Int: Int]. Thanks! Having the placeholder in the textual representation is also a nice improvement here. I made the textual representation a separate API under name because I was unsure how common it is to access a type's description, but if people think that's a better place to put this, then by all means :slightly_smiling_face: I imagine description would look like Type<_, _> and debugDescription could look like Module.Outer<_>.Type<_, _>?

I included all of the @usableFromInline, @inlinable, and @frozen as part of the detailed design because I wanted evolution to clearly see what we're committing to as ABI. As you can see, Type stores a value of type Metadata which is why I didn't name it as such (that API will be proposed at a later date).

5 Likes
40

I would like to be able to ask a Type if it is a class, struct, enum, etc. It looks like the currently proposed module would not allow this.

2 Likes
41

Excited for this! Getting KeyPaths for fields is important for our codebase, and we're currently relying on _forEachFieldWithKeyPath via @_spi(Reflection) import Swift, which is a little unfortunate. Further in the past, we did it using Mirror, which was at least equally unfortunate.
It would be great to have a proper API for this functionality!

We do need the KeyPaths to be Writable however, which isn't included in the original pitch.

But from the following I gather it would still be possible to make the KeyPaths from Field writable? Perhaps with a cast? Am I reading this correctly?

2 Likes
42

Without distracting too much from the current pitch, iterating through the constituents of a KeyPath is very interesting functionality to us as well.
Seems like it may also contribute towards enabling keypath codability?

1 Like
43

This is very possible! Do you think isClass, isStruct, etc. accessors are the way to go, or would you prefer something like type.kind == .class?

As of right now, I don't have plans to make an API that vends a mutable key path to a field just yet. I think I still need some more designing there to come up with a good solution that doesn't allow folks to ask for mutable key paths to properties they can't access.

44

Great! I think, an enum is the cleaner solution. It could also get properties such as isValueType so that you can ask type.kind.isValueType.

45

Probably the latter, because it's more ammendable to switch statements and exhaustiveness checking, as opposed to if/else if/else ladders checking every kind of isX property

3 Likes
46

That could be a bit deceiving. I think people will unknowingly call that when what they're looking for is hasValueSemantics, which is something that is actually pretty tough to nail down, certainly in any automated capacity.

(Instances of classes that have CoW can have value semantics, and instances of structs that contain references to objects can have reference semantics, so it's really not all that straight forward.)

1 Like
47

Maybe you're right.

48

Just finally getting to catching up on this. Like others, I'm very excited about the possibilities, and this looks like a great start!

I do hope we'll have an opportunity to iterate on the API design aspect a bit. At first blush I was concerned that the initial design of Type doesn't use a label to distinguish between initializing from a value or from a type, which is something that we took care to do with other APIs (such as MemoryLayout.stride, etc.). I do worry about whether confusion could arise when the value is a type.

Supposing we were to (say) distinguish Type(Foo.self) from Type(of: fooInstance), a question arises which actually is present even with the API as pitched: users might rightly be confused about the distinction between type(of:) and Type(...)—and more fundamentally, as you mention later, the distinction between a type and a Type.

While I empathize with not just sticking the API directly on Any.Type and possibly even all concrete types, I don't think that's the only alternative to inventing a top-level Type that is distinct from the actual type:

Have you considered a design where, for example, we have a reflected property on types as an "umbrella" for reflection (in the same vein that we have lazy in the language for lazy sequences)? That is to say:

for field in Dog.reflected.fields { ... }

// ...as opposed to:
for field in Type(Dog).fields { ... }
// `Type(Dog)` isn't `Dog.Type` or `Dog`...
// That's a lot of different "types" to understand!

// ...or:
for field in Dog.self.fields { ... }
// Now we avoid creating yet another `Type`, but either we make
// `self` even more "magical" or we pollute code completion for
// the concrete type.

This would moreover provide a nice symmetry where Dog.self gives you the type and Dog.reflected gives you the "reflected type." From a value, one would then write type(of: dog).reflected.

Other than that design question I would just echo previous commenters regarding the "partial type" terminology.

I'd also point out that while Field is nice and short, I believe the user-facing terminology (in TSPL, say; and indeed in the just-pitched declaration macros document) has called it a "stored property," UI code would use "field" to mean something very different, and I think there is value in trying to maintain consistency here throughout our documentation and APis.

Anyway, bravo on the overall pitch—very exciting that it exposes new possibilities for the language in an exciting way.

13 Likes
49

It sounds like your concern is not necessarily with the name Type, but rather the method in how one accesses a value of this type? X.reflected is an interesting approach, but would require compiler support (which is not a deal breaker). Is there a library approach we could take to solve this a different way? Could there be a global function in Reflection that could achieve this, similar to type(of:)? E.g:

for field in reflect(Dog.self).fields {}

for field in reflect(type(of: dog)).fields {}

Does X.reflected return a value of type Type? or is the following disallowed:

let x = Dog.reflected

Yeah, it seems lots of people agree this is not the best name for this thing. A few suggested GenericType which is really close in my opinion, but doesn't make a whole lot of sense for types like Int or String who have GenericTypes but aren't generic.

Would you prefer this to be called Property?

for prop in reflect(Dog.self).properties {}

This makes sense to me, but might be confusing as it doesn't return computed properties right now. Maybe in the future if we add reflective information for computed properties we just spell that out explicitly?

for cprop in reflect(Dog.self).computedProperties {}
1 Like
50

These are duals of the same problem.

As Ben Cohen pointed out in a different design discussion, Swift has (a large number of) technically public type names that are not in practice user-facing. For example, most users will never write PrefixUpTo, UnfoldFirstSequence, etc., but will instead get instances of such types by using much more common affordances such as the range operators, sequence(first:next:), etc.

With the API design that you propose, the name Type has big shoes to fill because it is very user-facing. So on the one hand, it would be very unwieldy to name that type something like ReflectedType or TypeReflection, because every use site would then be littered with extraneous information, but on the other hand, to keep the name succinct we’re left with something that is very much “overloaded” with meaning and therefore (pun intended) generic and in certain contexts inscrutable.

This is why I point to lazy and other accessors as a model here: the underlying lazy sequence types themselves have pretty specific names and aren’t all crammed into a type called Lazy. Adopting that pattern would allow you to have a contextually succinct user-facing spelling (e.g.: Dog.reflected) without constraining what you actually name the type (for example, you could rename Type—not that you should—to SwiftNextGenerationAwesomeSuperUsefulReflectionOfType without affecting user ergonomics in most uses).

I don’t see a reason to disallow it, but as I touch on above, it probably shouldn’t be named Type, which is confusable with the metatype, etc. Because the appearance of the type name will be significantly demoted and for many people appear only in diagnostics or debugging info, it can be given a more descriptive name without impacting the user experience.

I believe in the macro pitch it’s called storedProperties: I’d love for whatever terminology we choose to be preserved verbatim where possible between macros and reflection APIs.

8 Likes
51

Hi! This is looking great! Our use case might look like the following:

public protocol NewKeyPathIterable{}
public extension NewKeyPathIterable // Just so `Self` is concrete.
{
    var keyPaths: [PartialKeyPath<Self>]
    {
        var out = [PartialKeyPath<Self>]()
        guard #available(macOS 9999, *) else
        {
            print(" `guard #available(macOS 9999, *)` condition did not succeed.")
            return out
        }
        out.append(contentsOf: Type(Self.self).fields.compactMap{$0.keyPath as? PartialKeyPath<Self>})
        return out
    }
}

struct Restaurant: NewKeyPathIterable
{
    var burgerCost: Float
    var pieCost: Float
    var sodaCost: Float
}

let r = Restaurant(burgerCost: 4.99, pieCost: 1.99, sodaCost: 0.99)
for keypath in r.keyPaths // Optionally, get member names and iterate over them as well.
{
    print("Cost:",r[keyPath: keypath]) // Prints the cost of each item in the instance.
}

At the moment it doesn't look like we can get the KeyPath for a struct within a struct. For example, if we place an inner struct within Restaurant:

struct Sodas: NewKeyPathIterable
{
    var colaCost: Float
    var orangeCost: Float
    var lemonLimeCost: Float
}
struct Restaurant: NewKeyPathIterable
{
    var burgerCost: Float
    var pieCost: Float
    var sodas: Sodas
}

let r = Restaurant(burgerCost: 4.99, pieCost: 1.99, sodas: Sodas(colaCost: 0.99, orangeCost: 0.79, lemonLimeCost: 0.89))

then we get an EXC_BAD_ACCESS when iterating through the KeyPaths and attempting to access $0.keyPath in the extension above. This also appears to happen when we change Restaurant and Sodas into classes. I'm hoping that access to KeyPaths for structs-within-structs (and similar) doesn't get lost during the pitch/development phase!

6 Likes
52

On second thought (you could say I'm reflecting back on this :smiley:), if we're going to have users writing switch statements to discriminate apart various kinds of Type apart, I think we may as well have Type as a protocol, and have concrete types.

I.e. replace:

// I highly recommend the "mirror" terminology, to clear up variable names like this
let typeObject = Type(something)
switch typeObject.kind {
case .class: ...
case .struct: ...
case .function: ...
...
}

With:

switch Type(something) {
case let classObject as ClassType: ...
case let structObject as StructType: ...
case let functionObject as FunctionType: ...
...
}

That gives you non-nilable access to the fields that are pertinent to each case.

2 Likes
53

Maybe I overlooked it, but shouldn't (or can't) Type have a name property as well?
Whereby name returns the name of a struct or class as written in code. Similar to the fields of the type.
It would be nice and obvious to have, no?

Otherwise you have to go through a String initialiser (assuming this is still the current way to do it).

extension Type
{
	var name: String { String(describing: self.swiftType) }
}
1 Like
54

The type system representation proposed here recently came up in the review of SE-0385:

I agree that we should settle on one family of types to represent the type system that can be used throughout various language features, including the Reflection APIs proposed here, custom reflection metadata as proposed in SE-0385, and possibly other areas of the language that need to express requirements on declarations in Swift source code.

For example, I'm working on the design and implementation for attached macros, which would also benefit from a way to specify requirements on the "attached to" declaration, very similar to what's possible in SE-0385. Consider the @Persisted property wrapper from the Realm package. It may be possible to express this API as a macro that generates accessors directly, rather than using a property wrapper:

@attached(accessor) macro Persisted = #externalMacro(...)

class Dog {
  @Persisted var name: String
  @Persisted var age: Int
}

The @Persisted property wrapper today requires the wrapped value to conform to _Persistable. There's currently no way to express this requirement in an attached macro declaration. We could invent an ad-hoc representation of the various kinds of declarations, mirroring what SE-0385 did with the attachedTo: initializer parameter, but I think it would be much better to have a consistent representation across these various features.

9 Likes
55

One other thing that would be nice is if the Type struct were Codable so that we can encode/decode metatype information (e.g. Int.self, Double.self, Float.self, etc).
As a first pass, it may be useful to focus on Foundation primitives, and then handle Codability for custom types.
Solutions proposed in the past, such as Making a codable wrapper for metatypes, will I get into trouble by doing this?, rely on NSStringFromClass and NSClassFromString functions which have not been touched since 2016 and which do not work properly on Linux at the moment.

4 Likes