[Pitch] Reflection

Could you remove the manual line breaks? :sweat_smile: I read These on my phone

5 Likes

OTOH reading a post with manual 80-column wrapping on mobile does give it a bit of a poetic feeling :slight_smile:

5 Likes

Reflection.Type can be initialized from either a metatype or an instance, but will the latter ignore a CustomReflectable conformance? Should the initializers be failable, in case the -disable-reflection-metadata option has been used? Or could they have any Reflectable.Type and any Reflectable parameters, using the recently-pitched marker protocol?

Like the MemoryLayout.offset(of:) method, should the Field.offset property fail by returning nil rather than zero? Should the other MemoryLayout APIs (size, stride, alignment) be added to Reflection.Type?

Would [AnyHashable: Any] or [Never: Never] be more appropriate than [Int: Int] in the PartialType examples? Should the textual representation also include placeholders (e.g. "Dictionary<_, _>")? Instead of the name properties, could there be description and debugDescription properties, for unqualified and qualified names?

To avoid confusion between metatypes and metadata, I suggest renaming:

  • Type to Metadata
  • Type.swiftType to Metadata.type
  • Field.type to Field.metadata
  • Case.payloadType to Case.metadata

The @usableFromInline internal APIs should probably be removed from the detailed design, but the index and tag might be useful as public APIs. The @frozen and @inlinable attributes could also be removed for readability.

1 Like

I was thinking of [Void: Void], but I guess there's not always going to be a good equivalent -- what about AsyncLineSequence for example? Would I have to say something like AsyncLineSequence<TaskGroup<UInt8>>? There is no AnyAsyncSequence<T> type, and the AsyncSequence protocol doesn't currently support the generic syntax.

I think Reflection APIs really need to be more transparent to the compiler. As @benrimmington mentioned, one way is to explicitly mark types with the recently pitched Reflectable protocol. But even in that case, there is a variety of metadata that is emitted and that needlessly adds to binary size. Ideally, reflection APIs should also be known to the compiler so that they could be potentially constant-folded when the reflected types are known, e.g. Type(InternalType).fields. This is obviously quite an advanced feature, but it could conceivably make reflection really performant without complicating code or slowing down compilation with macros.

I’d like to see this “transparent reflection” direction addressed in the pitch, but I don’t know what would be the best way to implement it. LTO is certainly one way of eliminating unused metadata. To constant-fold, though, the compiler would need to be aware of reflective code in earlier stages of compilation. I’m not a compiler engineer, so I don’t know if that would be possible.

I’m really glad to see this feature being pitched and am excited about making reflection more powerful! But I think we should start to consider how reflection could be made efficient enough to be more accessible to low-resource environments (such as Wasm Swift apps).

Tentative +1 to this overall, but I hope implementation of this can include an evaluation of binary size impact. If these reflection features require increasing binary size, I hope they there would be a possibility to opt out? Or maybe even something opt-in aligned with what's discussed in "Pitch #3: Opt-in Reflection metadata" topic?

2 Likes

Besides the actual code size needed for these new types and their implementations, there is no binary size impacts on apps that don't use reflection because all of the APIs proposed make use of the already emitted data.

6 Likes

I'm not convinced that PartialType is needed, based on the use cases in the proposal.

  • PartialType.==(_:_:) could be replaced by a
    Type.isGenericallyEqual(to:) instance method.

  • PartialType.create(with:) could be replaced by a
    Type.init(_:genericArguments:) failable initializer.

3 Likes

+1 on using a different name other than partial types. I like genetic type.

type constructor is fine too but to me a constructor will always mean initializer.

2 Likes

Would we not want the implementation to bleed into the language model here by calling this something like UnboundGenericType? IMO GenericType alone is not clear enough about the fact that it represents a generic type without generic arguments applied. Would also be fine with something like “type constructor” even if that’s a bit academic.

5 Likes

Exactly, that's the whole point. Initializers are syntactic sugar for something akin to static func new(args...) -> Self, it's a function that takes some values and returns a new value of the type it's defined on.

In the same way, a type constructor is a "type function" that works with types, not values. It takes other types as arguments and returns a new type. In a "type-level pseudocode" we could write this as

func Array(Element: Type) -> Type

Thus Array by itself is not a type, it needs to be applied to a type argument to yield a new type, Array<Int> for example. Only we use angle brackets for applying such type-level functions, not parens.

Type function is too general as a name though. It's used as a term in languages that support dependent types. If Swift ever supports something close to dependent types, I hope we can keep "type functions" in mind for exploring that space in the future. OTOH, type constructors exactly match what we're trying to do here and is an established term of art for these things.

1 Like

I'd like to remove UnboundGenericType from the compiler at some point, since it is unnecessary and weird (it's not actually a type of an expression or value). We can implement all of the same behaviors in simpler ways.

7 Likes

Finally. Reflection is coming to Swift :sob::heart:

2 Likes

I'm really excited about more introspection capabilities in the language, but after reading the proposal, I think it's missing an important feature, and I don't quite understand another.

What I really want to do is be able to find all types in a module (or linked into the app) that conform to a given protocol, and then instantiate instances of those. Example (forgive me playing fast and loose with Swift syntax):

protocol ImageFilterPlugin {
	init(param1: Float, param2: Float)
    func process(image: Image) -> Image
}

…

struct MyImageFilterPlugin : ImageFilterPlugin {
    
    init(param1: Float, param2: Float) {
    }
    
    func process(image: Image) -> Image {
        <process and return an image>
    }
}

…

func
instantiateImageFilter(filterNamed: String)
	-> ImageFilterPlugin
{
	let filterType = Reflection.type<ImageFilterPlugin>(named: "MyImageFilterPlugin")
	let filter = filterType(param1: x, param2: y)
	return filter
}

func
listImageFilters()
{
	for filterType in Reflection.types(conformingTo: ImageFilterType.self) {
		print("Filter: \(filterType.name)")
	}
}

It would be even better if I could iterate over all the methods of a type, find the initializers, and choose one to call. Even better would be a way to annotate things to make them findable (the way Java does). I could annotate any type, property, method, function parameter, and act based on those annotations (search for, construct a call to, etc.).

The thing I'm not clear on in your examples: They show (e.g.) instantiating Types, but it's not clear to me how to instantiate an instance of that type.

8 Likes

Overall nice first step, great to see this!

I'd personally want to see a "vision" going along with this initial part, because the long term here is alluded to, but not really explored much. I'm a bit worried about offering such "partial" API, missing features to discover methods etc, even if just for registering them for later execution etc.

There is also overlap between reflection APIs and macros "compile time reflection", and it would be good to explore this overlap if we can at least provide similar (or shared!) APIs -- it would be a bit annoying if the runtime reflection API shape ends up completely different from the compile time one.

Specifically, I'm looking at this having worked with Scala reflection and macros before; where the two APIs were attempting to surface some common interfaces. Scala 2's reflect API offered an abstract API scala.reflect.api what had a subset of things that are accessible both from a macro "universe" as well as a runtime "universe". This may or may not be desirable for Swift, but it might be worth an invesigation -- especially as we're developing macros and reflection right now at the same time.

A particularly interesting aspect of macros is that they are based on the same API used also for Scala’s runtime reflection, provided in package scala.reflect.api . This enables the sharing of generic code between macros and implementations that utilize runtime reflection.

Similarly, Scala 3 also allows macros to access to the full reflection API from the macro runtime via the quotes.reflect module: Reflection | Macros in Scala 3 | Scala Documentation

I bring this up here now, since the initial expression macros pitch explicitly didn't cover any Type interactions yet, so there is room to make sure we make the best possible API happen.

For example, would it make sense to be able to get a Reflection.Type from a MacroEvaluationContext, by querying the compiler to resolve a given AST nodes type? Access to types will most definitely be necessary in more advanced macros, so it'd be interested to see if we can marry the two rather than each have their own almost-the-same "Type".

18 Likes

Yeah, this is a great point. We're effectively exposing Swift's type system as an API for users, and it would be best if there could be a single type API that works for runtime reflection, macros, and Swift-based tools.

However, what would this mean for our design? Does the representation of a type need to be something that can be used with several different sources of information at once, i.e., can you get a type from reflection information (say, from a T.self in the running program) and pass that type to some static reflection facility (say, that looks at the binary for a library that hasn't been loaded) or ask the compiler for more information about it from the corresponding source code?

I think the best way to come at this problem is to separately design the type representation we'd want for macros and/or a compiler (I assume they'd be the same, but one never knows), and then do some compare/contrast to see what unification would look like. I suppose that's on me to peek further down the road of providing type information to macros.

Doug

17 Likes

I agree partial type is maybe not the best name for this thing. I think type constructor would be a good name if that's the only thing this did (which is pretty much the only thing it does right now). I imagine in the future if we wanted to inspect "generic requirements of a generic type", then this would probably happen on partial types and wouldn't make sense with the name type constructor. Generic type is an interesting name because it fits nicely with the use case above, but it gets a little weird because the type doesn't necessarily need to be generic to have a "GenericType" (e.g. Int has a "GenericType"). That being said, I'm all ears for a better name :slightly_smiling_face:

Keypath introspection is definitely a much requested featured and I imagine we'll eventually want to be able to do that (either directly on keypaths, or some new facility in this Reflection module). I've held off on designing anything that space so far and only focused on what's being proposed.

The second library on that list happens to be one I made a couple of years ago :slightly_smiling_face: That library is intended to be a low level interface to the Swift runtime which allows for higher level APIs like the ones being proposed. The first library has some similarities (Property vs Field), but it looks like it was mostly used to get/set properties via string name (which one can do here by comparing Field.name and using Field.keyPath) and construct instances of types at runtime (using unsafe bits that doesn't always work).

-disable-reflection-metadata removes the reflective capabilities of Field and Case, so every other API available on Type and PartialType would still work because it uses information required by the runtime (not strippable).

Yeah, Field.offset should probably return nil in that case. We can for sure add all of the MemoryLayout APIs and more onto Reflection.Type, but it does feel redundant because you can get the same info from 2 separate facilities.

Yeah, those placeholder types are probably better to use than [Int: Int]. Thanks! Having the placeholder in the textual representation is also a nice improvement here. I made the textual representation a separate API under name because I was unsure how common it is to access a type's description, but if people think that's a better place to put this, then by all means :slightly_smiling_face: I imagine description would look like Type<_, _> and debugDescription could look like Module.Outer<_>.Type<_, _>?

I included all of the @usableFromInline, @inlinable, and @frozen as part of the detailed design because I wanted evolution to clearly see what we're committing to as ABI. As you can see, Type stores a value of type Metadata which is why I didn't name it as such (that API will be proposed at a later date).

5 Likes

I would like to be able to ask a Type if it is a class, struct, enum, etc. It looks like the currently proposed module would not allow this.

2 Likes

Excited for this! Getting KeyPaths for fields is important for our codebase, and we're currently relying on _forEachFieldWithKeyPath via @_spi(Reflection) import Swift, which is a little unfortunate. Further in the past, we did it using Mirror, which was at least equally unfortunate.
It would be great to have a proper API for this functionality!

We do need the KeyPaths to be Writable however, which isn't included in the original pitch.

But from the following I gather it would still be possible to make the KeyPaths from Field writable? Perhaps with a cast? Am I reading this correctly?

2 Likes

Without distracting too much from the current pitch, iterating through the constituents of a KeyPath is very interesting functionality to us as well.
Seems like it may also contribute towards enabling keypath codability?

1 Like