Where is reflection heading?

Evolution Discussion
,
1

Edit: renamed because although I start with a very (too) specific problem the real question it's building up to is: is there a long-term vision for reflection?

With a CaseIterable CodingKeys enum and Mirror I can almost implement a runtime-type-checked version of the compiler-magic Encodable implementation.

Side note: why would you want to?

The JSON API I'm coding to wasn't designed with type-safety in mind. I can get much tighter types by moving things around, but that means I can't use the compiler-provided Codable implementations. If I implement Decodable by hand, the compiler's all-members-initialized check makes it hard for me to get it wrong: forgetting a member is a compiler error. But there's no corresponding safety-net for the Encodable implementation: only tests will save me from adding a member (and associated coding key) but not updating encode(to:Encoder) to match. Because of this danger I'm looking for utility alternatives to a fully custom implementation for each struct representing an API object.

Almost, but not quite, because from a CodingKey value I can't get back to the enum case name, which I need to look up the matching value via Mirror(reflecting: myEncodable).

enum CodingKeys: String, CodingKey {
    case defaultName // default: .stringValue is also the myEncodable member name
    case nonDefaultName = "some-other-name" // .stringValue is not the myEncodable name, no access to "nonDefaultName"
}

I would have expected to be able to find the case name somehow via Mirror(reflecting: CodingKeys.nonDefaultName) but apparently not.

  1. Am I simply missing something? (I'm still on Xcode 9, btw.)
  2. If not: is there a deep design problem here, or just an implementation gap? Is there some sense that "reflection is incomplete but we know where we're going?" Reflection manifesto? ABI stability implications?
1 Like
2

tl;dr: The metadata to do this is already being compiled in; there just aren't any public APIs to access it.

If you have a copy of Swift master (or maybe a snapshot) installed, try this: Copy your CodingKeys sample to the clipboard, then run the command pbpaste | path/to/swiftc -emit-ir - | pbcopy (replacing the "path/to/swiftc"), and then paste the resulting code into an editor.

What you're looking at is the LLVM IR for that code—a sort of pseudo-assembly language form where most of the Swift-specific processing has already happened. After a brief header, it starts off by declaring a ton of constants. Somewhere around line 34, you'll see these two lines:

@0 = private unnamed_addr constant [12 x i8] c"defaultName\00"
@1 = private unnamed_addr constant [16 x i8] c"some-other-name\00"

These are the string values of those enum cases; as you might expect, you don't see nonDefaultName there. But keep looking. Further down, around line 112, you'll see something like this:

@10 = private constant [12 x i8] c"defaultName\00", section "__TEXT,__swift5_reflstr, regular, no_dead_strip"
@11 = private constant [15 x i8] c"nonDefaultName\00", section "__TEXT,__swift5_reflstr, regular, no_dead_strip"
@"$S4main10CodingKeysOMF" = internal constant { i32, i32, i16, i16, i32, i32, i32, i32, i32, i32, i32 } { i32 trunc (i64 sub (i64 ptrtoint (<{ [1 x i8], i32, i8 }>* @"symbolic \01____ 4main10CodingKeysO" to i64), i64 ptrtoint ({ i32, i32, i16, i16, i32, i32, i32, i32, i32, i32, i32 }* @"$S4main10CodingKeysOMF" to i64)) to i32), i32 0, i16 2, i16 12, i32 2, i32 0, i32 0, i32 trunc (i64 sub (i64 ptrtoint ([12 x i8]* @10 to i64), i64 ptrtoint (i32* getelementptr inbounds ({ i32, i32, i16, i16, i32, i32, i32, i32, i32, i32, i32 }, { i32, i32, i16, i16, i32, i32, i32, i32, i32, i32, i32 }* @"$S4main10CodingKeysOMF", i32 0, i32 7) to i64)) to i32), i32 0, i32 0, i32 trunc (i64 sub (i64 ptrtoint ([15 x i8]* @11 to i64), i64 ptrtoint (i32* getelementptr inbounds ({ i32, i32, i16, i16, i32, i32, i32, i32, i32, i32, i32 }, { i32, i32, i16, i16, i32, i32, i32, i32, i32, i32, i32 }* @"$S4main10CodingKeysOMF", i32 0, i32 10) to i64)) to i32) }, section "__TEXT,__swift5_fieldmd, regular, no_dead_strip", align 4

Here we see both cases mentioned by their names instead of their string values. The next line puts them (referred to as @10 and @11; I promise they're somewhere in that monster line) into a complicated data structure, which is assigned to a symbol named $S4main10CodingKeysOMF. If you run that name through swift demangle, it will say:

$S4main10CodingKeysOMF ---> reflection metadata field descriptor main.CodingKeys

So in the master branch, information about the cases of the CodingKeys enum is already being generated by the Swift complier and packaged alongside your executable code. There aren't any public APIs to access it yet, but we could add them in any future version. (It's out of scope for Swift 5, though—it's not critical for ABI stability.)

If you're impatient, though, you don't have to wait for someone else to design something. The reflection format's documentation seems to be out of date, but you could dig into the code, figure out how to parse it, and write a library to expose this metadata. It wouldn't be easy to build, of course, but it's definitely possible—and it might even turn into an evolution proposal one day.

Hope this helps!

13 Likes
3

Thanks very much, that covers everything I was hoping to learn and more that I didn't think to ask :-)

4

I do have one follow-up question, if you don't mind spending a bit more time here: if someone hypothetically were to write the metadata-access library, how would it fit into the compiler/language architecture? This is a real n00b question, please feel free to just throw me some google keywords and send me on my way; an example of another library (for a completely different purpose) that fits into the same place would also give me something to dig my teeth into. (Googling I find lots of rundowns on how to make a contribution to swift-the-language, but if I understand you correctly this could be a standalone library, importable from Swift code, but with access to metadata that "just plain swift code" normally doesn't have.)

1 Like
5

This is something that could be written in C++ or in Swift, and could be shipped with the standard library or distributed outside it.

Here's an article which gets into the details of the executable format on macOS and friends. The main thing that's relevant here is what sections are and how they're organized.

You'll need to be able to look up symbols at runtime (that is, look for a global variable/function by name). That's done with the dlsym() call. It has a man page, and I'm sure there are good articles about it.

In general, I think "Mach-O", "getsectiondata", and "dlsym" (and maybe "dladdr") are the key underlying technologies you care about. You might want to search for examples of these being used.

Now for Swift-specific stuff:

If you're using dlsym(), you'll need to know about Swift's name mangling. When Swift creates a symbol, it doesn't just use the name a human would write—that wouldn't handle overloading and various other things. Instead it adds lots of type information to the name, and also abbreviates certain common parts of a name (like "Int" or "String"). Here's the mangling documentation; here's C++ in the compiler which assembles mangled names.

If you can read C++ or another C-like language, I'd look at the Swift repo's swiftReflection library (headers, implementation). IIRC, this is some of the infrastructure behind the memory debugger in Xcode, which is a place where we're using this metadata already. Not everything here is relevant—this is actually meant to read the data in another process—but it might give you an idea.

There's another file in the Swift repo, SwiftReflectionTest.swift, which does much less actual parsing of the metadata, but it shows how you might do some of the low-level calls in Swift.

I don't know how far you'll get with this, but hopefully you can at least print a few interesting things. :^)

7 Likes
6

I'm sure I will get not very far at all, but the journey will be fascinating! Thanks for indulging me :-)

1 Like
7

Is there a formal roadmap for reflection? I'd sure like to see more capability (like the ability to instantiate structs/classes discovered by implemented prototype: Reflection in Swift 3?). It doesn't seem there's been much evolution in this area over the last few years.

4 Likes