Hello Swift Community,
TL;DR
Swift has various reflection mechanisms, but they’re fragmented. By combining them, a more flexible and efficient reflection system could be established, that could enable features like compiler-synthesized conformances.
This post simply aims to start a discussion about the future of reflection and cultivate ideas for specific features; no concrete features are proposed.
Current Issues
Swift does a great job of combining runtime features with compile-time optimizations. From generic calls being devirtualized into calls with direct dispatch, to specific types to magic functions like type(of:), Swift offers great features that blur the line between dynamic and static. Reflection, though, is firmly divided into these two categories: dynamic and static, with the former offering both high- and low-level APIs.
High-level dynamic reflection brings Mirror to mind, mostly used for debugging and reverse-engineering proprietary code. It provides an easy-to-use collection of fields, each containing a label and a value in the form of safe standard-library types: String and Any. Further, library authors are given the flexibility to customize their types’ mirror representation through the CustomReflectable protocol. This safety and flexibility, though, comes at a performance cost.
Low-level dynamic APIs are the polar opposites of Mirror, being fast and slightly less safe, but hiding their not-so-intuitive APIs from public awareness. One such API is the standard library’s _forEachField, which is only available to libraries shipping with the OS, like Foundation, and offers less refined APIs with types such as C strings and access to implementation details like metadata kind. Other libraries, like TokamakUI, reimplement compiler logic, essentially creating a reflection system from scratch. These APIs are less user-friendly and don’t allow for API-author customizability, but benefit from significantly increased performance and the ability to mutate fields. Still, compile-time metaprogramming is faster.
Static metaprogramming in Swift mainly takes the form of protocol-conformance synthesis. Arguably the most ubiquitous synthesized protocols are Hashable and Codable. These protocols don’t rely on metadata to synthesize a conformance at runtime, but integrate with the compiler to synthesize static conformances. Static synthesis also comes with compile-time diagnostics, which further increase safety. Albeit having great performance, conformance synthesis can result in greater binary sizes. What’s more, the compile-time synthesizable protocols need to be coded into the compiler, limiting library authors and increasing compiler complexity.
While current reflection mechanisms are varied and can satisfy many use cases, they are fragmented. This can confuse library authors from crafting lightweight APIs without boilerplate. Such confusion is avoidable, though, since the compiler already features optimization flags to aim for decreased binary size or increased speed, not to mention that it has knowledge of client code, allowing for more effective, per-use-case optimizations. The compiler’s current lack of knowledge about reflection also means that metadata has to emitted for all nominal types, leading to unnecessary binary-size increases. Apart from performance and size limitations, the current approach deprives library authors of the much-requested compiler-synthesized conformances. A solution could thus allow for less boilerplate and faster yet smaller binaries.
While I believe this is a good starting point for a discussion, I would like to propose a general direction for a solution. This solution should combine dynamic, less-customizable APIs with compiler support for producing compile-time diagnostics, and perhaps offering standard-library-like conformance synthesis.
Potential Solution
Targeted Reflection
To start off, one of the most evident issues with Swift is lack of opt-in reflection metadata. This is an obvious step, which could be solved with a marker protocol:
@_marker
protocol Reflectable {}
Only types conforming to this protocol will have metadata emitted. This approach would be a good starting point, but miss the case where reflection is only needed for synthesizing conformances.
Scoped Reflection
A new attribute @synthesized could be introduced, which would be applicable only in protocol extensions to declarations that can witness that and parent protocols’ requirements. Inside @synthesized declarations, Self is treated as conforming to Reflectable:
public protocol Dog {
var isParent: Bool { get }
}
extension Dog {
@synthesized
public var isParent: Bool {
// Self is treated as conforming to Reflectable and
// can be passed to generic functions.
hasParents(self)
}
func hasParents<DogType: Dog & Reflectable>(dog: DogType) -> Bool { ... }
}
Similarly to access-control modifiers, @synthesized will tell the compiler exactly where metadata is needed, enabling more aggressive optimizations through scoped reflection. Namely, the compiler would emit metadata for a type only if its conformance is witnessed by at least one @synthesized declaration. However, if that declaration (1) has all its requirements for Self: Reflectable removed, and (2) the type doesn’t conform to Reflectable, no metadata would be emitted.
To force the compiler to synthesize statically, as is required for standard-library protocols like Hashable, the standard library will have access to @synthesized(__statically). This attribute would require that the declaration be @inlinable if marked public. It would work by having the compiler inline all uses of the standard-library reflection API (as shown below), such that no requirements R: Reflectable are left. If there are still such references, implying use of another reflection API, an error would be emitted.
These methods would work well to precisely describe the reflection requirements of a program to the compiler, so that the reflection APIs can be very efficient. These new reflection APIs could, in fact, be distinguished into the conformance-synthesis type and the debugging type. Conformance-synthesis would be achieved by low-level dynamic and static reflection, while Mirror would mainly be used for debugging.
Reflection Mechanisms
The Mirror API would remain as is, providing a user-friendly, intuitive debugging experience. However, creating a Mirror would only be available to Reflectable-conforming types, similarly to conformance-synthesis reflection.
The existing conformance-synthesis reflection mechanisms suffer from a lack of user-friendly APIs. Therefore, a new API will be created, that will integrate with static synthesis and compile-time diagnostics:
struct ReflectedFields<Subject> {
struct Options: OptionSet {
static let diagnose: Options
}
init(_ type: Subject.Type, options: Options = .cache)
}
extension ReflectedFields: RandomAccessCollection {
typealias Element = Field
var startIndex: Int { get }
var endIndex: Int { get }
subscript(position: Int) -> Field { get }
}
extension ReflectedFields {
struct Diagnostic {
static let error: Diagnostic
static let warning: Diagnostic
}
// For static diagnostics, the type
// declaration's file and line will be used.
func diagnose(
_ diagnostic: Diagnostic,
_ message: StaticString,
file: StaticString = #file,
line: UInt = #line
)
}
extension ReflectedFields {
struct Field {
var path: PartialKeyPath<Subject> { get }
var name: String { get }
func diagnose(
_ diagnostic: Diagnostic,
_ message: StaticString,
file: StaticString = #file,
line: UInt = #line
)
}
}
This API would work dynamically like _forEachField and lack Mirror-like customizability to preserve speed. If used in a @synthesized declaration, though, the compiler would have the knowledge to inline properties, if deemed appropriate, to either synthesize statically (removing any runtime reflection) or to produce diagnostics.
To produce diagnostics, the compiler would evaluate the inlined declaration, whose implementation would, of course, be quite demanding. If it came across a Never-returning call or one of the above diagnose calls in all paths, it would emit that error. For example,
protocol Proto { func myRequirementWitness() }
extension Proto {
@synthesized(__statically)
func myRequirementWitness() {
let fields = ReflectedFields(Self.self)
fields.allSatisfy { field in
type(of: field.path).valueType == Int.self
}
}
}
struct MyStruct: Proto {
let a: Int, b: String
}
// myRequirementWitness() would become:
func myRequirementWitness() {
let fields = ReflectedFields<Self>(_fields: [
Field(path: \.a, name: "a"),
Field(path: \.b, name: "b")
]
let succeeded = fields.allSatisfy { field in
type(of: field.path).valueType == Int.self
}
if !succeeded { fields.diagnose(.error, "Failed") }
}
// Which would be constant-folded into:
func myRequirementWitness() {
var succeeded = true
if type(of: \Self.a.path).valueType != Int.self {
succeeded = false
}
if type(of: \Self.b.path).valueType != Int.self {
succeeded = false
}
if !succeded { fields.diagnose(.error, "Failed") }
}
// Then:
func myRequirementWitness() {
var succeeded = true
if Int.self != Int.self {
succeeded = false
}
if String.self != Int.self {
succeeded = false
}
if !succeded { fields.diagnose(.error, "Failed") }
}
// Lastly:
func myRequirementWitness() {
if !false { fields.diagnose(.error, "Failed") }
}
// So the error is printed
Example: Hashable
public protocol Hashable: Equatable {
func hash(into hasher: inout Hasher)
}
extension Hashable {
@synthesized(__statically)
@inline(__always)
public func hash(into hasher: inout Hasher) {
ReflectedFields(Self.self).allSatisfy { field in
guard let property = self[keyPath: field.path] as? Hashable else {
field.diagnose(.error, "\(field.name) does not conform to 'Hashable'")
}
hasher.combine(property)
}
}
}
Past Pitches
Structural
The Automatic Requirement Satisfaction in Plain Swift proposal introduced the Structural protocol, which was automatically synthesized for nominal types, encoding their structure into the type system. As Joe Groff put it:
KeyPathIterable
In the above post, Joe goes on to reference the Dynamic Property Iteration Using Key Paths proposal. This discussion was heavily inspired by KeyPathIterable which uses a collection of key paths to traverse the structure of a given type. The major difference between these solutions is that Reflectable integrates with the existing Mirror API. If however, synthesizing an array of key paths at compile time is found to be faster, the solution discussed in this post is flexible enough to accommodate such a change.
Implementation Path
Stage 1: Opt-in Metadata
Introducing opt-in metadata with Reflectable will likely be straightforward. Building on this, a version of @synthesized that just extends Reflectable should also be quite simple.
Stage 2: ReflectedFields
Crafting the API of a new reflection mechanism should be a thorough process, but the implementation can simply rely on the existing standard-library _forEachField function.
Stage 3: Dynamic → Static
Introducing inlining for static synthesis, and @synthesized(__statically) by extension, will likely be straightforward in terms of schematics, but will have a challenging implementation.
I’m interested to hear your thoughts!
.