[Pitch] Add CustomDebugDescription conformance to AnyKeyPath

Motivation

Currently, passing a keypath to print(), or to the po command in LLDB, yields the standard output for a swift object. This is not very useful. For example, given

struct Theme {
    var background: Color
    var foreground: Color
}

print(\Theme.backgroundColor) would have an output of roughly

Swift.KeyPath<Theme, Color>

which doesn't allow foregroundColor to be distinguished from any other property on Theme.

Ideally, the output would be

\Theme.backgroundColor

exactly as it was written in the program.

Swift Keypaths that represent variables visible in Objective-C can take advantage of the _kvcString property to access something resembling a useful output. However, emitting a value for _kvcString in all keypaths would create a code size increase that might not be acceptable for some Swift programs.

Design Goals

  1. make it easier to debug programs that use keypaths by outputting a description that matches how the keypath would be written as a keypath literal.

  2. potentially lay groundwork for more sophisticated reflection APIs to inspect keypaths, or, if there is significant interest in this, expand the scope to include those APIs

  3. do so without significant changes to how keypaths are stored in memory or otherwise modifying code paths that are not related to debugging/introspecting key paths. I'm assuming that decisions like not filling out the equivalent of _kvcString for every keypath were taken for some design purpose (either code size or minimizing memory usage), but if that's not the case then this proposal might be implemented more effectively in other ways

Summary of the implementation

A sample implementation is available here.

Much like the _project functions currently implemented in KeyPath.swift, this function would loop through the keypath's buffer, handling each segment as follows:

For offset segments, the implementation is simple: use _getRecursiveChildCount, _getChildOffset, and _getChildMetadata to get the string name of the property. I believe these are the same mechanisms used by Mirror today.

For optional chain, force-unwrap, etc. the function appends a hard coded "?" or "!", as appropriate.

For computed segments, call dladdr (or the appropriate equivalent on Windows) on the result of getter() in the ComputedAccessorsPtr. Demangle the result to get the property name.

Subscripts

Subscripts in Swift keypaths store their arguments in an opaque ArgumentRef struct, which does not have any way to retrieve a description of the data it's storing. For that reason, the output of a subscript declared as subscript(string: String) -> String would simply be \TypeName.subscript(_:).

13 Likes

Having printing a keypath give a best effort representation of what the key path contains seems reasonable, but it should be clear that it is only a best effort, since field names and symbols for computed property entry points aren't necessarily going to be available in an executable. It should be able to print placeholders for information it can't recover from what metadata is available. It would be interesting to also implement an LLDB formatter for key paths, since in the debugger you may have more information available from DWARF metadata or debug symbols in the dSYM.

7 Likes

but it should be clear that it is only a best effort, since field names and symbols for computed property entry points aren't necessarily going to be available in an executable. It should be able to print placeholders for information it can't recover from what metadata is available.

Can you elaborate on this? Specifically:

  • when will function symbols not be available in an executable? Do the symbol names get stripped in release? Or is this a hypothetical “if someone messes with the binary after it’s been built?” I guess the field names won’t be available when reflection metadata is turned off by compiler flag
  • what should the placeholders look like? I could do ‘\Theme..someField’ for those, or fall back to the old description in cases where no data is available

On a related note, is the next step for this pitch to write an SE, or is this a small enough change that I should be trying to polish the rough implementation into a pull request?

This would be something that goes through the Swift Evolution process.

Thanks!

+1 to this direction!

One thought: in cases with duplicate symbols and property names across modules the output as currently implemented could become more ambiguous to the programmer.

What would you think of maintaining the module name prefix for the Base?

Internal symbols are routinely stripped as part of the default Xcode build settings, as well as other build environments, so you generally shouldn't ever expect them to be present if they aren't absolutely required for dynamic linking at runtime. Developers can currently disable reflection metadata, and I think we'll eventually want to move toward an opt-in model for reflection metadata in release builds too, to get closer to a "don't pay for what you don't use" model.

The exact placeholder rendering, I don't have a strong opinion on, maybe something like .<unknown> with maybe some breadcrumb data that can still be used by a developer with a debugger to figure out what the component is, such as a byte offset for stored properties or the identity address of a computed component. There is a danger for too much noise there, but something like .<stored property at +n> or .<computed property 0xABCDEFG> might be acceptable.

3 Likes

Ah, yeah I knew about the metadata compiler flag, but I didn’t realize the symbols could be stripped too. Does this happen by default in debug builds?

Most of my testing has been in unit test targets in my employers project and the one that’s included in the sample, so I think all I’ve seen is symbols that are in the top level executable, and not in any library that will be statically or dynamically linked in. For context, I originally implemented some of this outside of the swift compiler for a project at my employer; and my goal was to come up with an identifier for key paths that was stable across invocations of the program in a release build and human-readable in a debug/unit testing build.

I’ll go with those placeholders, or something like them. I can break it up onto multiple lines, that should make it a little more clear when it gets too long. I think I will need to print out the original output as well in those cases, as the type of the key path might be more helpful for casual debugging than the memory addresses.

Do you have any opinions about how the demangling should be done btw? In the sample I exposed a function from the runtime to the stdlib, but it produces a lot more output than I need, and I didn’t see anything in the flags that can be passed to the demangler to only give the function name and not the additional context.

If I was going to do it, I'd write a little stdlib SPI function in the C++ runtime that demangles the symbol and extracts the property base name from the demangle parse tree, and just returns that basename as a string so it can be used by the swift-level implementation.

Thanks. I wrote the SPI function, it's a lot cleaner and lets me handle subscripts correctly.

However, I started writing the SE and immediately ran into a new issue around API resilience: based on my reading of the rules, this change violates the requirement that existing types can't add a conformance to a protocol unless that conformance is added at the same time the protocol is added.

Am I missing something? If not, what do you suggest I do? I see three possibilities:

  1. Backport it -- I'm not sure if any example exists for how to do this, but this is clearly the best option, if it's possible. Though it does seem like overkill for just this tiny protocol conformance
  2. Break API resilience. I assume this is out of the question, but I thought it was worth bringing up, especially since no one should be calling this in production, and many people will never even call it directly. Presumably it does not matter in the context of print() or po, which will use a dynamic cast that succeeds on newer versions and fails on prior versions
  3. Provide a var that gives the debug description without implementing the protocol. This has clear issues around discoverability, but it doesn't seem to create issues around usability: users in a debug session just need to write po myKeyPath.debugDescription instead of po myKeyPath. And if they really need the conformance, users can simply add it in their own modules (though of course this is a really bad practice)

I believe this is no longer strictly the case now that conformances can be declared with availability. So the extension KeyPath: CustomDebugStringConvertible conformance would just need an appropriate @available(SwiftStdlib X.Y, *) attribute.

1 Like

Nice, thanks for the link! Sounds like the API resilience docs need an update, they have a "TODO" to implement this.

2 Likes

Beware that adding a new conformance remains a potentially source- and ABI-breaking change. Being able to describe the availability of the new conformance allows newly built binaries to work correctly when back-deployed to earlier releases, but the new conformance may still cause binary compatibility issues when running previously built binaries that contain their own custom implementations for the same conformance.

I don't think this particular conformance would be likely to break anything in practice, but if bincompat tests proved otherwise, adding the conformance would suddenly get orders of magnitude more difficult.

4 Likes

Thanks for the info! I actually read your pitch + pr about String.Index a bit earlier when I was trying to research the ABI stability/resilience implications of this, still trying to understand what I was reading in terms of the modifications to the api-digester test etc.

Searching the swift repo I don't see any test that's readily identifiable as the "bincompat tests" suite? Am I missing something, or is there some documentation on how to test this? Is the way to test this a manual test: build a toolchain with my change, and try to build a program that has its own copy of this conformance and see what happens?

The “bincompat test suite” isn’t part of the Swift repository! Binary compatibility testing means running existing production app binaries against development builds of the Swift Standard Library, to see if anything breaks.

Unfortunately, this generally isn’t something one can do on their own — rather, it is part of preparing a new OS version. (One component of this is the beta testing period that precedes each new release. During the betas, existing app binaries are running on new OS binaries, so everyone running a beta version is participating in a large-scale binary compatibility test.)

Adding a new conformance for a pre-existing type to a pre-existing protocol may turn out to cause binary compatibility issues, in case existing binaries include retroactive conformances for the same. Because of the nature of bincompat testing, these will only show up after the feature has already landed, and it may be tricky to resolve them. In extreme cases, we may need to roll back the feature until we figure out how to resolve the issues.

Such problems do seem highly unlikely to occur with Custom[Debug]StringConvertible conformances — although we cannot entirely rule them out.

(We can do a bit of due diligence though while preparing for putting this through the evolution process. For example, anyone can search public source code to see if there are popular projects that include retroactive versions of the same conformance. If these searches come up with nothing, then that would be a useful signal. (The Swift stdlib team at Apple can do additional things in this vein to look for potential blockers.))

3 Likes

Thanks, this was very helpful. I did a search on github and didn't find anything, so that's some additional evidence it's safe.

1 Like