Are KeyPaths Codable?

porterchild · April 30, 2019, 1:24am

I've a pretty simple question: Can a KeyPath be encoded and decoded?

Our team decided to use KeyPaths partly because of a weakness of pointers, which is that a pointer is only valid addressing memory on the machine it was made on. This makes it less useful for our application, which is a distributed system on more that one computer. It seemed to us at the time that a KeyPath, since it is instance-agnostic, should be able to work on the machine it was made on as well as any other machine.
However, as I've tried to conform KeyPath to Codable, I've run into compiler errors every way I've approached it. I'm second guessing whether they are Codable, since I don't understand how KeyPaths are implemented.
As an ancillary question, is there a simple way to explain what a KeyPath actually is? Like how is it implemented?

Joe_Groff · April 30, 2019, 1:36am

KeyPaths are not currently Codable. This would be an interesting future direction, but if done naively, it could be a security issue, since dereferencing an arbitrary key path could lead to arbitrary code execution. Part of the eventual design for coding key paths would be to have some way of marking types, properties and other declarations as coding-safe, similar to Foundation's NSSecureCoding concept.

If you're interested in what a key path object looks like under the hood, this document in the swift compiler is still mostly accurate:

github.com

apple/swift/blob/main/docs/ABI/KeyPaths.md

# Key Path Memory Layout

**Key path objects** are laid out at runtime as a heap object with a
variable-sized payload containing a sequence of encoded components describing
how the key path traverses a value. When the compiler sees a key path literal,
it generates a **key path pattern** that can be efficiently interpreted by
the runtime to instantiate a key path object when needed. This document
describes the layout of both. The key path pattern layout is designed in such a
way that it can be transformed in-place into a key path object with a one-time
initialization in the common case where the entire path is fully specialized
and crosses no resilience boundaries.

## ABI Concerns For Key Paths

For completeness, this document describes the layout of both key path objects
and patterns; note however that the instantiated runtime layout of key path
objects is an implementation detail of the Swift runtime, and *only key path
patterns* are strictly ABI, since they are emitted by the compiler. The
runtime has the freedom to change the runtime layout of key path objects, but
will have to maintain the ability to instantiate from key path patterns emitted

This file has been truncated. show original

porterchild · April 30, 2019, 8:18pm

Perfect, thanks Joe. I'd been wondering about KeyPaths under the hood, but couldn't make any progress reading the KeyPaths.swift source. Glad you pointed that document out.

We've been using KeyPaths inside of a Lens object to give access to data in a uniform way. A BoundLens packages the data itself and the KeyPaths to mutate it. It's nice for an actor to have a way to mutate data without caring what it actually is. Great for some of our ML applications.

We've been hesitant to adopt Foundation in the past because our application will run on a cluster of Linux machines, and we aren't sure about access to all the extra Foundation functionality you can get on OSX compared to Linux. Do you see a workaround to be able to encode and decode KeyPaths through some Foundation functionality like NSSecureCoding, or anything else? Any pointers would be appreciated.

Joe_Groff · April 30, 2019, 8:23pm

Well, if you know the set of keys intend to allow to be serialized and deserialized up front, you can write the mapping yourself:

func keyString(for key: AnyKeyPath) -> String? {
  switch key {
  case \MyObject.foo: return "foo"
  case \MyObject.bar: return "bar"
  default: return nil
  }
}

func key(from string: String) -> AnyKeyPath? {
  switch string {
  case "foo": return \MyObject.foo
  case "bar": return \MyObject.bar
  default: return nil
  }
}

You might be able to use a code generator like Sourcery to automate the generation of these functions for you.

porterchild · May 29, 2019, 10:53pm

Ok, this makes perfect sense for serializing KeyPaths with a "length" of 1 (a Root type and a Value type with no types or indices in between).

But if you wanted to use your mapping for Keypaths of longer "lengths", wouldn't there be a combinatorial explosion of cases to generate?

for example, in order to have a key like this:
\Tree.branches[3].leaves[7])

you would have to define
case "Tree.branches[0].leaves[0]": return \Tree.branches[0].leaves[0]
case "Tree.branches[0].leaves[1]": return \Tree.branches[0].leaves[1]
case "Tree.branches[0].leaves[2]": return \Tree.branches[0].leaves[2]
...maybe thousands more...

Is there another way to do this manual mapping for longer KeyPaths?
Thanks

Joe_Groff · May 31, 2019, 10:26pm

Not with public API, unfortunately. There would need to be a way to iterate through the components of a key path.

porterchild · December 8, 2020, 9:17pm

Using .recursivelyAllKeyPaths from here, and a .recursivelyAllMemberNames that uses Mirror to traverse an object graph for member names, I've been able to make a mapping between name and KeyPath that enables making arbitrary-depth KeyPaths Codable.
My .recursivelyAllMemberNames is pretty hacky (including regex-ing mirror.description to pull out the width of a SIMD, for example). It would be nice if there was more Reflection functionality to get that in a less rickety way.

rxwei · December 9, 2020, 1:34am

You can try the (experimental) key path reflection library in swift-evolution-staging: GitHub - apple/swift-evolution-staging at reflection. @Alejandro implemented this based on runtime metadata. It has APIs like Reflection.allNamedKeyPaths(for:) which jointly iterates over names and key paths, and it's straightforward to make a recursive version out of this. Another alternative is _forEachFieldWithKeyPath(of:options:body:) in stdlib. None of these are stable API so do use them with caution :)

stephencelis · July 8, 2022, 10:13pm

Has there been any thought in revisiting support for this? Especially since it seems possible to do with a wrapper and _forEachFieldWithKeyPath?

Given how prevalent Codable has become it would seem to be a handy feature to have. Decoding SwiftUI NavigationPath view model data, for example, that happens to use a key path.

philipturner · July 9, 2022, 1:18am

Although not as part of the standard library, you could feasibly make a custom Swift package that conforms AnyKeyPath to Codable. I dug up a lot of internal Stdlib code for KeyPaths to work around the removal of _forEachFieldWithKeyPath from release toolchains, but the project also easily demonstrates how key paths are implemented behind the scenes. You might have to dig up a bit more Stdlib code, but you just need to extract the underlying key path components. The rest should be easy.

To add conformance to Codable, I suggest:

Fork swift-reflection-mirror.
Search the relevant files in apple/swift.
Implement Codable conformance inside the ReflectionMirror module.

Warning: Don't expect an app shipping with my Swift package (or a fork of one) to be accepted on the Apple App Store, because it's exposing a private API. But for something like a personal project or proof-of-concept, there should be no barriers to adding Codable conformance.

dasmomo · April 13, 2023, 11:03pm

Just wondering if there has been any progress on Codable Keypaths. Seems like none that I could find but maybe someone out there knows something?

I think this would be useful in my use case where multiple processes communicate.