Codable synthesis and decoding unavailable values

Recently I've been working on improving the Swift compiler's code generation for sources that include unavailable declarations. Declarations that are marked unavailable with the @available attribute are meant to be unreachable at runtime since the type checker only allows references to unavailable declarations inside declarations that are also unavailable:

@available(*, unavailable)
struct Unavailable {
  func foo() {
    print("Supposedly unreachable")
  }
}

Theoretically this means that machine code for unavailable declarations should not need to be emitted into the final binary. However, the existing compiler does not consider unavailability during code generation and as a result unavailable declarations are compiled normally, bloating the resulting binaries. Fixing this could significantly improve the code size of large cross-platform libraries, but of course nothing is ever as simple as it seems. While investigating fixing this I have found a long tail of type checking and code generation flaws that inhibit the optimization from being viable. So far I've been able to add missing diagnostics to the type checker or improve the correctness of generated code to resolve the issues. Today, though, I discovered a flaw related to Codable synthesis that has potential implications on errors thrown by decoders and I wanted to discuss how to address it with the community since the best fix might involve introducing a new case to DecodingError.

The problem

Enum declarations can contain unavailable cases. If you add an associated value to such a case and allow the compiler to synthesize a Codable conformance for the enum, with the Swift 5.8 compiler you're able to bypass type checking and instantiate an instance of an unavailable type at runtime if you craft the right payload for decoding:

import Foundation

@available(*, unavailable)
struct Unavailable: Codable {}

enum EnumWithUnavailableCase: Codable {
  case a
  @available(*, unavailable)
  case b(Unavailable)
}

let json = #"{"b":{"_0":{}}}"#
let encoded = json.data(using: .utf8)!
let decoded = try JSONDecoder().decode(EnumWithUnavailableCase.self, from: encoded)

// Oops, prints "b(main.Unavailable())"
print(decoded)

It's especially easy to imagine how this issue might occur for a cross platform app. The app could have an enum that represents platform specific concepts and some of the cases could be, for example, unavailable on iOS but available on macOS. The macOS and iOS variants of the app might communicate with each other with messages that include encoded representations of these platform specific values, and unexpected behavior could occur in the iOS app after decoding one of these messages.

Regardless of whether we want to optimize code size, the behavior that is reproducible with the example above needs to be fixed since allowing unavailable code to be executed defies programmer expectations and introduces a kind of undefined behavior. The question is what should happen at runtime instead.

Potential solutions

When decoding an enum value, a hand-written Decodable conformance would probably throw an error if a value of an unavailable case were encountered. The DecodingError type in the standard library currently has .typeMismatch, .valueNotFound, .keyNotFound, and .dataCorrupted cases. Of these cases, the only one that seems vaguely appropriate to describe an unavailable value is .dataCorrupted but it feels a bit misleading. The data represents a value that is incompatible with the current runtime, rather than being fully unrecognized. Some programs might want to catch an error that is specific to this situation and ignore the corresponding value but otherwise treat the condition as non-fatal.

My initial inclination is to propose the addition of a new DecodingError case to represent this failure:

public enum DecodingError: Error {
  // ...


  /// An indication that the data represents a value that is not representable
  /// at runtime because it has been marked unavailable with `@available`.
  ///
  /// As an associated value, this case contains the context for debugging.
  case unavailableValue(Context)
}

I'd like to get some preliminary feedback from the community on this approach to the problem. Does a new error seem necessary for this case? Or are there any alternative solutions you'd consider?

11 Likes

This is a really interesting edge case — glad you've brought it up here! It's extraordinarily surprising that you can end up with a state at runtime that's unrepresentable at compile time. :astonished:

Could you elaborate a bit on what you have in mind here? I can imagine an app maybe wanting to display a slightly more specific error message to an end user, but I'm not sure I can easily think of a case with an obvious fallback in case the data doesn't match what's possible at runtime. (Specifically, values in a payload are rarely considered in a vacuum; if part of my payload can't be decoded, it's generally rarely the case that other parts of the payload don't rely on that information, or can proceed by simply dropping it on the floor.)

I don't feel strongly about this at all, but: although it's less specific, one benefit of sticking with .dataCorrupted is that it's the same error you'd get if the case simply didn't exist at all — which I imagine is conceptually what we're going for here.

Given

enum E {
    case a(Int), b (String)
}

I would { "c": { ... } } to lead to a .dataCorrupted error on decode. I can imagine that adding an @unavailable case c would, then, change nothing about how the enum is decoded, and you'd get the same results.

3 Likes

Is it? IMO it seems like potentially-salient info that you’re trying to decode a case which the program knows about but cannot instantiate on the current system. Yes, you can encode this in the debug message, but it would be difficult to, say, use that information to direct the user to a “how to update your OS” page or similar.

Sorry, I should have clarified this. I intended to refer specifically to the fact that we're trying to move to a model where @unavailable types and values are not just inaccessible at compile-time, but also at runtime. With a change like this, an @unavailable type becomes a bit more like Never, so there are some parallels to be drawn to some of the recent discussion around Never as a Codable type in enum cases — and in general, a philosophical or conceptual question of "if an enum case is uninstantiable [because it contains Never, or is @unavailable], does it meaningfully exist? (and for what definition of "meaningful"?)"

I guess it would help to have a more explicit answer to this question to help shape decisions like this.

I completely agree — the error type here sends useful signal. I could have made this clearer in the last post, too. My thoughts revolve mainly around whether this signal is useful enough to turn into API; because API is Forever™, I think we're dealing with a decently high bar to clear, so I think a well-laid, concrete use-case would be helpful.

In my mind, there are two main user-facing scenarios to consider here:

  1. The type is unavailable right now, but could be available in the future (i.e., it's unavailable on your current OS version, but would be if you upgraded, as you mention)
  2. The type is unavailable altogether because of your platform
    • This could be because you're on a related platform which is missing those APIs, like trying to send across certain Apple Watch API data to macOS,
    • Or because you're on an unrelated platform which is entirely different, like trying to send macOS API data to Linux

If you're writing a cross-platform app, you can easily end up with a mixture of these two, and if you care about the message you send to your customers, the specific error message may need to be tailored to the specific reason why something is @unavailable.

If this is the case, you're almost certainly going to want to inspect the coding path of the context to see what failed, in which case you know which field is the issue, and you already know statically why that field is @unavailable. If so, you don't really need to access the error type at all to understand what to do next.

OTOH, I can definitely see the error type being useful as a convenience: if all of your @unavailable fields are unavailable for the same reason, you don't at all care about the context, and it would be helpful to be able to catch a single error type and be done with it.

I guess my question is: does this come up often enough to be worth turning into API?

(And to be explicitly clear, just in case: I'm trying to pose these questions not because I need to be convinced or anything [I'm just some guy; my opinion is irrelevant], but because I'm trying to help strengthen the case one way or another.)

5 Likes

Thank you for elaborating here, this all seems like the right framing and I agree these are important questions to answer as we consider what the "right" thing to do is here. I took your comment as a positive assertion that "unavailable cases should behave like they were not written in source" which seemed like exactly the opposite of the conclusion we reached on the "Never is Codable" proposal.

Even if we don't introduce a bespoke error type, dataCorrupted feels wrong here to me. You may have produced the data from encoding the exact same type definition on a different platform/OS version, so it's odd to surface that as the data being "corrupted" somehow.

3 Likes

Yeah, this has come up a few times recently, and I wholeheartedly agree. If I could go back in time, I'd consider naming this error something more generally-applicable like unexpectedData, but that's neither here nor there. (And, well, API is Forever™)

4 Likes

If I were implementing bespoke decoding in my app I would want to distinguish between data that is recognized but un-decodable and data that is completely unrecognized. More granular errors that point to the source of the issue are generally helpful when debugging. More importantly, seeing an error (perhaps in analytics) that indicates data decoded by my app could be genuinely corrupted is alarming. I want to rule out as many explanations for receiving this error as possible since without context it already represents such a wide number of potentially severe issues.

This convenience you identified is also one of the primary use cases in my mind. Situations where it would be useful may not come up often, though.

I'm definitely fine with the answer to this question being "it's valuable in theory but not common enough to justify additional API." Assuming there aren't considerations we haven't yet discussed, I'm going to work under the assumption that that's the conclusion here and that decoders should just throw dataCorrupted when encountering unavailable values.