What can you change in a non-exhaustive enum?

Kind of a hybrid idea, but hopefully one that circumvents the internal
issues that you outline here and simplifies the mental model. I'll
introduce it stepwise:

Currently, the @available annotation is supported for platforms and for
Swift versions; you propose extending support to arbitrary deployments
(e.g., MagicKit). Instead, suppose you extended support to arbitrary
versioned types (open, public, and @_versioned internal):

@_versioned(abi: 2)
public enum SpellKind {
  case hex, charm, curse
  @available(abi: SpellKind 1) case blight
  @available(abi: SpellKind 1.1) case jinx
}

Now, clearly, the value of "abi" is arbitrary inside the @_versioned
annotation; it's not really necessary for our limited purposes, and if
there's a typo and it's lower than the highest ABI version referenced in an
@available annotation, things get wonky. So, drop it:

public enum SpellKind {
  case hex, charm, curse
  @available(abi: 1) case blight
  @available(abi: 1.1) case jinx
}

This is looking like the original @abi(2) proposal that Dave Zarzycki
brought up. However, a key difference here: multiple cases can have the
same ABI "version" and would be ordered relative to each other by name;
that is, a user would only be annotating when new cases are added and
doesn't have to think about memory layout; Swift takes care of the rest.

This can be simplified further; these ABI version numbers are entirely
arbitrary. Suppose we instead extended the "@available(introduced:)" syntax
to allow dates or timestamps:

public enum SpellKind {
  case hex, charm, curse
  @available(*, introduced: 2017-09-30): case blight
  @available(*, introduced: 2017-10-12): case jinx
}

I suspect there are wrinkles to this scheme, but the overall idea here is
to salvage availability ordering but have some way to version a type in a
low-mental-overhead way instead of resorting to a syntax for manually
ordering cases.

···

On Sat, Sep 30, 2017 at 11:58 AM, <swift-dev-request@swift.org> wrote:

Message: 2
Date: Fri, 29 Sep 2017 18:21:44 -0700
From: Jordan Rose <jordan_rose@apple.com>
To: swift-dev <swift-dev@swift.org>
Subject: [swift-dev] What can you change in a non-exhaustive enum?
Message-ID: <31DF689E-1AD3-47CB-9FCE-6CBC7E34BC43@apple.com>
Content-Type: text/plain; charset="utf-8"

Hello again, swift-dev! This is a sort of follow-up to "What can you
change in a fixed-contents struct" from a few weeks ago, but this time
concerning enums. Worryingly, this seems to be an important consideration
even for non-exhaustive enums, which are normally the ones where we'd want
to allow a developer to do anything and everything that doesn't break
source compatibility.

[This only affects libraries with binary compatibility concerns. Libraries
distributed with an app can always allow the app to access the enum's
representation directly. That makes this an Apple-centric problem in the
near term.]

So, what's the issue? We want to make it efficient to switch over a
non-exhaustive enum, even from a client library that doesn't have access to
the enum's guts. We do this by asking for the enum's tag separately from
its payload (pseudocode):

switch getMyOpaqueEnumTag(&myOpaqueEnum) {
case 0:
  var payload: Int
  getMyOpaqueEnumPayload(&myOpaqueEnum, 0, &payload)
  doSomething(payload)
case 1:
  var payload: String
  getMyOpaqueEnumPayload(&myOpaqueEnum, 1, &payload)
  doSomethingElse(payload)
default:
  print("unknown case")
}

The tricky part is those constant values "0" and "1". We'd really like
them to be constants so that the calling code can actually emit a jump
table rather than a series of chained conditionals, but that means case
tags are part of the ABI, even for non-exhaustive enums.

Like with struct layout, this means we need a stable ordering for cases.
Since non-exhaustive enums can have new cases added at any time, we can't
do a simple alphabetical sort, nor can we do some kind of ordering on the
payload types. The naive answer, then, is that enum cases cannot be
reordered, even in non-exhaustive enums. This isn't great, because people
like being able to move deprecated enum cases to the end of the list, where
they're out of the way, but it's at least explainable, and consistent with
the idea of enums some day having a 'cases' property that includes all
cases.

Slava and I aren't happy with this, but we haven't thought of another
solution yet. The rest of this email will describe our previous idea, which
has a fatal flaw.

Availability Ordering

In a library with binary compatibility concerns, any new API that gets
added should always be explicitly annotated with an availability attribute.
Today that looks like this:

@available(macOS 10.13, iOS 11, tvOS 11, watchOS 4, *)

It's a model we only support for Apple platforms, but in theory it's
extendable to arbitrary "deployments". You ought to be able to say
`@available(MagicKit 5)` and have the compiler actually check that.

Let's say we had this model, and we were using it like this:

public enum SpellKind {
  case hex
  case charm
  case curse
  @available(MagicKit 5)
  case blight
  @available(MagicKit 5.1)
  case jinx
}

"Availability ordering" says that we can derive a canonical ordering from
the names of cases (which are API) combined with their versions. Since we
"know" that newly-added cases will always have a newer version than
existing cases, we can just put the older cases first. In this case, that
would give us a canonical ordering of [charm, curse, hex, blight, jinx].

The Fatal Flaw

It's time for MagicKit 6 to come out, and we're going to add a new
SpellKind:

@available(MagicKit 6)
case summoning
// [charm, curse, hex, blight, jinx, summoning]

We ship out a beta to our biggest clients, but realize we forgot a vital
feature. Beta 2 comes with another new SpellKind:

@available(MagicKit 6)
case banishing
// [charm, curse, hex, blight, jinx, banishing, summoning]

And now we're in trouble: anything built against the first beta expects
'summoning' to have tag 5, not 6. Our clients have to recompile everything
before they can even try out the new version of the library.

Can this be fixed? Sure. We could add support for beta versions to
`@available`, or fake it somehow with the existing version syntax. But in
both of these cases, it means you have to know what constitutes a
"release", so that you can be sure to use a higher number than the previous
"release". This could be made to work for a single library, but falls down
for an entire Apple OS. If the Foundation team wants to add a second new
enum case while macOS is still in beta, they're not going to stop and
recompile all of /System/Library/Frameworks just to try out their change.

So, availability ordering is great when you have easily divisible
"releases", but falls down when you want to make a change "during a
release".

Salvaging Availability Ordering?

- We could still sort by availability, so that you can reorder the
sections but not the individual cases in them. That doesn't seem very
useful, though.

- We could say "this is probably rare", and state that anything added "in
the same release" needs to get an explicit annotation for ordering
purposes. (This is equivalent to the `@abi(2)` Dave Zarzycki mentioned in
the previous thread—it's not the default but it's there if you need it.)

- We could actually require libraries to annotate all of their "releases",
but in order to apply that within Apple we'd need some translation from
library versions (like "Foundation 1258") to OS versions ("macOS 10.11.4"),
and then we'd still need to figure out what to do about betas. (And there's
a twist, at least at Apple, where a release's version number isn't decided
until the new source code is submitted.)

- There might be something clever that I haven't thought of yet.

This kind of known ordering isn't just good for enum cases; it could also
be applied to protocol witnesses, so that those could be directly
dispatched like C++ vtables. (I don't think we want to restrict reordering
of protocol requirements, as much as it would make our lives easier.) So if
anyone has any brilliant ideas, Slava and I would love to hear them!

Jordan

Message: 2
Date: Fri, 29 Sep 2017 18:21:44 -0700
From: Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>>
To: swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>>
Subject: [swift-dev] What can you change in a non-exhaustive enum?
Message-ID: <31DF689E-1AD3-47CB-9FCE-6CBC7E34BC43@apple.com <mailto:31DF689E-1AD3-47CB-9FCE-6CBC7E34BC43@apple.com>>
Content-Type: text/plain; charset="utf-8"

Hello again, swift-dev! This is a sort of follow-up to "What can you change in a fixed-contents struct" from a few weeks ago, but this time concerning enums. Worryingly, this seems to be an important consideration even for non-exhaustive enums, which are normally the ones where we'd want to allow a developer to do anything and everything that doesn't break source compatibility.

[This only affects libraries with binary compatibility concerns. Libraries distributed with an app can always allow the app to access the enum's representation directly. That makes this an Apple-centric problem in the near term.]

So, what's the issue? We want to make it efficient to switch over a non-exhaustive enum, even from a client library that doesn't have access to the enum's guts. We do this by asking for the enum's tag separately from its payload (pseudocode):

switch getMyOpaqueEnumTag(&myOpaqueEnum) {
case 0:
  var payload: Int
  getMyOpaqueEnumPayload(&myOpaqueEnum, 0, &payload)
  doSomething(payload)
case 1:
  var payload: String
  getMyOpaqueEnumPayload(&myOpaqueEnum, 1, &payload)
  doSomethingElse(payload)
default:
  print("unknown case")
}

The tricky part is those constant values "0" and "1". We'd really like them to be constants so that the calling code can actually emit a jump table rather than a series of chained conditionals, but that means case tags are part of the ABI, even for non-exhaustive enums.

Like with struct layout, this means we need a stable ordering for cases. Since non-exhaustive enums can have new cases added at any time, we can't do a simple alphabetical sort, nor can we do some kind of ordering on the payload types. The naive answer, then, is that enum cases cannot be reordered, even in non-exhaustive enums. This isn't great, because people like being able to move deprecated enum cases to the end of the list, where they're out of the way, but it's at least explainable, and consistent with the idea of enums some day having a 'cases' property that includes all cases.

Slava and I aren't happy with this, but we haven't thought of another solution yet. The rest of this email will describe our previous idea, which has a fatal flaw.

Availability Ordering

In a library with binary compatibility concerns, any new API that gets added should always be explicitly annotated with an availability attribute. Today that looks like this:

@available(macOS 10.13, iOS 11, tvOS 11, watchOS 4, *)

It's a model we only support for Apple platforms, but in theory it's extendable to arbitrary "deployments". You ought to be able to say `@available(MagicKit 5)` and have the compiler actually check that.

Let's say we had this model, and we were using it like this:

public enum SpellKind {
  case hex
  case charm
  case curse
  @available(MagicKit 5)
  case blight
  @available(MagicKit 5.1)
  case jinx
}

"Availability ordering" says that we can derive a canonical ordering from the names of cases (which are API) combined with their versions. Since we "know" that newly-added cases will always have a newer version than existing cases, we can just put the older cases first. In this case, that would give us a canonical ordering of [charm, curse, hex, blight, jinx].

The Fatal Flaw

It's time for MagicKit 6 to come out, and we're going to add a new SpellKind:

@available(MagicKit 6)
case summoning
// [charm, curse, hex, blight, jinx, summoning]

We ship out a beta to our biggest clients, but realize we forgot a vital feature. Beta 2 comes with another new SpellKind:

@available(MagicKit 6)
case banishing
// [charm, curse, hex, blight, jinx, banishing, summoning]

And now we're in trouble: anything built against the first beta expects 'summoning' to have tag 5, not 6. Our clients have to recompile everything before they can even try out the new version of the library.

Can this be fixed? Sure. We could add support for beta versions to `@available`, or fake it somehow with the existing version syntax. But in both of these cases, it means you have to know what constitutes a "release", so that you can be sure to use a higher number than the previous "release". This could be made to work for a single library, but falls down for an entire Apple OS. If the Foundation team wants to add a second new enum case while macOS is still in beta, they're not going to stop and recompile all of /System/Library/Frameworks just to try out their change.

So, availability ordering is great when you have easily divisible "releases", but falls down when you want to make a change "during a release".

Salvaging Availability Ordering?

- We could still sort by availability, so that you can reorder the sections but not the individual cases in them. That doesn't seem very useful, though.

- We could say "this is probably rare", and state that anything added "in the same release" needs to get an explicit annotation for ordering purposes. (This is equivalent to the `@abi(2)` Dave Zarzycki mentioned in the previous thread—it's not the default but it's there if you need it.)

- We could actually require libraries to annotate all of their "releases", but in order to apply that within Apple we'd need some translation from library versions (like "Foundation 1258") to OS versions ("macOS 10.11.4"), and then we'd still need to figure out what to do about betas. (And there's a twist, at least at Apple, where a release's version number isn't decided until the new source code is submitted.)

- There might be something clever that I haven't thought of yet.

This kind of known ordering isn't just good for enum cases; it could also be applied to protocol witnesses, so that those could be directly dispatched like C++ vtables. (I don't think we want to restrict reordering of protocol requirements, as much as it would make our lives easier.) So if anyone has any brilliant ideas, Slava and I would love to hear them!

Jordan

Kind of a hybrid idea, but hopefully one that circumvents the internal issues that you outline here and simplifies the mental model. I'll introduce it stepwise:

Currently, the @available annotation is supported for platforms and for Swift versions; you propose extending support to arbitrary deployments (e.g., MagicKit). Instead, suppose you extended support to arbitrary versioned types (open, public, and @_versioned internal):

@_versioned(abi: 2)
public enum SpellKind {
  case hex, charm, curse
  @available(abi: SpellKind 1) case blight
  @available(abi: SpellKind 1.1) case jinx
}

Now, clearly, the value of "abi" is arbitrary inside the @_versioned annotation; it's not really necessary for our limited purposes, and if there's a typo and it's lower than the highest ABI version referenced in an @available annotation, things get wonky. So, drop it:

public enum SpellKind {
  case hex, charm, curse
  @available(abi: 1) case blight
  @available(abi: 1.1) case jinx
}

This is looking like the original @abi(2) proposal that Dave Zarzycki brought up. However, a key difference here: multiple cases can have the same ABI "version" and would be ordered relative to each other by name; that is, a user would only be annotating when new cases are added and doesn't have to think about memory layout; Swift takes care of the rest.

This can be simplified further; these ABI version numbers are entirely arbitrary. Suppose we instead extended the "@available(introduced:)" syntax to allow dates or timestamps:

public enum SpellKind {
  case hex, charm, curse
  @available(*, introduced: 2017-09-30): case blight
  @available(*, introduced: 2017-10-12): case jinx
}

I suspect there are wrinkles to this scheme, but the overall idea here is to salvage availability ordering but have some way to version a type in a low-mental-overhead way instead of resorting to a syntax for manually ordering cases.

Pretty much. If you want a stable order, you need to manually annotate each change to the enum’s layout with some incrementing identifier which can be sorted. Ideally, we would use the module’s version number, but as Jordan points out, during development layout can change even between versions.

So the simplest answer to me is to give each enum it’s own mini version number. You could write them out, as in your examples. Another approach might be to introduce a version-break statement in the case-list, with the condition that those version-breaks cannot be reordered, but everything inside of them can be:

enum SpellKind {
    case hex, charm, curse
    @new-version case blight
    @new-version
     case jinx
     case summoning
     case ...
}

A library developer could add as many of those as they need for internal testing, and selectively remove them as appropriate for each Beta/GM release (or they might keep them, if they decide to promise binary compatibility between beta and GM).

- Karl

···

On 30. Sep 2017, at 20:23, Xiaodi Wu via swift-dev <swift-dev@swift.org> wrote:
On Sat, Sep 30, 2017 at 11:58 AM, <swift-dev-request@swift.org <mailto:swift-dev-request@swift.org>> wrote:

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Ah I suppose this wouldn’t really work with @available since you can’t group cases in to a single availability context to provide a stable ordering within. So you will have to write the number out.

Still, I don’t think there is any way to do it other than an arbitrary, enum-scoped build/release number. It doesn’t mean anything other than to provide a level of ABI versioning below the module’s version number, like @abi(2). I don’t agree with supporting arbitrary tags or dates: they need to obviously increment, with minimal opportunities to get it wrong.

enum SpellKind {
    case hex
    case charm
    case curse
    @available(MagicKit 5) case blight
    @available(MagicKit 5.1) case jinx
    @available(MagicKit 6) case summoning
    @available(MagicKit 6, build: 2) case banishing
}

- Karl

···

On 4. Oct 2017, at 01:37, Karl Wagner <razielim@gmail.com> wrote:

On 30. Sep 2017, at 20:23, Xiaodi Wu via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

On Sat, Sep 30, 2017 at 11:58 AM, <swift-dev-request@swift.org <mailto:swift-dev-request@swift.org>> wrote:
Message: 2
Date: Fri, 29 Sep 2017 18:21:44 -0700
From: Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>>
To: swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>>
Subject: [swift-dev] What can you change in a non-exhaustive enum?
Message-ID: <31DF689E-1AD3-47CB-9FCE-6CBC7E34BC43@apple.com <mailto:31DF689E-1AD3-47CB-9FCE-6CBC7E34BC43@apple.com>>
Content-Type: text/plain; charset="utf-8"

Hello again, swift-dev! This is a sort of follow-up to "What can you change in a fixed-contents struct" from a few weeks ago, but this time concerning enums. Worryingly, this seems to be an important consideration even for non-exhaustive enums, which are normally the ones where we'd want to allow a developer to do anything and everything that doesn't break source compatibility.

[This only affects libraries with binary compatibility concerns. Libraries distributed with an app can always allow the app to access the enum's representation directly. That makes this an Apple-centric problem in the near term.]

So, what's the issue? We want to make it efficient to switch over a non-exhaustive enum, even from a client library that doesn't have access to the enum's guts. We do this by asking for the enum's tag separately from its payload (pseudocode):

switch getMyOpaqueEnumTag(&myOpaqueEnum) {
case 0:
  var payload: Int
  getMyOpaqueEnumPayload(&myOpaqueEnum, 0, &payload)
  doSomething(payload)
case 1:
  var payload: String
  getMyOpaqueEnumPayload(&myOpaqueEnum, 1, &payload)
  doSomethingElse(payload)
default:
  print("unknown case")
}

The tricky part is those constant values "0" and "1". We'd really like them to be constants so that the calling code can actually emit a jump table rather than a series of chained conditionals, but that means case tags are part of the ABI, even for non-exhaustive enums.

Like with struct layout, this means we need a stable ordering for cases. Since non-exhaustive enums can have new cases added at any time, we can't do a simple alphabetical sort, nor can we do some kind of ordering on the payload types. The naive answer, then, is that enum cases cannot be reordered, even in non-exhaustive enums. This isn't great, because people like being able to move deprecated enum cases to the end of the list, where they're out of the way, but it's at least explainable, and consistent with the idea of enums some day having a 'cases' property that includes all cases.

Slava and I aren't happy with this, but we haven't thought of another solution yet. The rest of this email will describe our previous idea, which has a fatal flaw.

Availability Ordering

In a library with binary compatibility concerns, any new API that gets added should always be explicitly annotated with an availability attribute. Today that looks like this:

@available(macOS 10.13, iOS 11, tvOS 11, watchOS 4, *)

It's a model we only support for Apple platforms, but in theory it's extendable to arbitrary "deployments". You ought to be able to say `@available(MagicKit 5)` and have the compiler actually check that.

Let's say we had this model, and we were using it like this:

public enum SpellKind {
  case hex
  case charm
  case curse
  @available(MagicKit 5)
  case blight
  @available(MagicKit 5.1)
  case jinx
}

"Availability ordering" says that we can derive a canonical ordering from the names of cases (which are API) combined with their versions. Since we "know" that newly-added cases will always have a newer version than existing cases, we can just put the older cases first. In this case, that would give us a canonical ordering of [charm, curse, hex, blight, jinx].

The Fatal Flaw

It's time for MagicKit 6 to come out, and we're going to add a new SpellKind:

@available(MagicKit 6)
case summoning
// [charm, curse, hex, blight, jinx, summoning]

We ship out a beta to our biggest clients, but realize we forgot a vital feature. Beta 2 comes with another new SpellKind:

@available(MagicKit 6)
case banishing
// [charm, curse, hex, blight, jinx, banishing, summoning]

And now we're in trouble: anything built against the first beta expects 'summoning' to have tag 5, not 6. Our clients have to recompile everything before they can even try out the new version of the library.

Can this be fixed? Sure. We could add support for beta versions to `@available`, or fake it somehow with the existing version syntax. But in both of these cases, it means you have to know what constitutes a "release", so that you can be sure to use a higher number than the previous "release". This could be made to work for a single library, but falls down for an entire Apple OS. If the Foundation team wants to add a second new enum case while macOS is still in beta, they're not going to stop and recompile all of /System/Library/Frameworks just to try out their change.

So, availability ordering is great when you have easily divisible "releases", but falls down when you want to make a change "during a release".

Salvaging Availability Ordering?

- We could still sort by availability, so that you can reorder the sections but not the individual cases in them. That doesn't seem very useful, though.

- We could say "this is probably rare", and state that anything added "in the same release" needs to get an explicit annotation for ordering purposes. (This is equivalent to the `@abi(2)` Dave Zarzycki mentioned in the previous thread—it's not the default but it's there if you need it.)

- We could actually require libraries to annotate all of their "releases", but in order to apply that within Apple we'd need some translation from library versions (like "Foundation 1258") to OS versions ("macOS 10.11.4"), and then we'd still need to figure out what to do about betas. (And there's a twist, at least at Apple, where a release's version number isn't decided until the new source code is submitted.)

- There might be something clever that I haven't thought of yet.

This kind of known ordering isn't just good for enum cases; it could also be applied to protocol witnesses, so that those could be directly dispatched like C++ vtables. (I don't think we want to restrict reordering of protocol requirements, as much as it would make our lives easier.) So if anyone has any brilliant ideas, Slava and I would love to hear them!

Jordan

Kind of a hybrid idea, but hopefully one that circumvents the internal issues that you outline here and simplifies the mental model. I'll introduce it stepwise:

Currently, the @available annotation is supported for platforms and for Swift versions; you propose extending support to arbitrary deployments (e.g., MagicKit). Instead, suppose you extended support to arbitrary versioned types (open, public, and @_versioned internal):

@_versioned(abi: 2)
public enum SpellKind {
  case hex, charm, curse
  @available(abi: SpellKind 1) case blight
  @available(abi: SpellKind 1.1) case jinx
}

Now, clearly, the value of "abi" is arbitrary inside the @_versioned annotation; it's not really necessary for our limited purposes, and if there's a typo and it's lower than the highest ABI version referenced in an @available annotation, things get wonky. So, drop it:

public enum SpellKind {
  case hex, charm, curse
  @available(abi: 1) case blight
  @available(abi: 1.1) case jinx
}

This is looking like the original @abi(2) proposal that Dave Zarzycki brought up. However, a key difference here: multiple cases can have the same ABI "version" and would be ordered relative to each other by name; that is, a user would only be annotating when new cases are added and doesn't have to think about memory layout; Swift takes care of the rest.

This can be simplified further; these ABI version numbers are entirely arbitrary. Suppose we instead extended the "@available(introduced:)" syntax to allow dates or timestamps:

public enum SpellKind {
  case hex, charm, curse
  @available(*, introduced: 2017-09-30): case blight
  @available(*, introduced: 2017-10-12): case jinx
}

I suspect there are wrinkles to this scheme, but the overall idea here is to salvage availability ordering but have some way to version a type in a low-mental-overhead way instead of resorting to a syntax for manually ordering cases.

Pretty much. If you want a stable order, you need to manually annotate each change to the enum’s layout with some incrementing identifier which can be sorted. Ideally, we would use the module’s version number, but as Jordan points out, during development layout can change even between versions.

So the simplest answer to me is to give each enum it’s own mini version number. You could write them out, as in your examples. Another approach might be to introduce a version-break statement in the case-list, with the condition that those version-breaks cannot be reordered, but everything inside of them can be:

enum SpellKind {
    case hex, charm, curse
    @new-version case blight
    @new-version
     case jinx
     case summoning
     case ...
}

A library developer could add as many of those as they need for internal testing, and selectively remove them as appropriate for each Beta/GM release (or they might keep them, if they decide to promise binary compatibility between beta and GM).

- Karl

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

So, an update! This came up while I was talking to members of the core team, and ChrisL felt very strongly that restricting reordering of enum elements was a no-go, since it would be the only part of the language that worked this way (even if it only mattered for binary frameworks). Ted also rightly pointed out that any such language-level restriction would have to be reviewed by the core team.

So where does that leave us?

- The naive implementation is to turn a switch into an if-else chain, unfortunately requiring one function call per case to match.

- A slightly more complex solution keeps a single 'getMyOpaqueEnumTag' entry point (see original email), but exposes symbols for every tag. The values of the symbols would be kept in alphabetical order, which allows the generated code to do a binary search over the cases they care about. This still means N symbols, but a switch that involves several of them doesn't necessarily have to take linear time.

- Joe Groff came up with this idea that also involves sorted symbols:

switch indexForMyOpaqueEnumTag(&myOpaqueEnum, [MyOpaqueEnum.intCase, MyOpaqueEnum.stringCase]) {
case 0:
  var payload: Int
  getMyOpaqueEnumPayload(&myOpaqueEnum, MyOpaqueEnum.intCase, &payload)
  doSomething(payload)
case 1:
  var payload: String
  getMyOpaqueEnumPayload(&myOpaqueEnum, MyOpaqueEnum.stringCase, &payload)
  doSomethingElse(payload)
default:
  print("unknown case")
}

In this example, the actual tag values for 'intCase' and 'stringCase' might not be 0 and 1, but 'indexForMyOpaqueEnumTag' can do the binary search to find out which enum we're asking for. Like the previous solution, you only have to check the cases you care about, but this time the binary search is in the callee, rather than the client.

- Use availability ordering, plus some kind of explicit annotation for when multiple cases are added within the same release. (In this thread people have suggested dates, ad-hoc sub-version numbers, and plain integer values.)

I appreciate everyone's creativity with solving the availability ordering problem, but I don't want to tie us to a checker that will tell you if you screw up or a history scraper that will implicitly add the right annotations. (I don't think those are bad ideas, but they're a whole extra pile of work on top of the main implementation!) That leaves explicit annotations of some kind, and that leaves us in a worse place than Objective-C. Which is permitted, but not desirable.

At this point in time I think the second option is the best one we have: it's relatively simple to implement, it supports everything Objective-C does, and it doesn't make the availability model even more complicated. It is going to be less efficient than actually knowing the case numbers at compile time, though. Still, as Slava's pointed out, we can still change this after we go ABI-stable; doing something more efficient will just be limited based on deployment target.

Jordan

I like Joe’s idea here, with the extension that the client should have just one of these arrays that contains all the symbols that it knows about at the time it was compiled:

I.e. in the client:

static myKnownOpaqueEnumCases = [MyOpaqueEnum.intCase, MyOpaqueEnum.middleCase, MyOpaqueEnum.stringCase];

switch indexForMyOpaqueEnumTag(&myOpaqueEnum, myKnownOpaqueEnumCases) {
case 0: //…
case 2: //…
default: //...
}

This optimizes for space in the client, because you have one array instead of one per potentially-different-sets-of-cases switch, but more importantly this allows for an optimization inside indexForMyOpaqueEnumTag(). If the count of the array passed in from the client is equal to the count of all known cases in the callee, then you can immediately return the internal enum tag value instead of performing a binary search.

(If the client expects cases that the callee doesn’t have, the link would have failed for a missing symbol, if the callee has more cases the count won’t match, so if the count is equal the cases in both object files have to be identical.) This returns the common runtime case (when the client is up to date with the callee) to being O(1).

The cost being, if you don’t take that fast path, maybe you have a few more entries in the cases array to binary search over than that particular switch statement needed.

- Greg

···

On Oct 12, 2017, at 2:25 PM, Jordan Rose <jordan_rose@apple.com> wrote:

So, an update! This came up while I was talking to members of the core team, and ChrisL felt very strongly that restricting reordering of enum elements was a no-go, since it would be the only part of the language that worked this way (even if it only mattered for binary frameworks). Ted also rightly pointed out that any such language-level restriction would have to be reviewed by the core team.

So where does that leave us?

- The naive implementation is to turn a switch into an if-else chain, unfortunately requiring one function call per case to match.

- A slightly more complex solution keeps a single 'getMyOpaqueEnumTag' entry point (see original email), but exposes symbols for every tag. The values of the symbols would be kept in alphabetical order, which allows the generated code to do a binary search over the cases they care about. This still means N symbols, but a switch that involves several of them doesn't necessarily have to take linear time.

- Joe Groff came up with this idea that also involves sorted symbols:

switch indexForMyOpaqueEnumTag(&myOpaqueEnum, [MyOpaqueEnum.intCase, MyOpaqueEnum.stringCase]) {
case 0:
  var payload: Int
  getMyOpaqueEnumPayload(&myOpaqueEnum, MyOpaqueEnum.intCase, &payload)
  doSomething(payload)
case 1:
  var payload: String
  getMyOpaqueEnumPayload(&myOpaqueEnum, MyOpaqueEnum.stringCase, &payload)
  doSomethingElse(payload)
default:
  print("unknown case")
}

In this example, the actual tag values for 'intCase' and 'stringCase' might not be 0 and 1, but 'indexForMyOpaqueEnumTag' can do the binary search to find out which enum we're asking for. Like the previous solution, you only have to check the cases you care about, but this time the binary search is in the callee, rather than the client.

- Use availability ordering, plus some kind of explicit annotation for when multiple cases are added within the same release. (In this thread people have suggested dates, ad-hoc sub-version numbers, and plain integer values.)

I appreciate everyone's creativity with solving the availability ordering problem, but I don't want to tie us to a checker that will tell you if you screw up or a history scraper that will implicitly add the right annotations. (I don't think those are bad ideas, but they're a whole extra pile of work on top of the main implementation!) That leaves explicit annotations of some kind, and that leaves us in a worse place than Objective-C. Which is permitted, but not desirable.

At this point in time I think the second option is the best one we have: it's relatively simple to implement, it supports everything Objective-C does, and it doesn't make the availability model even more complicated. It is going to be less efficient than actually knowing the case numbers at compile time, though. Still, as Slava's pointed out, we can still change this after we go ABI-stable; doing something more efficient will just be limited based on deployment target.

Jordan

Our worry when discussing it was that someone might have an autogenerated 1000-case enum, and passing an entire page worth of resolved symbols might not be worth it.

(It is of course questionable for someone to have an autogenerated 1000-case enum as part of their binary interface, and then for someone to try to switch over it. But matching against one or two of the cases shouldn't be as expensive as matching against all 1000 known cases.)

Jordan

···

On Oct 12, 2017, at 15:20, Greg Titus <greg@omnigroup.com> wrote:

I like Joe’s idea here, with the extension that the client should have just one of these arrays that contains all the symbols that it knows about at the time it was compiled:

I.e. in the client:

static myKnownOpaqueEnumCases = [MyOpaqueEnum.intCase, MyOpaqueEnum.middleCase, MyOpaqueEnum.stringCase];

switch indexForMyOpaqueEnumTag(&myOpaqueEnum, myKnownOpaqueEnumCases) {
case 0: //…
case 2: //…
default: //...
}

This optimizes for space in the client, because you have one array instead of one per potentially-different-sets-of-cases switch, but more importantly this allows for an optimization inside indexForMyOpaqueEnumTag(). If the count of the array passed in from the client is equal to the count of all known cases in the callee, then you can immediately return the internal enum tag value instead of performing a binary search.

(If the client expects cases that the callee doesn’t have, the link would have failed for a missing symbol, if the callee has more cases the count won’t match, so if the count is equal the cases in both object files have to be identical.) This returns the common runtime case (when the client is up to date with the callee) to being O(1).

The cost being, if you don’t take that fast path, maybe you have a few more entries in the cases array to binary search over than that particular switch statement needed.

- Greg

On Oct 12, 2017, at 2:25 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:

So, an update! This came up while I was talking to members of the core team, and ChrisL felt very strongly that restricting reordering of enum elements was a no-go, since it would be the only part of the language that worked this way (even if it only mattered for binary frameworks). Ted also rightly pointed out that any such language-level restriction would have to be reviewed by the core team.

So where does that leave us?

- The naive implementation is to turn a switch into an if-else chain, unfortunately requiring one function call per case to match.

- A slightly more complex solution keeps a single 'getMyOpaqueEnumTag' entry point (see original email), but exposes symbols for every tag. The values of the symbols would be kept in alphabetical order, which allows the generated code to do a binary search over the cases they care about. This still means N symbols, but a switch that involves several of them doesn't necessarily have to take linear time.

- Joe Groff came up with this idea that also involves sorted symbols:

switch indexForMyOpaqueEnumTag(&myOpaqueEnum, [MyOpaqueEnum.intCase, MyOpaqueEnum.stringCase]) {
case 0:
  var payload: Int
  getMyOpaqueEnumPayload(&myOpaqueEnum, MyOpaqueEnum.intCase, &payload)
  doSomething(payload)
case 1:
  var payload: String
  getMyOpaqueEnumPayload(&myOpaqueEnum, MyOpaqueEnum.stringCase, &payload)
  doSomethingElse(payload)
default:
  print("unknown case")
}

In this example, the actual tag values for 'intCase' and 'stringCase' might not be 0 and 1, but 'indexForMyOpaqueEnumTag' can do the binary search to find out which enum we're asking for. Like the previous solution, you only have to check the cases you care about, but this time the binary search is in the callee, rather than the client.

- Use availability ordering, plus some kind of explicit annotation for when multiple cases are added within the same release. (In this thread people have suggested dates, ad-hoc sub-version numbers, and plain integer values.)

I appreciate everyone's creativity with solving the availability ordering problem, but I don't want to tie us to a checker that will tell you if you screw up or a history scraper that will implicitly add the right annotations. (I don't think those are bad ideas, but they're a whole extra pile of work on top of the main implementation!) That leaves explicit annotations of some kind, and that leaves us in a worse place than Objective-C. Which is permitted, but not desirable.

At this point in time I think the second option is the best one we have: it's relatively simple to implement, it supports everything Objective-C does, and it doesn't make the availability model even more complicated. It is going to be less efficient than actually knowing the case numbers at compile time, though. Still, as Slava's pointed out, we can still change this after we go ABI-stable; doing something more efficient will just be limited based on deployment target.

Jordan

The nice thing about this setup is that it degrades nicely for the 1000-case enum. You have a static check in the compiler for if theNumberOfKnownEnumCases >= SOME_REASONABLE_MAX (or — in the future — as complicated a decision as you want given the number of total cases and the number of cases you care about in that particular switch) then you pass in an array of just the switched against cases instead, and the implementation of indexForMyOpaqueEnumTag() remains exactly the same in the callee (the early out matching count fails, and you binary search).

- Greg

···

On Oct 12, 2017, at 3:38 PM, Jordan Rose <jordan_rose@apple.com> wrote:

Our worry when discussing it was that someone might have an autogenerated 1000-case enum, and passing an entire page worth of resolved symbols might not be worth it.

(It is of course questionable for someone to have an autogenerated 1000-case enum as part of their binary interface, and then for someone to try to switch over it. But matching against one or two of the cases shouldn't be as expensive as matching against all 1000 known cases.)

Jordan

On Oct 12, 2017, at 15:20, Greg Titus <greg@omnigroup.com <mailto:greg@omnigroup.com>> wrote:

I like Joe’s idea here, with the extension that the client should have just one of these arrays that contains all the symbols that it knows about at the time it was compiled:

I.e. in the client:

static myKnownOpaqueEnumCases = [MyOpaqueEnum.intCase, MyOpaqueEnum.middleCase, MyOpaqueEnum.stringCase];

switch indexForMyOpaqueEnumTag(&myOpaqueEnum, myKnownOpaqueEnumCases) {
case 0: //…
case 2: //…
default: //...
}

This optimizes for space in the client, because you have one array instead of one per potentially-different-sets-of-cases switch, but more importantly this allows for an optimization inside indexForMyOpaqueEnumTag(). If the count of the array passed in from the client is equal to the count of all known cases in the callee, then you can immediately return the internal enum tag value instead of performing a binary search.

(If the client expects cases that the callee doesn’t have, the link would have failed for a missing symbol, if the callee has more cases the count won’t match, so if the count is equal the cases in both object files have to be identical.) This returns the common runtime case (when the client is up to date with the callee) to being O(1).

The cost being, if you don’t take that fast path, maybe you have a few more entries in the cases array to binary search over than that particular switch statement needed.

- Greg

On Oct 12, 2017, at 2:25 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:

So, an update! This came up while I was talking to members of the core team, and ChrisL felt very strongly that restricting reordering of enum elements was a no-go, since it would be the only part of the language that worked this way (even if it only mattered for binary frameworks). Ted also rightly pointed out that any such language-level restriction would have to be reviewed by the core team.

So where does that leave us?

- The naive implementation is to turn a switch into an if-else chain, unfortunately requiring one function call per case to match.

- A slightly more complex solution keeps a single 'getMyOpaqueEnumTag' entry point (see original email), but exposes symbols for every tag. The values of the symbols would be kept in alphabetical order, which allows the generated code to do a binary search over the cases they care about. This still means N symbols, but a switch that involves several of them doesn't necessarily have to take linear time.

- Joe Groff came up with this idea that also involves sorted symbols:

switch indexForMyOpaqueEnumTag(&myOpaqueEnum, [MyOpaqueEnum.intCase, MyOpaqueEnum.stringCase]) {
case 0:
  var payload: Int
  getMyOpaqueEnumPayload(&myOpaqueEnum, MyOpaqueEnum.intCase, &payload)
  doSomething(payload)
case 1:
  var payload: String
  getMyOpaqueEnumPayload(&myOpaqueEnum, MyOpaqueEnum.stringCase, &payload)
  doSomethingElse(payload)
default:
  print("unknown case")
}

In this example, the actual tag values for 'intCase' and 'stringCase' might not be 0 and 1, but 'indexForMyOpaqueEnumTag' can do the binary search to find out which enum we're asking for. Like the previous solution, you only have to check the cases you care about, but this time the binary search is in the callee, rather than the client.

- Use availability ordering, plus some kind of explicit annotation for when multiple cases are added within the same release. (In this thread people have suggested dates, ad-hoc sub-version numbers, and plain integer values.)

I appreciate everyone's creativity with solving the availability ordering problem, but I don't want to tie us to a checker that will tell you if you screw up or a history scraper that will implicitly add the right annotations. (I don't think those are bad ideas, but they're a whole extra pile of work on top of the main implementation!) That leaves explicit annotations of some kind, and that leaves us in a worse place than Objective-C. Which is permitted, but not desirable.

At this point in time I think the second option is the best one we have: it's relatively simple to implement, it supports everything Objective-C does, and it doesn't make the availability model even more complicated. It is going to be less efficient than actually knowing the case numbers at compile time, though. Still, as Slava's pointed out, we can still change this after we go ABI-stable; doing something more efficient will just be limited based on deployment target.

Jordan

Hi Jordan,

Bertrand (ex-Apple) would sometimes say during API design that “easy things things should be easy, and hard things should be possible”.

I don’t think you guys need to go out of your way to make the autogenerated 1000-case enum scenario “easy". If people are clever enough to do that, then they can surely handle the fallout of this change. :slight_smile:

Dave

···

On Oct 12, 2017, at 18:38, Jordan Rose via swift-dev <swift-dev@swift.org> wrote:

Our worry when discussing it was that someone might have an autogenerated 1000-case enum, and passing an entire page worth of resolved symbols might not be worth it.

(It is of course questionable for someone to have an autogenerated 1000-case enum as part of their binary interface, and then for someone to try to switch over it. But matching against one or two of the cases shouldn't be as expensive as matching against all 1000 known cases.)

Jordan

On Oct 12, 2017, at 15:20, Greg Titus <greg@omnigroup.com <mailto:greg@omnigroup.com>> wrote:

I like Joe’s idea here, with the extension that the client should have just one of these arrays that contains all the symbols that it knows about at the time it was compiled:

I.e. in the client:

static myKnownOpaqueEnumCases = [MyOpaqueEnum.intCase, MyOpaqueEnum.middleCase, MyOpaqueEnum.stringCase];

switch indexForMyOpaqueEnumTag(&myOpaqueEnum, myKnownOpaqueEnumCases) {
case 0: //…
case 2: //…
default: //...
}

This optimizes for space in the client, because you have one array instead of one per potentially-different-sets-of-cases switch, but more importantly this allows for an optimization inside indexForMyOpaqueEnumTag(). If the count of the array passed in from the client is equal to the count of all known cases in the callee, then you can immediately return the internal enum tag value instead of performing a binary search.

(If the client expects cases that the callee doesn’t have, the link would have failed for a missing symbol, if the callee has more cases the count won’t match, so if the count is equal the cases in both object files have to be identical.) This returns the common runtime case (when the client is up to date with the callee) to being O(1).

The cost being, if you don’t take that fast path, maybe you have a few more entries in the cases array to binary search over than that particular switch statement needed.

- Greg

On Oct 12, 2017, at 2:25 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:

So, an update! This came up while I was talking to members of the core team, and ChrisL felt very strongly that restricting reordering of enum elements was a no-go, since it would be the only part of the language that worked this way (even if it only mattered for binary frameworks). Ted also rightly pointed out that any such language-level restriction would have to be reviewed by the core team.

So where does that leave us?

- The naive implementation is to turn a switch into an if-else chain, unfortunately requiring one function call per case to match.

- A slightly more complex solution keeps a single 'getMyOpaqueEnumTag' entry point (see original email), but exposes symbols for every tag. The values of the symbols would be kept in alphabetical order, which allows the generated code to do a binary search over the cases they care about. This still means N symbols, but a switch that involves several of them doesn't necessarily have to take linear time.

- Joe Groff came up with this idea that also involves sorted symbols:

switch indexForMyOpaqueEnumTag(&myOpaqueEnum, [MyOpaqueEnum.intCase, MyOpaqueEnum.stringCase]) {
case 0:
  var payload: Int
  getMyOpaqueEnumPayload(&myOpaqueEnum, MyOpaqueEnum.intCase, &payload)
  doSomething(payload)
case 1:
  var payload: String
  getMyOpaqueEnumPayload(&myOpaqueEnum, MyOpaqueEnum.stringCase, &payload)
  doSomethingElse(payload)
default:
  print("unknown case")
}

In this example, the actual tag values for 'intCase' and 'stringCase' might not be 0 and 1, but 'indexForMyOpaqueEnumTag' can do the binary search to find out which enum we're asking for. Like the previous solution, you only have to check the cases you care about, but this time the binary search is in the callee, rather than the client.

- Use availability ordering, plus some kind of explicit annotation for when multiple cases are added within the same release. (In this thread people have suggested dates, ad-hoc sub-version numbers, and plain integer values.)

I appreciate everyone's creativity with solving the availability ordering problem, but I don't want to tie us to a checker that will tell you if you screw up or a history scraper that will implicitly add the right annotations. (I don't think those are bad ideas, but they're a whole extra pile of work on top of the main implementation!) That leaves explicit annotations of some kind, and that leaves us in a worse place than Objective-C. Which is permitted, but not desirable.

At this point in time I think the second option is the best one we have: it's relatively simple to implement, it supports everything Objective-C does, and it doesn't make the availability model even more complicated. It is going to be less efficient than actually knowing the case numbers at compile time, though. Still, as Slava's pointed out, we can still change this after we go ABI-stable; doing something more efficient will just be limited based on deployment target.

Jordan

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev