Would refactoring Metatypes affect ABI?

DevAndArtist · March 4, 2018, 1:23pm

I tried to understand the ABI manifesto yet I failed to understand whether or not refactoring metatypes would affect ABI. From my perspective metatypes is an underestimated topic. No one really cares about metatypes, yet most of us use them. Furthermore I don't understand why metatypes almost in every conversation are only associated with reflections, where the truth is that metatypes can do a lot more than just allow new possibilities for reflections. In fact I'm more interested in the smaller things:

Fixing magical type(of:) function
Allowing subtype(of:named:) function (there was some work done letely that could be re-used for that)
Removing the last value-style syntax .Type and .Protocol from the language (not counting .self here)
Allowing custom nested types to be called Type and Protocol
Making the difference between static and dynamic metatypes and their partly covariant relation clear
Merging MemoryLayout directly into static matatypes
We could allow extensions on static metatype and perhaps on dynamic metatype to add members or conform them to protocols. This could allow us to remove some standard library hacks, like the non-Equatable-related == operators for types.
Conforming static metatypes to Hashable would allow us to use metatypes for concrete types as hashable keys.
```
let dict: [AnyHashable: String] = [
  Int.self: "so cool", 
  String.self, "even cooler"
]
```
Fixing issues in generic and associated type contexts
Allowing new generic constraints where one can describe the relationship of a concrete existentials
Removing a boundary which was the source for a lot of confusion:
- Confused by generic behaviour, not clear if it is a bug
- Support for protocols as parameters for constrained generics
- Error: ‘MyProtocol’ is not convertible to ‘AnyObject’
- Issue with typealias of multiple protocols
- SR-6109: AnyObject as associated type constraint causes EXC_BAD_ACCESS
- Some conversations from Twitter (I hope you guys don't mind for linking it here, if you do mind I'll remove it):
  - Conversation #1
  - Conversation #2
Extending with reflection capabilities in the future

I'm sure there are more topics and issues related to metatypes. That is the reason why I call metatypes the underestimated topic. It would be really great if someone could clarify the main question about ABI, because I have a feeling that after ABI is declared stable that the chance to fix all those issues will be gone forever.

Decisions about the ABI will have long-term ramifications and may limit the ways in which the language can grow and evolve in the future. Future Swift versions can add new, orthogonal aspects to the ABI, but any inefficiencies or inflexibilities present when stability is declared will (effectively) persist forever for that platform.

Source: ABI Stability Manifesto

Edit:

Here is the proposal where @beccadax helped us a lot revising it after we withdrew our first rushed attempt two years ago before Swift 3 came out. The revised version is mostly theoretical and can be a little confusing at first glance, so be prepared when reading. If you have any questions regarding the proposal you can always ask. I will try to answer them as good as I can.

Short version of the proposal (click to unfold)

static metatype Type<T> (was T.Type and T.Protocol before)
dynamic metatype AnyType<T> (was T.Type before)
in current Swift static and dynamic metatype are more or less merged
partly covariant relation:
- for protocols and existentials:
  - Type<SomeProto> : AnyType<Any> (this mimics T.Protocol behavior)
  - AnyType<SomeProto> : AnyType<Any>
- other types (except functions - please read the proposal to find out why functions are a little different)
  - Type<T> : AnyType<T> (note the difference to protocols and existentials)
  - AnyType<T> : AnyType<Any>
- AnyType<T> is a subtype of itself, so the following is true: AnyType<T> : AnyType<T>

hartbit · March 6, 2018, 7:02am

I'd be very interested to find out how feasible this will be in the future post-ABI stability.

anthonylatsis · March 7, 2018, 5:27pm

As far as I understand, a change will affect ABI stability if there is no general mapping from your ideas' implementation into an equivalent implementation for previous language versions.

Considering your proposal introduces new ways of expressing type constraints, I assume it does affect ABI stability.

DevAndArtist · March 15, 2018, 10:56am

Dear @Chris_Lattner3 since you've started creating the Swift language, you may know the roots of the metatypes that we're currently dealing with. I would really appreciate if you or any other core team member could clarify what happens to this particular topic post ABI stability or if refactoring metatypes is a necessary step before ABI stability is declared. Since not every contributor does know everything about the language I'd happy to see if someone could at least CC this thread to the developer that worked on the current metatypes, so that we can get some more insights.

Chris_Lattner3 · March 15, 2018, 11:47pm

Hi Adrian,

I'm not super up to date with the ABI issues in the recently changing work for ABI stability. I'd ask folks like @Joe_Groff @John_McCall and @Slava_Pestov. If they don't know themselves, they will know who would know
-Chris

John_McCall · March 16, 2018, 1:21pm

The only parts of this which affect ABI stability, I think, are (1) the part which changes the type system by introducing different kinds of metatypes and, relatedly, (2) the decision about what the type of Self is in a static class method or property. The former would need to be expressible in runtime type metadata, which it currently is not; the latter might change how a bunch of code is compiled in ways that can’t be retrofitted.

I would still be very interested in getting this change for 5.0.

Joe_Groff · March 16, 2018, 3:58pm

I don't think we fundamentally need any new kinds of metatype than currently exist, and we should resist adding new kinds of "magic" types if we can. If we were going to change how information about types gets represented in API, that should be doable using structs that hold metatype values.

anthonylatsis · March 16, 2018, 9:04pm

There is no question of magic types, rather, the magic types you are talking about is in some sense what we currently have. The idea is to reconsider the metatype system so that it becomes more complete, exposed and hence able to be exploited. Currently the metatype system is relatively incomplete and therefore mostly hidden from the consumer. We can only use metatypes explicitly for passing types as instances (.Type, not counting .self, is all we have control over). Refactoring them gives Swift an opportunity for and is key to a huge leap in generics and abstract type usage, as well as further solidifying the STL and fixing some unstable and problematic APIs that @DevAndArtist mentioned in the root post. In terms of evolution, it is a great and relevant proposal.

Joe_Groff · March 16, 2018, 9:08pm

I'm not debating the merits of his proposal. I'm saying I don't think anything involving metatypes needs any more information than we already pass at the ABI level. A metatype's representation is a pointer to its type metadata record, and whatever information you want to find out about the type at runtime can be derived from that.

anthonylatsis · March 17, 2018, 12:19am

Pardon for misunderstanding. So you believe it will be possible to constrain a generic parameter to protocol-only types using the metatypes we currently have?

Joe_Groff · March 17, 2018, 12:22am

"Is a protocol" is a constraint that wouldn't need any additional information to "witness" the requirement. We do anticipate that there will be type system features we add in future Swifts that will need runtime support to fully implement, and we hope to provide some ability to back-deploy that kind of thing into OSes running older runtimes.

anthonylatsis · March 17, 2018, 12:34am

I see. Thanks for pointing that out.