ABI in Layman's terms?

Evolution Discussion
1

Could someone explain in simple terms what the ABI is and what sorts of things might affect it?

I had thought it was the layout in memory of structs/classes/etc… and how the program knows where to go to find a particular field. This seems to be incorrect though, as I have seen many features that I would assume have some affect on this layout ruled “out of scope for phase 1”. For example, I would think that many generics features would have an impact on the ABI, or the idea of COW (via secret boxing) for structs, or even union types.

At the very least, I would think that mix-ins would have a fairly significant effect.

I am obviously missing something here, and I want to provide constructive effort to the community (as opposed to distracting from the task at hand), so I would appreciate some clarification/guidance...

Thanks,
Jon

2

Could someone explain in simple terms what the ABI is and what sorts of things might affect it?

The ABI for a compiled programming language is the set of rules for how all of its interoperating features are implemented in the compiled result.

For example, in Java the primary ABI is the JVM specification plus some common-sense rules about how language features are mapped to JVM features. JVMs do not typically interoperate with other code at a direct binary level, and so the details of actual memory layout are not ABI.

Not all language features require interoperation, or only require it a certain kind of opaque interoperation. For example:

  - Type aliases do not currently have any ABI impact because by design they are erased at compile time. However, a complete reflection design might provide some mechanism for introspecting type-aliases at runtime, which would require information about type aliases to be present in the compiled output; the representation of that information would be ABI.

  - Closures in most programming languages do not allow external code to modify their captured values; only the closure's invocation function can access them, and that is generated together with the capturing code, so the exact methodology of performing a capture is not ABI. However, arbitrary code can invoke the closure, and so the opaque representation of a closure value and the process for invoking it are ABI.

And so on.

John.

···

On Aug 11, 2016, at 1:52 AM, Jonathan Hull via swift-evolution <swift-evolution@swift.org> wrote:

I had thought it was the layout in memory of structs/classes/etc… and how the program knows where to go to find a particular field. This seems to be incorrect though, as I have seen many features that I would assume have some affect on this layout ruled “out of scope for phase 1”. For example, I would think that many generics features would have an impact on the ABI, or the idea of COW (via secret boxing) for structs, or even union types.

At the very least, I would think that mix-ins would have a fairly significant effect.

I am obviously missing something here, and I want to provide constructive effort to the community (as opposed to distracting from the task at hand), so I would appreciate some clarification/guidance...

Thanks,
Jon
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

3

ABI stability means that changes will have to be backwards compatible after
a certain stage. If we can add mixins feature without modifying old code
(and its SIL and IR and whatever), then we are fine.

···

2016-08-11 11:52 GMT+03:00 Jonathan Hull via swift-evolution < swift-evolution@swift.org>:

Could someone explain in simple terms what the ABI is and what sorts of
things might affect it?

I had thought it was the layout in memory of structs/classes/etc… and how
the program knows where to go to find a particular field. This seems to be
incorrect though, as I have seen many features that I would assume have
some affect on this layout ruled “out of scope for phase 1”. For example,
I would think that many generics features would have an impact on the ABI,
or the idea of COW (via secret boxing) for structs, or even union types.

At the very least, I would think that mix-ins would have a fairly
significant effect.

4

Right, I understand that part. ABI stability means that we can run past binaries without recompiling.

What I want to know is what sorts of things affect this stability and which things don’t. I guess I want heuristics for knowing whether an idea will affect the ABI, and thus needs to be talked about now. I don’t want to distract from the process by proposing features which can be tackled later, but at the same time I REALLY don’t want to wait until phase 2 and then be told the feature can never be added because it would break the ABI (and I should have proposed it during phase 1).

Thanks,
Jon

···

On Aug 11, 2016, at 4:24 AM, Anton Zhilin <antonyzhilin@gmail.com> wrote:

2016-08-11 11:52 GMT+03:00 Jonathan Hull via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>>:
Could someone explain in simple terms what the ABI is and what sorts of things might affect it?

I had thought it was the layout in memory of structs/classes/etc… and how the program knows where to go to find a particular field. This seems to be incorrect though, as I have seen many features that I would assume have some affect on this layout ruled “out of scope for phase 1”. For example, I would think that many generics features would have an impact on the ABI, or the idea of COW (via secret boxing) for structs, or even union types.

At the very least, I would think that mix-ins would have a fairly significant effect.

ABI stability means that changes will have to be backwards compatible after a certain stage. If we can add mixins feature without modifying old code (and its SIL and IR and whatever), then we are fine.

5

I'd also like to understand this more and this answer does not completely satisfy me. I understand backwards compatibility, especially in terms of source breaking changes.

I have more difficulties understanding what breaks or not the ABI and how to make educated guesses about what features will require breaking it.

···

On 11 Aug 2016, at 13:24, Anton Zhilin via swift-evolution <swift-evolution@swift.org> wrote:

2016-08-11 11:52 GMT+03:00 Jonathan Hull via swift-evolution <swift-evolution@swift.org>:

Could someone explain in simple terms what the ABI is and what sorts of things might affect it?

I had thought it was the layout in memory of structs/classes/etc… and how the program knows where to go to find a particular field. This seems to be incorrect though, as I have seen many features that I would assume have some affect on this layout ruled “out of scope for phase 1”. For example, I would think that many generics features would have an impact on the ABI, or the idea of COW (via secret boxing) for structs, or even union types.

At the very least, I would think that mix-ins would have a fairly significant effect.

ABI stability means that changes will have to be backwards compatible after a certain stage. If we can add mixins feature without modifying old code (and its SIL and IR and whatever), then we are fine.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

6

This article has helped me a lot:

https://github.com/apple/swift/blob/master/docs/ABI.rst

And generally all the documents in the docs folder will paint a nice picture of what to expect. Not that I would say that I fully understand the ABI, but I think that after reading all the docs I will have a better way of understanding what to expect.

···

On Aug 11, 2016, at 1:33 PM, Jonathan Hull via swift-evolution <swift-evolution@swift.org> wrote:

Right, I understand that part. ABI stability means that we can run past binaries without recompiling.

What I want to know is what sorts of things affect this stability and which things don’t. I guess I want heuristics for knowing whether an idea will affect the ABI, and thus needs to be talked about now. I don’t want to distract from the process by proposing features which can be tackled later, but at the same time I REALLY don’t want to wait until phase 2 and then be told the feature can never be added because it would break the ABI (and I should have proposed it during phase 1).

Thanks,
Jon

On Aug 11, 2016, at 4:24 AM, Anton Zhilin <antonyzhilin@gmail.com <mailto:antonyzhilin@gmail.com>> wrote:

2016-08-11 11:52 GMT+03:00 Jonathan Hull via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>>:
Could someone explain in simple terms what the ABI is and what sorts of things might affect it?

I had thought it was the layout in memory of structs/classes/etc… and how the program knows where to go to find a particular field. This seems to be incorrect though, as I have seen many features that I would assume have some affect on this layout ruled “out of scope for phase 1”. For example, I would think that many generics features would have an impact on the ABI, or the idea of COW (via secret boxing) for structs, or even union types.

At the very least, I would think that mix-ins would have a fairly significant effect.

ABI stability means that changes will have to be backwards compatible after a certain stage. If we can add mixins feature without modifying old code (and its SIL and IR and whatever), then we are fine.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

7

I'd also like to understand this more and this answer does not completely satisfy me. I understand backwards compatibility, especially in terms of source breaking changes.

I have more difficulties understanding what breaks or not the ABI and how to make educated guesses about what features will require breaking it

This is hard to explain right now without detailed knowledge of IRGen and SILGen internals. However we are planning on relaxing the restrictions as much as possible, so that source-compatible changes remain binary compatible. For example, we would like to be able to add new fields to structs, change computed properties to stored and vice versa, insert new classes in a hierarchy, add cases to enums, and so on. This falls under the umbrella of "resilience".

Here's a document outlining what will be ABI compatible and what will not -- keep in mind that a good chunk of this is not yet implemented:

···

On Aug 11, 2016, at 4:27 AM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:

On 11 Aug 2016, at 13:24, Anton Zhilin via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

2016-08-11 11:52 GMT+03:00 Jonathan Hull via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>>:
Could someone explain in simple terms what the ABI is and what sorts of things might affect it?

I had thought it was the layout in memory of structs/classes/etc… and how the program knows where to go to find a particular field. This seems to be incorrect though, as I have seen many features that I would assume have some affect on this layout ruled “out of scope for phase 1”. For example, I would think that many generics features would have an impact on the ABI, or the idea of COW (via secret boxing) for structs, or even union types.

At the very least, I would think that mix-ins would have a fairly significant effect.

ABI stability means that changes will have to be backwards compatible after a certain stage. If we can add mixins feature without modifying old code (and its SIL and IR and whatever), then we are fine.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

8

I'd also like to understand this more and this answer does not completely satisfy me. I understand backwards compatibility, especially in terms of source breaking changes.

I have more difficulties understanding what breaks or not the ABI and how to make educated guesses about what features will require breaking it

This is hard to explain right now without detailed knowledge of IRGen and SILGen internals. However we are planning on relaxing the restrictions as much as possible, so that source-compatible changes remain binary compatible. For example, we would like to be able to add new fields to structs, change computed properties to stored and vice versa, insert new classes in a hierarchy, add cases to enums, and so on. This falls under the umbrella of "resilience".

Here's a document outlining what will be ABI compatible and what will not -- keep in mind that a good chunk of this is not yet implemented:

https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst

As a follow-up explanation, when we talk about the ABI, we're really talking about three orthogonal "axes" along which we would like to "move" without breaking compatibility with existing binaries:

- The first axis is the machine-level calling conventions and memory layout. For example, what registers to pass function arguments and returns in, the rules for alignment and padding of fields in an aggregate type, which entry points the Swift runtime exports and what their behavior should be. Once we commit to a stable ABI along this axis, you will get interoperability between *compiler versions* -- the same exact library built with one version of the compiler will remain compatible with clients after being recompiled with another version, because their conventions will match up. Note that this does not help you if the library itself changes in any way.

- The second axis is the resilience work I called out in my previous e-mail. Here, we're trying to define language features and implementation techniques that allow a library to evolve in a forward-compatible manner, as long as the developer follows certain guidelines. Here, the goal is if you should be able to compile your library, make some changes to add new APIs, and recompile it *with the same compiler*, without breaking downstream clients, as long as you follow the library evolution guidelines (Also, you can imagine one day having an 'ABI diff' tool to automate this).

- The third axis is the standard library itself. Stability of runtime interfaces and the extra indirection to enable resilience is all great, but it won't help you as long as the standard library API is evolving in a non-backwards compatible manner -- for example, if we remove a method on String. So once the other two areas have been addressed, the last thing to lock down is the standard library interface itself.

···

On Aug 11, 2016, at 1:48 PM, Slava Pestov via swift-evolution <swift-evolution@swift.org> wrote:

On Aug 11, 2016, at 4:27 AM, David Hart via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On 11 Aug 2016, at 13:24, Anton Zhilin via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

2016-08-11 11:52 GMT+03:00 Jonathan Hull via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>>:
Could someone explain in simple terms what the ABI is and what sorts of things might affect it?

I had thought it was the layout in memory of structs/classes/etc… and how the program knows where to go to find a particular field. This seems to be incorrect though, as I have seen many features that I would assume have some affect on this layout ruled “out of scope for phase 1”. For example, I would think that many generics features would have an impact on the ABI, or the idea of COW (via secret boxing) for structs, or even union types.

At the very least, I would think that mix-ins would have a fairly significant effect.

ABI stability means that changes will have to be backwards compatible after a certain stage. If we can add mixins feature without modifying old code (and its SIL and IR and whatever), then we are fine.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

9

I think perhaps the other thing that might matter here is that we can always add to the ABI in backwards-compatible ways. So, for instance, we don't have to nail down all generics features immediately just because they affect the ABI; we just need to nail down the ABIs of existing features, plus any new features which will change the design of existing standard library features. If we ever add, say, higher-kinded types, they will certainly have some kind of representation in the ABI, but as long as they don't affect the binary representation of non-higher-kinded types, that won't really affect compatibility with existing code.

Similarly, there's a straightforward way to implement COWed structs in a backwards compatible way: treat them as a struct wrapping a reference to an object containing the data. As long as the compiler implements COWed structs in that fashion, or in any other way that's expressible in the existing ABI, COW structs are only an issue if we redesign inlineable parts of the standard library to use them.

(Correct me if I'm wrong, because I might be, but that's the impression I have.)

···

On Aug 11, 2016, at 1:57 PM, Slava Pestov via swift-evolution <swift-evolution@swift.org> wrote:

Here's a document outlining what will be ABI compatible and what will not -- keep in mind that a good chunk of this is not yet implemented:

https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst

As a follow-up explanation, when we talk about the ABI, we're really talking about three orthogonal "axes" along which we would like to "move" without breaking compatibility with existing binaries:

- The first axis is the machine-level calling conventions and memory layout. For example, what registers to pass function arguments and returns in, the rules for alignment and padding of fields in an aggregate type, which entry points the Swift runtime exports and what their behavior should be. Once we commit to a stable ABI along this axis, you will get interoperability between *compiler versions* -- the same exact library built with one version of the compiler will remain compatible with clients after being recompiled with another version, because their conventions will match up. Note that this does not help you if the library itself changes in any way.

- The second axis is the resilience work I called out in my previous e-mail. Here, we're trying to define language features and implementation techniques that allow a library to evolve in a forward-compatible manner, as long as the developer follows certain guidelines. Here, the goal is if you should be able to compile your library, make some changes to add new APIs, and recompile it *with the same compiler*, without breaking downstream clients, as long as you follow the library evolution guidelines (Also, you can imagine one day having an 'ABI diff' tool to automate this).

- The third axis is the standard library itself. Stability of runtime interfaces and the extra indirection to enable resilience is all great, but it won't help you as long as the standard library API is evolving in a non-backwards compatible manner -- for example, if we remove a method on String. So once the other two areas have been addressed, the last thing to lock down is the standard library interface itself.

--
Brent Royal-Gordon
Architechies

10

This is an excellent explanation! I think I (mostly) understand.

Thanks,
Jon

···

On Aug 11, 2016, at 9:16 AM, John McCall <rjmccall@apple.com> wrote:

On Aug 11, 2016, at 1:52 AM, Jonathan Hull via swift-evolution <swift-evolution@swift.org> wrote:
Could someone explain in simple terms what the ABI is and what sorts of things might affect it?

The ABI for a compiled programming language is the set of rules for how all of its interoperating features are implemented in the compiled result.

For example, in Java the primary ABI is the JVM specification plus some common-sense rules about how language features are mapped to JVM features. JVMs do not typically interoperate with other code at a direct binary level, and so the details of actual memory layout are not ABI.

Not all language features require interoperation, or only require it a certain kind of opaque interoperation. For example:

- Type aliases do not currently have any ABI impact because by design they are erased at compile time. However, a complete reflection design might provide some mechanism for introspecting type-aliases at runtime, which would require information about type aliases to be present in the compiled output; the representation of that information would be ABI.

- Closures in most programming languages do not allow external code to modify their captured values; only the closure's invocation function can access them, and that is generated together with the capturing code, so the exact methodology of performing a capture is not ABI. However, arbitrary code can invoke the closure, and so the opaque representation of a closure value and the process for invoking it are ABI.

And so on.

John.

I had thought it was the layout in memory of structs/classes/etc… and how the program knows where to go to find a particular field. This seems to be incorrect though, as I have seen many features that I would assume have some affect on this layout ruled “out of scope for phase 1”. For example, I would think that many generics features would have an impact on the ABI, or the idea of COW (via secret boxing) for structs, or even union types.

At the very least, I would think that mix-ins would have a fairly significant effect.

I am obviously missing something here, and I want to provide constructive effort to the community (as opposed to distracting from the task at hand), so I would appreciate some clarification/guidance...

Thanks,
Jon
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

11

Just to put in perspective: If we don't define now (probably we won't) how
to add storage properties to existing types and/or define storage
properties in protocols, we will never get this?

I remember long time ago, Greg Parker presented some new refcount
representation, and one of advantages is someday adding storage properties
into classes using side allocation.

Is this new refcount in swift?

(ref:
https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160314/001424.html
)

···

Em qui, 11 de ago de 2016 às 17:57, Slava Pestov via swift-evolution < swift-evolution@swift.org> escreveu:

On Aug 11, 2016, at 1:48 PM, Slava Pestov via swift-evolution < > swift-evolution@swift.org> wrote:

On Aug 11, 2016, at 4:27 AM, David Hart via swift-evolution < > swift-evolution@swift.org> wrote:

I'd also like to understand this more and this answer does not completely
satisfy me. I understand backwards compatibility, especially in terms of
source breaking changes.

I have more difficulties understanding what breaks or not the ABI and how
to make educated guesses about what features will require breaking it

This is hard to explain right now without detailed knowledge of IRGen and
SILGen internals. However we are planning on relaxing the restrictions as
much as possible, so that source-compatible changes remain binary
compatible. For example, we would like to be able to add new fields to
structs, change computed properties to stored and vice versa, insert new
classes in a hierarchy, add cases to enums, and so on. This falls under the
umbrella of "resilience".

Here's a document outlining what will be ABI compatible and what will not
-- keep in mind that a good chunk of this is not yet implemented:

https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst

As a follow-up explanation, when we talk about the ABI, we're really
talking about three orthogonal "axes" along which we would like to "move"
without breaking compatibility with existing binaries:

- The first axis is the machine-level calling conventions and memory
layout. For example, what registers to pass function arguments and returns
in, the rules for alignment and padding of fields in an aggregate type,
which entry points the Swift runtime exports and what their behavior should
be. Once we commit to a stable ABI along this axis, you will get
interoperability between *compiler versions* -- the same exact library
built with one version of the compiler will remain compatible with clients
after being recompiled with another version, because their conventions will
match up. Note that this does not help you if the library itself changes in
any way.

- The second axis is the resilience work I called out in my previous
e-mail. Here, we're trying to define language features and implementation
techniques that allow a library to evolve in a forward-compatible manner,
as long as the developer follows certain guidelines. Here, the goal is if
you should be able to compile your library, make some changes to add new
APIs, and recompile it *with the same compiler*, without breaking
downstream clients, as long as you follow the library evolution guidelines
(Also, you can imagine one day having an 'ABI diff' tool to automate this).

- The third axis is the standard library itself. Stability of runtime
interfaces and the extra indirection to enable resilience is all great, but
it won't help you as long as the standard library API is evolving in a
non-backwards compatible manner -- for example, if we remove a method on
String. So once the other two areas have been addressed, the last thing to
lock down is the standard library interface itself.

On 11 Aug 2016, at 13:24, Anton Zhilin via swift-evolution < > swift-evolution@swift.org> wrote:

2016-08-11 11:52 GMT+03:00 Jonathan Hull via swift-evolution <
swift-evolution@swift.org>:

Could someone explain in simple terms what the ABI is and what sorts of
things might affect it?

I had thought it was the layout in memory of structs/classes/etc… and how
the program knows where to go to find a particular field. This seems to be
incorrect though, as I have seen many features that I would assume have
some affect on this layout ruled “out of scope for phase 1”. For example,
I would think that many generics features would have an impact on the ABI,
or the idea of COW (via secret boxing) for structs, or even union types.

At the very least, I would think that mix-ins would have a fairly
significant effect.

ABI stability means that changes will have to be backwards compatible
after a certain stage. If we can add mixins feature without modifying old
code (and its SIL and IR and whatever), then we are fine.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

12

I would just like to add that one very important part of calling conventions is memory ownership.

All memory ownership conventions that end up in the stable ABI have to be supported after it has been declared stable. Adding new ways of transferring (or not transferring) ownership could be handled by adding new types of name mangling, I guess, but since old binaries would use the old model, everyone calling them would have to implement the old conventions as well.

If I'm not totally wrong.

/Magnus

···

11 Aug. 2016 22:57 Slava Pestov via swift-evolution <swift-evolution@swift.org> wrote:

- The first axis is the machine-level calling conventions and memory layout. For example, what registers to pass function arguments and returns in, the rules for alignment and padding of fields in an aggregate type, which entry points the Swift runtime exports and what their behavior should be. Once we commit to a stable ABI along this axis, you will get interoperability between *compiler versions* -- the same exact library built with one version of the compiler will remain compatible with clients after being recompiled with another version, because their conventions will match up. Note that this does not help you if the library itself changes in any way.