What can you change in a fixed-contents struct?

Hey, all. In preparation for the several proposals we have to come this year, I cleaned up docs/LibraryEvolution.rst <https://github.com/apple/swift/pull/11742> a little bit based on what's changed in Swift 4. This is mostly just mentioning things about generic subscripts, but there is one issue that I remember John bringing up some time in this past year: once you've made a struct fixed-contents, what can you change about its stored properties?

To opt out of this flexibility, a struct may be marked '@fixedContents'. This promises that no stored properties will be added to or removed from the struct, even non-public ones.

Interestingly, this means that you can have non-public members of a fixed-contents struct. This is actually pretty sensible: it's the equivalent of a C++ class with non-public fields but a defaulted, inlinable copy constructor. Any inlinable members can access these properties directly as well; it's just outsiders that can't see them. But if inlinable code can reference these things, and if we really want them to be fast, that means they have to have a known offset at compile-time.

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

At the moment I'm inclined to go with "allow renaming" just because that's what C does. It's not great because you're allowed to reorder nearly everything else in the language, but there's a "least surprise" principle here that I think is still useful. It'd be surprising for the name of a non-public property to affect your library's ABI.

(In theory we could also make different decisions for public and non-public fields, because it's much less likely to want to change the name of a public property. But you could do it without breaking source compatibility by introducing a computed property at the same time as you do the renaming. It's up to us whether that should break binary compatibility or not.)

Note that because the layout algorithm isn't public, Swift isn't going to allow the thing from C where you have an existing field and you split it into two smaller fields. You can use computed properties for that instead. (Strictly speaking this probably isn't even good C because the fields in your struct can affect by-value calling conventions.)

By the way, I'm putting this on swift-dev because we're nowhere near a proposal and I want to keep the discussion narrow for now, but of course this point will be called out when the fixed-contents attribute—whatever its final form—goes to swift-evolution for a proper review.

Thanks!
Jordan

In a somewhat similar semantic situation, Objective-C non-fragile ivars chose to allow reordering and disallow renaming. One reason is that we wanted to prevent accidental ABI breaks as best we could. The assumption was that a name change is probably deliberate, but an order change is likely to be an accidental effect of source code rearrangement. In addition, erroneous name changes are more likely to be caught downstream (for example when clients fail to build because they didn't change their use of the name), but erroneous order changes are less likely to be caught later.

The goal of preventing accidental ABI breaks becomes less important if Swift gets good ABI-diffing tools that can help catch errors after they are introduced. ObjC had no effective way to verify that your current ABI matched your previously-published ABI.

···

On Sep 5, 2017, at 11:59 AM, Jordan Rose via swift-dev <swift-dev@swift.org> wrote:

Hey, all. In preparation for the several proposals we have to come this year, I cleaned up docs/LibraryEvolution.rst <https://github.com/apple/swift/pull/11742> a little bit based on what's changed in Swift 4. This is mostly just mentioning things about generic subscripts, but there is one issue that I remember John bringing up some time in this past year: once you've made a struct fixed-contents, what can you change about its stored properties?

To opt out of this flexibility, a struct may be marked '@fixedContents'. This promises that no stored properties will be added to or removed from the struct, even non-public ones.

Interestingly, this means that you can have non-public members of a fixed-contents struct. This is actually pretty sensible: it's the equivalent of a C++ class with non-public fields but a defaulted, inlinable copy constructor. Any inlinable members can access these properties directly as well; it's just outsiders that can't see them. But if inlinable code can reference these things, and if we really want them to be fast, that means they have to have a known offset at compile-time.

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

At the moment I'm inclined to go with "allow renaming" just because that's what C does. It's not great because you're allowed to reorder nearly everything else in the language, but there's a "least surprise" principle here that I think is still useful. It'd be surprising for the name of a non-public property to affect your library's ABI.

--
Greg Parker gparker@apple.com <mailto:gparker@apple.com> Runtime Wrangler

Hi Jordan,

Thanks for thinking about this. For whatever it may be worth, I’m concerned about 1) the ability to reorder declarations and 2) the “either/or” nature of this proposal.

First, reordering: The ability to reorder declarations is deeply ingrained into the subconsciousness of Swift programmers and for good reasons. I think adding localized declaration order sensitivity is likely to be very brittle and regrettable in the long run. I also think this problem is made worse by having a type declaration context attribute because it can easily get lost in the noise of nontrivial declarations. The net result is that people will frequently forget about local order sensitivity. Yes, the compiler can engage in heroics and compare the previous module ABI and the new module ABI for conflicts, but that seems risky, complex, slow, and differently error prone.

Second, the “either/or” nature of this proposal: What do you think about a lightweight “belt and suspenders” solution whereby @fixedContents requires that stored properties be lightly annotated with their layout order? For example:

@fixedContents(3) struct Foo {
  @abi(0) var x: Int
  func a() {
    // a thousand lines of ABI layout distraction
  }
  @abi(1) var y: Double
  func b() {
    // another thousand lines of ABI layout distraction
  }
  @abi(2) var z: String
}

That would enable both renaming and reordering, would it not? This approach would also aid the compiler in quickly detecting hastily added/deleted declarations too because the count of @abi([0-9]+) declarations wouldn’t match the number passed to @fixedContents.

Dave

···

On Sep 5, 2017, at 14:59, Jordan Rose via swift-dev <swift-dev@swift.org> wrote:

Hey, all. In preparation for the several proposals we have to come this year, I cleaned up docs/LibraryEvolution.rst <https://github.com/apple/swift/pull/11742> a little bit based on what's changed in Swift 4. This is mostly just mentioning things about generic subscripts, but there is one issue that I remember John bringing up some time in this past year: once you've made a struct fixed-contents, what can you change about its stored properties?

To opt out of this flexibility, a struct may be marked '@fixedContents'. This promises that no stored properties will be added to or removed from the struct, even non-public ones.

Interestingly, this means that you can have non-public members of a fixed-contents struct. This is actually pretty sensible: it's the equivalent of a C++ class with non-public fields but a defaulted, inlinable copy constructor. Any inlinable members can access these properties directly as well; it's just outsiders that can't see them. But if inlinable code can reference these things, and if we really want them to be fast, that means they have to have a known offset at compile-time.

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

At the moment I'm inclined to go with "allow renaming" just because that's what C does. It's not great because you're allowed to reorder nearly everything else in the language, but there's a "least surprise" principle here that I think is still useful. It'd be surprising for the name of a non-public property to affect your library's ABI.

(In theory we could also make different decisions for public and non-public fields, because it's much less likely to want to change the name of a public property. But you could do it without breaking source compatibility by introducing a computed property at the same time as you do the renaming. It's up to us whether that should break binary compatibility or not.)

Note that because the layout algorithm isn't public, Swift isn't going to allow the thing from C where you have an existing field and you split it into two smaller fields. You can use computed properties for that instead. (Strictly speaking this probably isn't even good C because the fields in your struct can affect by-value calling conventions.)

By the way, I'm putting this on swift-dev because we're nowhere near a proposal and I want to keep the discussion narrow for now, but of course this point will be called out when the fixed-contents attribute—whatever its final form—goes to swift-evolution for a proper review.

Thanks!
Jordan
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Hey, all. In preparation for the several proposals we have to come this year, I cleaned up docs/LibraryEvolution.rst <https://github.com/apple/swift/pull/11742> a little bit based on what's changed in Swift 4. This is mostly just mentioning things about generic subscripts, but there is one issue that I remember John bringing up some time in this past year: once you've made a struct fixed-contents, what can you change about its stored properties?

To opt out of this flexibility, a struct may be marked '@fixedContents'. This promises that no stored properties will be added to or removed from the struct, even non-public ones.

Interestingly, this means that you can have non-public members of a fixed-contents struct. This is actually pretty sensible: it's the equivalent of a C++ class with non-public fields but a defaulted, inlinable copy constructor. Any inlinable members can access these properties directly as well; it's just outsiders that can't see them. But if inlinable code can reference these things, and if we really want them to be fast, that means they have to have a known offset at compile-time.

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

At the moment I'm inclined to go with "allow renaming" just because that's what C does. It's not great because you're allowed to reorder nearly everything else in the language, but there's a "least surprise" principle here that I think is still useful. It'd be surprising for the name of a non-public property to affect your library's ABI.

I think this is the right compromise to make. In addition to your points, I would add:
  - Property ordering can significantly impact performance, both by changing the size of a struct and changing the locality characteristics of accesses to it.
  - Performance tuning often happens throughout the development of a library, but it is quite common for renaming requests to land late, often as part of a final code/design review. Programmers would be quite surprised and dismayed if those changes invalidated most of their tuning.
  - Even very complex layout algorithms are still more likely than not to place subsequent properties near to each other. To the extent that users cluster related properties that are accessed together — and I think they notably do — this will tend to improve locality. Ordering by property name is likely to produce something more like a random jumble.
  - Reordering properties in the source code is a fairly simple and sensible way for an expert to influence the layout of a struct in a way that, practically speaking, can be combined with an easy-to-write layout-printing tool to yield total control over the layout, especially if there's an attribute to force Swift to use a specific layout algorithm. In contrast, as tempting as it is to say that Swift should allow an exact layout to be specified, that sounds like it'd be a horrific feature to actually use.

John.

···

On Sep 5, 2017, at 2:58 PM, Jordan Rose <jordan_rose@apple.com> wrote:

(In theory we could also make different decisions for public and non-public fields, because it's much less likely to want to change the name of a public property. But you could do it without breaking source compatibility by introducing a computed property at the same time as you do the renaming. It's up to us whether that should break binary compatibility or not.)

Note that because the layout algorithm isn't public, Swift isn't going to allow the thing from C where you have an existing field and you split it into two smaller fields. You can use computed properties for that instead. (Strictly speaking this probably isn't even good C because the fields in your struct can affect by-value calling conventions.)

By the way, I'm putting this on swift-dev because we're nowhere near a proposal and I want to keep the discussion narrow for now, but of course this point will be called out when the fixed-contents attribute—whatever its final form—goes to swift-evolution for a proper review.

Thanks!
Jordan

This is going against the grain, but I think we should order by name and therefore allow reordering, but not renaming. If an API is public, renaming is obviously going to be source-breaking and could easily be ABI-breaking; I don't think it's that hard to explain that renaming can also be ABI-breaking when you've declared your type's layout is part of your module's ABI. And we can always introduce a renaming attribute that causes the property to be laid out and linked by its old name:

  @fixedContents struct Foo {
    @renamed(from: x, in: iOS 14)
    var xCount: Int
    
    @renamed(from: y, in: iOS 14)
    var yAverage: Double
    
    @renamed(from: z, in: iOS 14)
    var zIdentifier: String
  }

We could detect properties appearing and disappearing in our compatibility checker tool and help people add the missing attributes. We could provide fix-its for renames of public APIs. We could use the name `_` to allow a type to reserve space for future expansion, or remove a property that is no longer used. We could add a syntax to `@renamed` that permits the space used by old properties to be subdivided into new ones. And we can always add ways to manually control the layout of a type in future versions of Swift; they would be usable both with and without `@fixedContents`, and would be orthogonal to `@fixedContents`.

(In theory, we could do this with `@available`, but its current renaming support requires a dummy declaration.)

As for the keyword…maybe `public(layout)` or `public(storage)`? People are familiar with the idea that you have to be careful when you change something that's public, so it wouldn't be surprising that a type with a public layout would be sensitive to changes to its memory layout.

···

On Sep 5, 2017, at 11:59 AM, Jordan Rose via swift-dev <swift-dev@swift.org> wrote:

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

--
Brent Royal-Gordon
Architechies

There are a couple of other places where ordering of declarations matters in the ABI. The two most important ones I can think of right now are virtual methods and protocol methods. Both are called by performing an indexed lookup into a vtable. For methods, we were thinking that the ordering should first sort by availability annotation, and then by the mangled name of the method. This would allow re-ordering as well as adding new methods, as long as they had a ‘newer’ availability than anything that has come before.

However, I like the idea of an @_abi (or similar) attribute as an escape hatch of some kind, in case we screw something up.

Slava

···

On Sep 5, 2017, at 1:01 PM, David Zarzycki via swift-dev <swift-dev@swift.org> wrote:

Hi Jordan,

Thanks for thinking about this. For whatever it may be worth, I’m concerned about 1) the ability to reorder declarations and 2) the “either/or” nature of this proposal.

First, reordering: The ability to reorder declarations is deeply ingrained into the subconsciousness of Swift programmers and for good reasons.

Good point. One slight difference here is that if you do break the ABI with non-fragile ivars, it only matters if there are direct references to them from outside the library. For @fixedContents structs, the client may have direct references to any fields, even the non-public ones, in order to properly implement the value copy operation.

Jordan

···

On Sep 5, 2017, at 15:52, Greg Parker <gparker@apple.com> wrote:

On Sep 5, 2017, at 11:59 AM, Jordan Rose via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hey, all. In preparation for the several proposals we have to come this year, I cleaned up docs/LibraryEvolution.rst <https://github.com/apple/swift/pull/11742> a little bit based on what's changed in Swift 4. This is mostly just mentioning things about generic subscripts, but there is one issue that I remember John bringing up some time in this past year: once you've made a struct fixed-contents, what can you change about its stored properties?

To opt out of this flexibility, a struct may be marked '@fixedContents'. This promises that no stored properties will be added to or removed from the struct, even non-public ones.

Interestingly, this means that you can have non-public members of a fixed-contents struct. This is actually pretty sensible: it's the equivalent of a C++ class with non-public fields but a defaulted, inlinable copy constructor. Any inlinable members can access these properties directly as well; it's just outsiders that can't see them. But if inlinable code can reference these things, and if we really want them to be fast, that means they have to have a known offset at compile-time.

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

At the moment I'm inclined to go with "allow renaming" just because that's what C does. It's not great because you're allowed to reorder nearly everything else in the language, but there's a "least surprise" principle here that I think is still useful. It'd be surprising for the name of a non-public property to affect your library's ABI.

In a somewhat similar semantic situation, Objective-C non-fragile ivars chose to allow reordering and disallow renaming. One reason is that we wanted to prevent accidental ABI breaks as best we could. The assumption was that a name change is probably deliberate, but an order change is likely to be an accidental effect of source code rearrangement. In addition, erroneous name changes are more likely to be caught downstream (for example when clients fail to build because they didn't change their use of the name), but erroneous order changes are less likely to be caught later.

The goal of preventing accidental ABI breaks becomes less important if Swift gets good ABI-diffing tools that can help catch errors after they are introduced. ObjC had no effective way to verify that your current ABI matched your previously-published ABI.

Hm. This is definitely an option, but I don't think it's really an acceptable user experience. This feels like it'll drive people all the way to declaring their types in C because Swift makes it too hard.

We do expect to have a tool to diff old and new modules at some point, but we could do something even simpler here: make a public symbol with the struct's layout in its name. That way, even 'nm' can tell if the symbol disappears. (Of course, public symbols do have a small cost, since this might not actually be the best idea.)

Another idea would be to restrict @fixedContents to require that all stored properties appear contiguously in the struct, possibly even pinned to the top or bottom, to indicate that order's not completely arbitrary. Again, that's just an improvement, not full protection.

Jordan

···

On Sep 5, 2017, at 13:00, David Zarzycki <zarzycki@icloud.com> wrote:

Hi Jordan,

Thanks for thinking about this. For whatever it may be worth, I’m concerned about 1) the ability to reorder declarations and 2) the “either/or” nature of this proposal.

First, reordering: The ability to reorder declarations is deeply ingrained into the subconsciousness of Swift programmers and for good reasons. I think adding localized declaration order sensitivity is likely to be very brittle and regrettable in the long run. I also think this problem is made worse by having a type declaration context attribute because it can easily get lost in the noise of nontrivial declarations. The net result is that people will frequently forget about local order sensitivity. Yes, the compiler can engage in heroics and compare the previous module ABI and the new module ABI for conflicts, but that seems risky, complex, slow, and differently error prone.

Second, the “either/or” nature of this proposal: What do you think about a lightweight “belt and suspenders” solution whereby @fixedContents requires that stored properties be lightly annotated with their layout order? For example:

@fixedContents(3) struct Foo {
  @abi(0) var x: Int
  func a() {
    // a thousand lines of ABI layout distraction
  }
  @abi(1) var y: Double
  func b() {
    // another thousand lines of ABI layout distraction
  }
  @abi(2) var z: String
}

That would enable both renaming and reordering, would it not? This approach would also aid the compiler in quickly detecting hastily added/deleted declarations too because the count of @abi([0-9]+) declarations wouldn’t match the number passed to @fixedContents.

Dave

On Sep 5, 2017, at 14:59, Jordan Rose via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hey, all. In preparation for the several proposals we have to come this year, I cleaned up docs/LibraryEvolution.rst <https://github.com/apple/swift/pull/11742> a little bit based on what's changed in Swift 4. This is mostly just mentioning things about generic subscripts, but there is one issue that I remember John bringing up some time in this past year: once you've made a struct fixed-contents, what can you change about its stored properties?

To opt out of this flexibility, a struct may be marked '@fixedContents'. This promises that no stored properties will be added to or removed from the struct, even non-public ones.

Interestingly, this means that you can have non-public members of a fixed-contents struct. This is actually pretty sensible: it's the equivalent of a C++ class with non-public fields but a defaulted, inlinable copy constructor. Any inlinable members can access these properties directly as well; it's just outsiders that can't see them. But if inlinable code can reference these things, and if we really want them to be fast, that means they have to have a known offset at compile-time.

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

At the moment I'm inclined to go with "allow renaming" just because that's what C does. It's not great because you're allowed to reorder nearly everything else in the language, but there's a "least surprise" principle here that I think is still useful. It'd be surprising for the name of a non-public property to affect your library's ABI.

(In theory we could also make different decisions for public and non-public fields, because it's much less likely to want to change the name of a public property. But you could do it without breaking source compatibility by introducing a computed property at the same time as you do the renaming. It's up to us whether that should break binary compatibility or not.)

Note that because the layout algorithm isn't public, Swift isn't going to allow the thing from C where you have an existing field and you split it into two smaller fields. You can use computed properties for that instead. (Strictly speaking this probably isn't even good C because the fields in your struct can affect by-value calling conventions.)

By the way, I'm putting this on swift-dev because we're nowhere near a proposal and I want to keep the discussion narrow for now, but of course this point will be called out when the fixed-contents attribute—whatever its final form—goes to swift-evolution for a proper review.

Thanks!
Jordan
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

Ya, given how vtables work, the ABI is essentially API once ABI-stability matters. Having something like "@_slot() func” and "@_slots() class” would also yield a nice “belt and suspenders” approach for those people that want to enable ABI stability on a given declaration. If anything, given how many methods exist on classes and the strong desire to reorder methods, the “@_slot()” attribute would be really nice.

Dave

···

On Sep 5, 2017, at 18:14, Slava Pestov <spestov@apple.com> wrote:

On Sep 5, 2017, at 1:01 PM, David Zarzycki via swift-dev <swift-dev@swift.org> wrote:

Hi Jordan,

Thanks for thinking about this. For whatever it may be worth, I’m concerned about 1) the ability to reorder declarations and 2) the “either/or” nature of this proposal.

First, reordering: The ability to reorder declarations is deeply ingrained into the subconsciousness of Swift programmers and for good reasons.

There are a couple of other places where ordering of declarations matters in the ABI. The two most important ones I can think of right now are virtual methods and protocol methods. Both are called by performing an indexed lookup into a vtable. For methods, we were thinking that the ordering should first sort by availability annotation, and then by the mangled name of the method. This would allow re-ordering as well as adding new methods, as long as they had a ‘newer’ availability than anything that has come before.

However, I like the idea of an @_abi (or similar) attribute as an escape hatch of some kind, in case we screw something up.

Hi Jordan and David,

Reading this from a user’s perspective, I’d be very much in favour of something like the @abi annotation. The core issue I see with the ‘allow renaming’ approach is that users coming to Swift from other languages will do so with the expectation that ordering is significant (i.e. it’s somewhat intuitive that stored properties will be stored in the order they’re declared in languages such as C/C++). Given that the layout algorithm’s requirements in Swift mean that the order may be completely different from the definition order, I feel that having order be significant in this one area (reinforcing the expectation) is misleading and a potential source of confusion; I think it’s better to have the declaration order be consistently meaningless.

With that said, being able to rename properties is still very necessary; hence I feel that something like the @abi is a good solution, given that @fixedContents structs are likely not a novice level feature. I suppose the issue with that solution is it does remove the ability for the compiler to tightly reorder and pack properties if it follows the declaration order exactly; the question is whether that’s even something that should be a goal for @fixedContents structs, or whether something like a compiler warning for sub-optimal alignment could be a better approach.

Having an explicit @abi annotation could also go some way to addressing the issue of manual layout with structs. I’m primarily thinking of graphics contexts, wherein you need to declare e.g. a struct with vertex layout, and then that needs to be specified when passed to the GPU. The current solution is to declare in C code and import into Swift, which is cumbersome; having @fixedContents structs be the method of choice when a deterministic layout is needed seems to make sense. If combined with a hypothetical MemoryLayout<Type>.offsetOf(property) Swift would, I think, meet that need.

I guess my closing question is: under what circumstances would a user declare a @fixedContents struct and not care about its layout in memory; in other words, where might a novice need to use this feature?

Thomas

···

On 6/09/2017, at 8:01 AM, David Zarzycki via swift-dev <swift-dev@swift.org> wrote:

Hi Jordan,

Thanks for thinking about this. For whatever it may be worth, I’m concerned about 1) the ability to reorder declarations and 2) the “either/or” nature of this proposal.

First, reordering: The ability to reorder declarations is deeply ingrained into the subconsciousness of Swift programmers and for good reasons. I think adding localized declaration order sensitivity is likely to be very brittle and regrettable in the long run. I also think this problem is made worse by having a type declaration context attribute because it can easily get lost in the noise of nontrivial declarations. The net result is that people will frequently forget about local order sensitivity. Yes, the compiler can engage in heroics and compare the previous module ABI and the new module ABI for conflicts, but that seems risky, complex, slow, and differently error prone.

Second, the “either/or” nature of this proposal: What do you think about a lightweight “belt and suspenders” solution whereby @fixedContents requires that stored properties be lightly annotated with their layout order? For example:

@fixedContents(3) struct Foo {
  @abi(0) var x: Int
  func a() {
    // a thousand lines of ABI layout distraction
  }
  @abi(1) var y: Double
  func b() {
    // another thousand lines of ABI layout distraction
  }
  @abi(2) var z: String
}

That would enable both renaming and reordering, would it not? This approach would also aid the compiler in quickly detecting hastily added/deleted declarations too because the count of @abi([0-9]+) declarations wouldn’t match the number passed to @fixedContents.

Dave

On Sep 5, 2017, at 14:59, Jordan Rose via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hey, all. In preparation for the several proposals we have to come this year, I cleaned up docs/LibraryEvolution.rst <https://github.com/apple/swift/pull/11742> a little bit based on what's changed in Swift 4. This is mostly just mentioning things about generic subscripts, but there is one issue that I remember John bringing up some time in this past year: once you've made a struct fixed-contents, what can you change about its stored properties?

To opt out of this flexibility, a struct may be marked '@fixedContents'. This promises that no stored properties will be added to or removed from the struct, even non-public ones.

Interestingly, this means that you can have non-public members of a fixed-contents struct. This is actually pretty sensible: it's the equivalent of a C++ class with non-public fields but a defaulted, inlinable copy constructor. Any inlinable members can access these properties directly as well; it's just outsiders that can't see them. But if inlinable code can reference these things, and if we really want them to be fast, that means they have to have a known offset at compile-time.

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

At the moment I'm inclined to go with "allow renaming" just because that's what C does. It's not great because you're allowed to reorder nearly everything else in the language, but there's a "least surprise" principle here that I think is still useful. It'd be surprising for the name of a non-public property to affect your library's ABI.

(In theory we could also make different decisions for public and non-public fields, because it's much less likely to want to change the name of a public property. But you could do it without breaking source compatibility by introducing a computed property at the same time as you do the renaming. It's up to us whether that should break binary compatibility or not.)

Note that because the layout algorithm isn't public, Swift isn't going to allow the thing from C where you have an existing field and you split it into two smaller fields. You can use computed properties for that instead. (Strictly speaking this probably isn't even good C because the fields in your struct can affect by-value calling conventions.)

By the way, I'm putting this on swift-dev because we're nowhere near a proposal and I want to keep the discussion narrow for now, but of course this point will be called out when the fixed-contents attribute—whatever its final form—goes to swift-evolution for a proper review.

Thanks!
Jordan
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

This is going against the grain, but I think we should order by name and therefore allow reordering, but not renaming. If an API is public, renaming is obviously going to be source-breaking and could easily be ABI-breaking; I don't think it's that hard to explain that renaming can also be ABI-breaking when you've declared your type's layout is part of your module's ABI.

However, @fixedContents structs can also contain private stored properties. Renaming a private property is not source-breaking (but if we sort by name when performing layout, it will be ABI breaking).

As for the keyword…maybe `public(layout)` or `public(storage)`? People are familiar with the idea that you have to be careful when you change something that's public, so it wouldn't be surprising that a type with a public layout would be sensitive to changes to its memory layout.

I’m hesitant to use a keyword rather than an @attribute for this, because (for the most part) attributes don’t change the language semantics of a declaration, only its implementation (of course there are exceptions, like @objc which introduces restrictions, etc).

Slava

···

On Sep 6, 2017, at 7:57 PM, Brent Royal-Gordon via swift-dev <swift-dev@swift.org> wrote:

On Sep 5, 2017, at 11:59 AM, Jordan Rose via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Having an explicit @abi annotation could also go some way to addressing the issue of manual layout with structs. I’m primarily thinking of graphics contexts, wherein you need to declare e.g. a struct with vertex layout, and then that needs to be specified when passed to the GPU. The current solution is to declare in C code and import into Swift, which is cumbersome; having @fixedContents structs be the method of choice when a deterministic layout is needed seems to make sense. If combined with a hypothetical MemoryLayout<Type>.offsetOf(property) Swift would, I think, meet that need.

@fixedContents is not meant for this purpose. We don’t want to guarantee that the layout matches what you would write in C. Of course once the ABI is stable, the layout of a fixed contents struct will be fixed — it just might not be what your GPU expects.

So perhaps @fixedContents is not the best name. We expect that “define your struct in C” is still the way to go for layout compatibility with other languages and systems.

I guess my closing question is: under what circumstances would a user declare a @fixedContents struct and not care about its layout in memory; in other words, where might a novice need to use this feature?

It’s more about performance. By promising to the compiler that you will not change the layout of the struct, uses of the struct from other modules can use more efficient access patterns (for example, direct loads/stores of properties instead of calling accessors).

Slava

···

On Sep 5, 2017, at 2:48 PM, Thomas Roughton via swift-dev <swift-dev@swift.org> wrote:

Thomas

On 6/09/2017, at 8:01 AM, David Zarzycki via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hi Jordan,

Thanks for thinking about this. For whatever it may be worth, I’m concerned about 1) the ability to reorder declarations and 2) the “either/or” nature of this proposal.

First, reordering: The ability to reorder declarations is deeply ingrained into the subconsciousness of Swift programmers and for good reasons. I think adding localized declaration order sensitivity is likely to be very brittle and regrettable in the long run. I also think this problem is made worse by having a type declaration context attribute because it can easily get lost in the noise of nontrivial declarations. The net result is that people will frequently forget about local order sensitivity. Yes, the compiler can engage in heroics and compare the previous module ABI and the new module ABI for conflicts, but that seems risky, complex, slow, and differently error prone.

Second, the “either/or” nature of this proposal: What do you think about a lightweight “belt and suspenders” solution whereby @fixedContents requires that stored properties be lightly annotated with their layout order? For example:

@fixedContents(3) struct Foo {
  @abi(0) var x: Int
  func a() {
    // a thousand lines of ABI layout distraction
  }
  @abi(1) var y: Double
  func b() {
    // another thousand lines of ABI layout distraction
  }
  @abi(2) var z: String
}

That would enable both renaming and reordering, would it not? This approach would also aid the compiler in quickly detecting hastily added/deleted declarations too because the count of @abi([0-9]+) declarations wouldn’t match the number passed to @fixedContents.

Dave

On Sep 5, 2017, at 14:59, Jordan Rose via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hey, all. In preparation for the several proposals we have to come this year, I cleaned up docs/LibraryEvolution.rst <https://github.com/apple/swift/pull/11742> a little bit based on what's changed in Swift 4. This is mostly just mentioning things about generic subscripts, but there is one issue that I remember John bringing up some time in this past year: once you've made a struct fixed-contents, what can you change about its stored properties?

To opt out of this flexibility, a struct may be marked '@fixedContents'. This promises that no stored properties will be added to or removed from the struct, even non-public ones.

Interestingly, this means that you can have non-public members of a fixed-contents struct. This is actually pretty sensible: it's the equivalent of a C++ class with non-public fields but a defaulted, inlinable copy constructor. Any inlinable members can access these properties directly as well; it's just outsiders that can't see them. But if inlinable code can reference these things, and if we really want them to be fast, that means they have to have a known offset at compile-time.

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

At the moment I'm inclined to go with "allow renaming" just because that's what C does. It's not great because you're allowed to reorder nearly everything else in the language, but there's a "least surprise" principle here that I think is still useful. It'd be surprising for the name of a non-public property to affect your library's ABI.

(In theory we could also make different decisions for public and non-public fields, because it's much less likely to want to change the name of a public property. But you could do it without breaking source compatibility by introducing a computed property at the same time as you do the renaming. It's up to us whether that should break binary compatibility or not.)

Note that because the layout algorithm isn't public, Swift isn't going to allow the thing from C where you have an existing field and you split it into two smaller fields. You can use computed properties for that instead. (Strictly speaking this probably isn't even good C because the fields in your struct can affect by-value calling conventions.)

By the way, I'm putting this on swift-dev because we're nowhere near a proposal and I want to keep the discussion narrow for now, but of course this point will be called out when the fixed-contents attribute—whatever its final form—goes to swift-evolution for a proper review.

Thanks!
Jordan
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Right. The hypothetical 'abi' attribute that Dave mentions is "just" a stable identifier that can be used instead of "name" or "declaration order", rather than something that can control layout.

We did say that the layout of a tuple that contains C-compatible types will always match the layout of an equivalent C struct, so if you need a guaranteed layout in Swift you can do it that way too. But Slava's right that it's probably easiest to just define your struct in C.

Jordan

···

On Sep 5, 2017, at 15:08, Slava Pestov via swift-dev <swift-dev@swift.org> wrote:

On Sep 5, 2017, at 2:48 PM, Thomas Roughton via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Having an explicit @abi annotation could also go some way to addressing the issue of manual layout with structs. I’m primarily thinking of graphics contexts, wherein you need to declare e.g. a struct with vertex layout, and then that needs to be specified when passed to the GPU. The current solution is to declare in C code and import into Swift, which is cumbersome; having @fixedContents structs be the method of choice when a deterministic layout is needed seems to make sense. If combined with a hypothetical MemoryLayout<Type>.offsetOf(property) Swift would, I think, meet that need.

@fixedContents is not meant for this purpose. We don’t want to guarantee that the layout matches what you would write in C. Of course once the ABI is stable, the layout of a fixed contents struct will be fixed — it just might not be what your GPU expects.

So perhaps @fixedContents is not the best name. We expect that “define your struct in C” is still the way to go for layout compatibility with other languages and systems.

We expect that “define your struct in C” is still the way to go for layout compatibility with other languages and systems.

Are there plans (however tentative) to eventually make it possible to specify the exact memory layout of a struct in Swift?

It’s not clear what ‘exact memory layout’ means. In C, you don’t get to specify the exact layout either — there are rules around alignment and padding and some non-standard attributes for controlling them, but that’s not quite the same thing.

It seems like something we will have to tackle in order for Swift to become a serious systems-level programming language.

We do plan on documenting Swift’s struct (and tuple) layout algorithms as part of ABI stability, because we will need to freeze them (at least for types that cross resilience boundaries) to actually have a stable ABI. So you could, with knowledge of the layout algorithm, define a struct and make assumptions about it’s layout in memory, based on your knowledge of the algorithm as it is implemented in the version of Swift you are using. Not quite the same as explicitly specifying a layout, though.

Slava

···

On Sep 5, 2017, at 6:37 PM, Nevin Brackett-Rozinsky via swift-dev <swift-dev@swift.org> wrote:
On Tue, Sep 5, 2017 at 6:08 PM, Slava Pestov via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Nevin
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

I’m a big fan of syntactically pinning the stored properties at the top whenever their layout is part of the ABI.
-Andy

···

On Sep 5, 2017, at 5:11 PM, Jordan Rose via swift-dev <swift-dev@swift.org> wrote:

Another idea would be to restrict @fixedContents to require that all stored properties appear contiguously in the struct, possibly even pinned to the top or bottom, to indicate that order's not completely arbitrary. Again, that's just an improvement, not full protection.

Are there plans (however tentative) to eventually make it possible to
specify the exact memory layout of a struct in Swift?

It seems like something we will have to tackle in order for Swift to become
a serious systems-level programming language.

Nevin

···

On Tue, Sep 5, 2017 at 6:08 PM, Slava Pestov via swift-dev < swift-dev@swift.org> wrote:

We expect that “define your struct in C” is still the way to go for layout
compatibility with other languages and systems.

To be exact, we need to be able to universally agree on the layout algorithm used for a specific struct. That doesn't require us to only have one such algorithm; we can improve the struct layout algorithm in future releases as long as we have some way of ensuring that previously-fragile structs don't use the improved parts. Since Swift libraries will need to have a minimum Swift "deployment target", the most obvious way to pick an algorithm woulds simply be to use the layout algorithm from the library's minimum Swift target version at the time when the struct was first made fragile. That should be easy enough to record in a library's ABI description.

We could do something similar with tuples by finding the minimum target of all the element types, but the complexity would be less localized. There's merit to keeping the tuple layout algorithm simple, especially if it lets us make stronger memory-model statements about tuple elements. We also have a very straightforward answer to someone concerned about tuple memory layout efficient: just use a struct.

John.

···

On Sep 5, 2017, at 9:41 PM, Slava Pestov via swift-dev <swift-dev@swift.org> wrote:

On Sep 5, 2017, at 6:37 PM, Nevin Brackett-Rozinsky via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

On Tue, Sep 5, 2017 at 6:08 PM, Slava Pestov via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
We expect that “define your struct in C” is still the way to go for layout compatibility with other languages and systems.

Are there plans (however tentative) to eventually make it possible to specify the exact memory layout of a struct in Swift?

It’s not clear what ‘exact memory layout’ means. In C, you don’t get to specify the exact layout either — there are rules around alignment and padding and some non-standard attributes for controlling them, but that’s not quite the same thing.

It seems like something we will have to tackle in order for Swift to become a serious systems-level programming language.

We do plan on documenting Swift’s struct (and tuple) layout algorithms as part of ABI stability, because we will need to freeze them (at least for types that cross resilience boundaries) to actually have a stable ABI. So you could, with knowledge of the layout algorithm, define a struct and make assumptions about it’s layout in memory, based on your knowledge of the algorithm as it is implemented in the version of Swift you are using. Not quite the same as explicitly specifying a layout, though.

Hi Jordan,

I really doubt that “belt and suspenders” ABI attributes would drive people to C, but reasonable people can certainly disagree on that.

Bertrand, when he was at Apple, used to say that “easy things should be easy, and hard things should be possible”.

I think ABI related attributes fall into the latter category. In particular, I think that trying to make ABI attributes too convenient only does programmers a disservice in the long run because most programmers aren't experts, and the cost of ignorance is huge when trying to do ABI design.

With these thoughts in mind, I think that is reasonable for the language to say: “If you want explicit control over a dimension of ABI decisions, then you must deal with all of the associated complexity. Here is a pointer to the documentation on dimension X that you were/are trying to explicitly manage. If that is ’too hard’ for you, then you probably shouldn’t be locking down this dimension of complexity yet.”

Dave

···

On Sep 5, 2017, at 20:11, Jordan Rose <jordan_rose@apple.com> wrote:

Hm. This is definitely an option, but I don't think it's really an acceptable user experience. This feels like it'll drive people all the way to declaring their types in C because Swift makes it too hard.

We do expect to have a tool to diff old and new modules at some point, but we could do something even simpler here: make a public symbol with the struct's layout in its name. That way, even 'nm' can tell if the symbol disappears. (Of course, public symbols do have a small cost, since this might not actually be the best idea.)

Another idea would be to restrict @fixedContents to require that all stored properties appear contiguously in the struct, possibly even pinned to the top or bottom, to indicate that order's not completely arbitrary. Again, that's just an improvement, not full protection.

Jordan

On Sep 5, 2017, at 13:00, David Zarzycki <zarzycki@icloud.com <mailto:zarzycki@icloud.com>> wrote:

Hi Jordan,

Thanks for thinking about this. For whatever it may be worth, I’m concerned about 1) the ability to reorder declarations and 2) the “either/or” nature of this proposal.

First, reordering: The ability to reorder declarations is deeply ingrained into the subconsciousness of Swift programmers and for good reasons. I think adding localized declaration order sensitivity is likely to be very brittle and regrettable in the long run. I also think this problem is made worse by having a type declaration context attribute because it can easily get lost in the noise of nontrivial declarations. The net result is that people will frequently forget about local order sensitivity. Yes, the compiler can engage in heroics and compare the previous module ABI and the new module ABI for conflicts, but that seems risky, complex, slow, and differently error prone.

Second, the “either/or” nature of this proposal: What do you think about a lightweight “belt and suspenders” solution whereby @fixedContents requires that stored properties be lightly annotated with their layout order? For example:

@fixedContents(3) struct Foo {
  @abi(0) var x: Int
  func a() {
    // a thousand lines of ABI layout distraction
  }
  @abi(1) var y: Double
  func b() {
    // another thousand lines of ABI layout distraction
  }
  @abi(2) var z: String
}

That would enable both renaming and reordering, would it not? This approach would also aid the compiler in quickly detecting hastily added/deleted declarations too because the count of @abi([0-9]+) declarations wouldn’t match the number passed to @fixedContents.

Dave

On Sep 5, 2017, at 14:59, Jordan Rose via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hey, all. In preparation for the several proposals we have to come this year, I cleaned up docs/LibraryEvolution.rst <https://github.com/apple/swift/pull/11742> a little bit based on what's changed in Swift 4. This is mostly just mentioning things about generic subscripts, but there is one issue that I remember John bringing up some time in this past year: once you've made a struct fixed-contents, what can you change about its stored properties?

To opt out of this flexibility, a struct may be marked '@fixedContents'. This promises that no stored properties will be added to or removed from the struct, even non-public ones.

Interestingly, this means that you can have non-public members of a fixed-contents struct. This is actually pretty sensible: it's the equivalent of a C++ class with non-public fields but a defaulted, inlinable copy constructor. Any inlinable members can access these properties directly as well; it's just outsiders that can't see them. But if inlinable code can reference these things, and if we really want them to be fast, that means they have to have a known offset at compile-time.

Now, we don't plan to stick to C's layout for structs, even fixed-contents structs. We'd really like users to not worry about manually packing things into trailing alignment space. But we still need a way to lay out fields consistently; if you have two stored properties with the same type, one of them has to go first. There are two ways to do this: sort by name, and sort by declaration order. That means we can either allow reordering or allow renaming, but not both. Which do people think is more important?

At the moment I'm inclined to go with "allow renaming" just because that's what C does. It's not great because you're allowed to reorder nearly everything else in the language, but there's a "least surprise" principle here that I think is still useful. It'd be surprising for the name of a non-public property to affect your library's ABI.

(In theory we could also make different decisions for public and non-public fields, because it's much less likely to want to change the name of a public property. But you could do it without breaking source compatibility by introducing a computed property at the same time as you do the renaming. It's up to us whether that should break binary compatibility or not.)

Note that because the layout algorithm isn't public, Swift isn't going to allow the thing from C where you have an existing field and you split it into two smaller fields. You can use computed properties for that instead. (Strictly speaking this probably isn't even good C because the fields in your struct can affect by-value calling conventions.)

By the way, I'm putting this on swift-dev because we're nowhere near a proposal and I want to keep the discussion narrow for now, but of course this point will be called out when the fixed-contents attribute—whatever its final form—goes to swift-evolution for a proper review.

Thanks!
Jordan
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

There's nothing inherently wrong with having a 'packedLayout' attribute or similar (as long as the compiler can enforce alignment requirements for non-trivial types), but neither is it urgent since you can do the same thing in C. It's a feature that can be added at any time, just not to existing fixed-contents structs.

Jordan

···

On Sep 5, 2017, at 18:37, Nevin Brackett-Rozinsky via swift-dev <swift-dev@swift.org> wrote:

On Tue, Sep 5, 2017 at 6:08 PM, Slava Pestov via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
We expect that “define your struct in C” is still the way to go for layout compatibility with other languages and systems.

Are there plans (however tentative) to eventually make it possible to specify the exact memory layout of a struct in Swift?

It seems like something we will have to tackle in order for Swift to become a serious systems-level programming language.

This is going against the grain, but I think we should order by name and therefore allow reordering, but not renaming. If an API is public, renaming is obviously going to be source-breaking and could easily be ABI-breaking; I don't think it's that hard to explain that renaming can also be ABI-breaking when you've declared your type's layout is part of your module's ABI.

However, @fixedContents structs can also contain private stored properties. Renaming a private property is not source-breaking (but if we sort by name when performing layout, it will be ABI breaking).

I understand that. I think this is explicable if you think of `@fixedContents` as intentionally compromising the privacy of private stored properties. And I think that attaching this feature to the `public` keyword would help to encourage that mindset.

Another way to encourage a "don't change this" mindset would be to use a word like "frozen" in the attribute/keyword. It wouldn't be surprising if you caused problems by changing something you've "frozen". Example:

  @frozen(var) public struct Foo { … }
  @frozen(case) public enum Bar { … }

As for the keyword…maybe `public(layout)` or `public(storage)`? People are familiar with the idea that you have to be careful when you change something that's public, so it wouldn't be surprising that a type with a public layout would be sensitive to changes to its memory layout.

I’m hesitant to use a keyword rather than an @attribute for this, because (for the most part) attributes don’t change the language semantics of a declaration, only its implementation (of course there are exceptions, like @objc which introduces restrictions, etc).

I feel like I must not understand what you mean by semantics vs. implementation, because most of the attributes I can think of handle things I would consider to be "semantics". `@escaping`, `@autoclosure`, `@available`, `@discardableResult`, etc.

···

On Sep 6, 2017, at 9:52 PM, Slava Pestov <spestov@apple.com> wrote:

On Sep 6, 2017, at 7:57 PM, Brent Royal-Gordon via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

--
Brent Royal-Gordon
Architechies