Yeah I understand that memory layouts are not explicitly guaranteed in Swift, but since C-style layouts are currently reliable, and since this is a "feature" that people are currently taking advantage of (for instance as discussed by @Torust in this thread) I think it's worth understanding the costs and benefits of breaking said feature.
As far as the opt-in nature of library evolution, I still think it makes sense to be able to opt out on a per-struct bases. It's not clear to me why libraries declaring datatypes to be consumed by the GPU and libraries using library evolution should be mutually exclusive.