I believe the way to fix that is to systematically remove "Unsafe" prefixes, make unsafe functions callable only from functions marked "unsafe" (as we discussed), and possibly simplify the API in favour of, say, "data.bytes()" instead of "data.withBytes { ... }" – unsafe code won't be much less safer as a result but would look simpler and even similar to what you could do in C/C++.
I'd also like to see some built-in sanctioned way to have non atomic retain count operations – that will only work for objects accessible from a single thread (along with a precondition check that will fire if I am using those pinned to thread objects from different threads. And a placement new analogue from C++.