Swift allows you to pass a &x ref to a function taking Unsafe[Mutable]Pointer, as long as x is mutable.
There's no way to pass an immutable UnsafePointer that points to immutable data. Any such immutable data has to be copied into a mutable variable before the &x ref works.
Allow for using &x with immutable values if and only if the reference is passed as a parameter to a function expecting UnsafePointer
It's easy to cast an UnsafePointer to an UnsafeMutablePointer and then mutate it. This problem exists in C and C++ as well. But UnsafePointer/UnsafeMutablePointer can already be used to bypass many of the safety mechanisms of the language, so I don't think this is a big deal. Any such mutations will either crash (because it's mutating something that's not in a writeable page), or it will invoke undefined behavior, but that shouldn't surprise anyone.
Do nothing and hope that at some point Swift introduces some form of first-class support for explicit immutable references. That's probably not happening any time soon.