withUnsafePointer doesn't just produce a pointer; it also bounds the time when that accesses can be made through that pointer. Making the code you've written work would in general require the implementation to be maximally conservative about memory analysis for any variable whose address might be taken and escaped as a pointer, which essentially means any variable with any abstract uses at all. That is essentially the world that C and C++ compilers have always lived in, but Swift aims to do better.
The fact that withUnsafePointer and other closure-based scoping APIs don't compose reasonably with features like yield is a problem that demands a general solution. We should not sacrifice basic goals of the system to work around it in the short term.