borrow accessors return a borrow already as implemented in SE-507. & in Swift means inout, not "by reference", and it would be inappropriate to use in borrow accessors. Also, borrow and mutate accessors have different ABIs and caller-facing behavior from the coroutine variants, so clients of a library must know whether an accessor is yielding or not, and it wouldn't be appropriate for the change to be completely implicit hidden in the implementation of the accessors.
We have the borrow x operator on our roadmap as a way of explicitly stating an expression should be passed by borrow in situations where we currently implicitly copy. It is partially implemented in the main branch. It would be great to have help completing the implementation if you're interested.