Related to this point, it's worth considering whether we might want "borrowed" versions of other standard library types.
For instance, a recent pitch suggested that we might add a type called Unicode.UTF8.ValidBufferView for text processing over non-owned storage. What if we instead called that a BorrowedString?
There would be nice some symmetry to having both BorrowedArray and BorrowedString, and I think having that as a more established pattern could minimise some of your concerns.
Anyway, I don't want to labour the point too much because my actual opinion is that BorrowedBuffer is also fine. But it's worth thinking about these things.