The high-level property you care about with uniqueness checking is not “is the reference count of this object exactly 1”, it’s “can any other context observe this object”. As a direct result, it is only meaningful to ask it of an owned reference that you have exclusive access to, because if the other contexts can access the reference simultaneously, they can of course observe the object no matter what its reference count is. The way we express exclusive access in Swift is with inout.
The standard rule in Swift is that new values you construct are owned values which own their component values. This is a rule that allows values to present a uniform ABI where there is a single representation for values of the type, whether borrowed or owned, regardless of the nature of its component types. If you want to allow borrowed-by-construction values which can be constructed from borrows of the components, that is sensible, but such values cannot have the same representation as an owned value for all possible component types. This is because not all types in Swift have the property that a borrowed value can be relocated in memory. (In general, this is not satisfied by types whose representation is address-sensitive, e.g. because values are registered with a runtime by address, or because values can point into themselves. Among other things, we must assume all non-trivially-copyable C++ types do not have this property.) As a result, a borrowed-by-construction value must either store addresses of borrowed values (giving them a non-uniform representation) or disallow component types whose borrows cannot be relocated.
An alternative that might work in some cases would be to change the type to always store a borrow of its component. This is not currently expressible in Swift without falling back on unsafe pointers. Such a type would be inherently lifetime-restricted because a value would always be dependent on another value. But that is something that can be expressed in e.g. Rust.
Anyway, the optimization limitation you’re probably running into is that I believe SIL doesn’t know how to put together these borrowed-by-construction values even in the cases where it can support them, so if the optimizer can’t completely eliminate the need for the aggregate, it will need to produce an owned value.