So, you want this to have reference semantics (when mutable). Okay.
In that case, you can possibly get some abstraction with a protocol that both conform to, but yeah, Swift doesn't have struct inheritance, mainly to avoid the "slicing" problem that's well-known in C++. (In theory a language could support some kind of "inheritance without subtyping", but at some point the added complexity doesn't pay for itself.)
At some point we may get something like [Proposal Draft] automatic protocol forwarding, which (I think) would help a lot for cases like this.