You say this last, but this is the key question. Let me first address the Array note:
I think we all agree that pointers are reference types. Any method on a value type that exposes a pointer necessarily vends you a reference type from that value type. Because of the way Swift uses scoped accessors, these with functions temporarily transform a value type into a reference type for the duration of their usage. That means that this example is similar in kind to the question of collection indices.
This is a necessary and desirable behaviour of their usage. For example, the small-String optimization makes a pointer to the storage of a String literally unusable outside the scope of the call, meaning that escaping the bit-value of that pointer can't possibly provide any notion of identity that's useful. However, within the scope of the call that identity is absolutely useful. Put another way, all of these with functions pin a value into a location such that they can expose a reference type derived from a specific value.
So I agree with you: those functions don't prevent Array from being a value type. But they do that by only being useful as identifiers for the duration of the with call, where they temporarily expose a reference to a specific instance of the type, changing the semantics.
Let's be clear: generically, collection indices are reference types. Indices are the canonical example of a reference. Indeed, a pointer is simply an index into memory space, and they're the go-to example of a reference type.
There's some mental gymastics to go through here though, because of the fact that the type of many indices is a value type. Array.Index is Int, for example, and Int is unquestionably a value type. However, my argument is that Array.Index is still a reference type, just as pointers and file descriptors are. The rules on Collections are quite clear: an Index may be used with the original collection, or a derived slice, and nothing else.
So let's return to your incisive question:
There is no contradiction here. One can always derive a value with reference semantics from a variable by asking for a pointer to that variable. This is true for all variables in Swift: withUnsafePointer(to:) will give you a pointer, and pointers are always reference types.
Importantly though, the pointer is the reference type, not the thing it points to. The thing it points to still has value semantics, and while you can mutate through that reference all day, when you hold the bare type you are holding something with value semantics. Put another way, one can always "refer" to a value: it doesn't eliminate the semantic of the thing being referred to.
Certainly Dave Abrahams agrees with you: ValueSemantic protocol - #3 by dabrahams. However, even on that post he notes that classes make life very tricky here. Regardless, I think this is now a disagreement on what the words mean, rather than a disagreement on any technical aspect, so for the sake of not further derailing this thread I'll withdraw.