How to prevent Swift from copying c++ types

I'm having trouble understanding what UnsafePointer means in Swift. If I have a vector in c++ that contains items I want Swift to work with, and if on the Swift side I have it set up to use UnsafeMutablePointer (or any other Unsafe type), it seems as though it's still copying the data from the vector anyway. The only way it seems like the language won't copy is if I make the vector store pointers and actually allocate the objects with new, but that's overhead and maintenance that would be nice to avoid.

Am I using Unsafe incorrectly? Is it expected that passing a c++ type to a function, constructor, etc that takes an UnsafePointer would copy the c++ type?