It can go both ways. Which probably shouldn't surprise anyone. Objective-C is (essentially) a superset of C, which means you can drop down to basically writing glorified assembly if you want (or literally assembly, inline, in fact). Swift doesn't really let you do that. Even using things that nominally are C-like, such as Unsafe*Pointer, don't necessarily give you C-level performance (and they create code that is way more difficult to read than the C equivalent).
On the other hand, one of the first substantial programs I rewrote in Swift a bajillion years ago relied heavily on mapping strings to strings and was about a thousand times faster in Swift, because even back in Swift 2 Dictionary & String had some advantages over NSDictionary & NSString.
On the other other hand, to this day there's still plenty of basic String methods that are slower than their NSString equivalents (especially if you use the true String methods rather than the replacements you get from importing Foundation, e.g. for contains) and/or otherwise have terribly inefficient implementations (e.g. contains again).
And that's where I think there might be a disconnect - I suspect when some people say "Swift is fast like C++!" they're thinking about the naked language, not accounting for the libraries.
That's pretty much my feeling, too. I use Swift for a lot of stuff - happily - but for anything that I anticipate being performance-sensitive, I'm hesitant. For instance, lately I've been discovering that string processing in Swift is full of landmines (relating to both functionality and performance) and I've been resorting to workarounds as gross as rewriting Swift CLI tools into shell scripts.
The recent improvements to C++ interoperability make me less concerned about Swift's performance, in a way, but also worry me as potentially leading to a Python-like "answer" to performance concerns - essentially "don't use Python". 
Though I'm somewhat reassured by much experience using Objective-C++, which was mostly the best of both worlds, so maybe Swift++ will be too.