There's surely a lot of low-hanging fruit here. For example, there's this program which makes Swift look 56x slower than C if you take a naïve view.
The explicit call to the needless clear() accounts for a lot of the slowdown, and the needless use of reference-semantic class also accounts for a lot. Just say no to reference semantics. This gist shows you can easily cut the time by a factor of .458 which would put Swift's performance at 25.14s in their table, 25.6x slower than C.
Of course if you write the Swift code just like the C code, with unsafe pointers, it's faster than the C, which is part of why I find the whole thing frustrating.