It's true up to a point. It's also about how much abstractions and overhead you can bypass when you need more performance.
It also depends on how optimized fundamental types are implemented:
And you can detect those thanks to benchmarks.
Another explanation could be that most of what you can do to make Swift fast is only known by Apple Engineers and undocumented unlike C++.