The discussion of String performance comes up around Advent of Code time every year it seems. 
An older, but still good, view of how Swift Strings work is Piercing the String Veil.
By default, the Swift String APIs are made to be flexible and safe to use, but they aren't seeking out total performance. As you've found you can drop down to the bytes if you want to, but it's a lot uglier down there.
That said, there are some things that make dealing with the bytes a bit easier such as the UInt8(ascii:) or UTF8.CodeUnit(ascii:) so you can get a utf8 element without needing to look up all the byte codes. You can also use the cString tools on String to make things a bit less cumbersome to deal with. It's still not a ton of fun though.
If I were to make your code into a bit more idiomatic Swift I would use the enumerated() method so that you automatically get an index counter and to simplify it a bit for the reader.
let whitespace = [UInt8(ascii: "\n"), UTF8.CodeUnit(ascii: " ")]
input.utf8.withContiguousStorageIfAvailable { data in
var start = 0
for (idx, char) in data.enumerated() {
if whitespace.contains(char) {
if start < idx {
let num = Int(
String(decoding: data[start..<idx], as: UTF8.self)
)!
xs.append(num)
swapArrays(&xs, &ys)
}
start = idx + 1
}
}
}
return (xs, ys)
}