One important use-case is when you need a deinit. It's not just about (or even primarily about?) avoiding copies.
Linear types were originally invented for resource management, yeah (well, originally it was a curiosity from formal logic, until people realized it could be applied to resource management). Not sure why you see the infectious nature is a downside -- it's sort of the whole point.
Not every resource lifetime is scoped to the dynamic extent of a single function call. You might want to initialize a resource in one function and "hand it off" somewhere else, or store it somewhere.
I'm generally a "language feature skeptic" too and there are other people better positioned to defend this specific feature than I am, but generally, it seems to me that it's better from a language design standpoint if you can explicitly declare that something has a certain property and have this property be enforced at the type system level, instead of relying on a sufficiently smart optimizer. Optimizations can be hard to reason about, especially across function boundaries, are prone to being defeated by small code changes.
It is true that SILGen emits too many copies and the SIL optimizer is not able to eliminate some of them, and that should be addressed independently of ~Copyable, because in any case, most types are not expected to be ~Copyable. However, "too many" copies is one thing, but in some applications even one unnecessary copy is too much.
[Edit: One more thing is that while Swift doesn't have guaranteed call-once closures (yet?), those are another obvious application for noncopyable types which comes up repeatedly.]