Swift is a Safe, Fast, and Expressive language. I'm not sure I 100% agree with the ordering of "Fast" and "Expressive" there, but "Safe" is paramount, and it is not possible for the compiler to guarantee safety in this case.
You're welcome to propose a language feature to say "I know what I'm doing, let me call this" (at the call site) or "I know what I'm doing, make the convenience initializers available" (at the class declaration site). Both of those would be valid languages changes. But I think the default behavior is the correct default, and you do have a way to say what you mean here, albeit a wordy one.