-cross-module-optimization (CMO) is still in an "experimental" phase. Eventually we want to enable it by default (unless a module is compiled with -enable-library-evolution).
And yes, CMO has some tradeoffs. On the one hand it can improve performance significantly, e.g. if a module calls a generic function in another module. On the other hand this can lead to a code size increase.
The concept of CMO is comparable to LTO, but it's much more powerful, because it enables the swift (SIL) optimizer to optimize across modules (and not only the LLVM optimizer).
BTW, there is a (undocumented) Xcode build setting to enable CMO: SWIFT_CROSS_MODULE_OPTIMIZATION
It would be great if you could file bugs for those crashes - or for any other CMO-related problems.