The case for trapping on access by index is well-summarized here. The index you're using was calculated in some way, and if it goes out of bounds, there's a serious bug in your logic that shouldn't be handled at run time through a failed optional unwrap.
Most of the time that you're working with a collection, you don't have perfect knowledge about its contents. A hard-coded [1, 2, 3] is the exception, not the rule, so if random traps, then you have to do an emptiness check before using it most of the time. It's safer /and/ more convenient in the common case for the API to do the emptiness check for you.
That's not to say that optional element-at-index or non-optional random element APIs would not be useful sometimes, but they should not be the default. For what it's worth, I think the [1, 2, 3].random() case would ideally be handled by some kind of type-level support for non-empty collections. But failing that, adding a ! is a minimal burden on the developer.
I've worked on projects that forbid ! as well, but I think that's a cultural issue in the Swift community that's slowly dying out. The feature exists in the language precisely for cases like this. If allowing ! is out of your control, it's simple to get the same behavior while bypassing your linter by adding a Collection extension that unwraps the random element and fatalErrors when that fails.