It comes from Swift’s ability to add extensions that can add protocol conformances and methods. While I think the probability that collisions happen is low, it still means that if you follow strict SemVer, adding these kinds of things would require a major bump.
Yeah, strongly agreed — I argue for that one bit precisely above (I believe adding public protocol conformances to public classes is a semver major break).
There are a few things that are less obvious — for example, adding new public or open methods to open classes is a potential source break (because a subclass may be introducing them for their own use). But in general I would argue that the leeway one has in minors, with a little care in design, can be sufficient to introduce significant changes.