It depends what you mean by average and by worst case.
For example, Array.append is amortized O(1), but sometimes an individual call to append will take O(n) because the array needs to grow. But But because of how Array's growth strategy works, you can think of this as kind of "averaging out" to O(1).
But that's not what's happening with Set. Here, I think the justification for the complexity is that it's not reasonable to have to consider the case where all values hash the same. The expectation is that all types must provide a high-quality hash. Now that Swift will auto-generate one for you, and makes it easy to write custom ones, anything else can be considered a bug in the conforming type (possibly this could be emphasized more in documentation). This assumption, combined with a decent implementation of Set, means it's OK to assume constant-time lookup, because collisions should be modest and Set should never deteriorate into just being an unordered list in practice.
This is similar to the requirement that your implementation of == be reflexive. If it isn't, then that will break Array's == under some circumstances. But that doesn't mean we need to sprinkle caveats into the documentation for Array, that would just end up spreading FUD.