I'm trying some performance tests and it seems like IndexSet and Set<Int> are about the same in Debug build, but in a Release build Set<Int> is doing much better.
While I'm not too familiar with the internals, my understanding is that it optimizes its contents into Ranges whenever possible to conserve memory. For example an IndexSet of [1, 2, 3, 4, 5, 7] will be represented under the hood as [1...5, 7...7].