To the contrary – it is the very broad definition of what can be a Sequence that requires a definition like this. The limitation is not just with Set. You could not map a Range to Self either – the range requirements are even more restrictive than a set's.
A better example would be RangeReplaceableCollection. It redefines filter to return Self, because it does not change the Element type, and RRC has all the components needed to build filter (init() and append(_:)). But you cannot define a similar RRC version of map because you need higher-kinded types to return Self<T> for some mapping Element -> T
It is this potential for confusion over even the basic aspects that is one of the many reasons why my personal view is that HKTs probably do not belong in Swift, should be moved from the "Maybe" to the "Unlikely" section of the generics manifesto, and should maybe even be added to the Commonly Rejected list, at least for now.
(another being the operators proposed here, which are presented as motivating examples but IMO are hostile to even intermediate programmers not familiar with the concepts)