A type nested inside a generic type can be "lifted" to a top-level type whose generic arguments and requirements are the concatenation of the outer type and inner type. Eg,
struct Outer<T> where T : Equatable {
struct Inner<U> where U : Sequence, U.Element == T { ... }
}
is the same as this:
struct OuterInner<T, U> where T : Equatable, U : Sequence, U.Element == T { ... }
Now suppose you have a type like this:
protocol Outer {
struct Inner<U> where U : Sequence, U.Element == Self { ... }
}
Since a protocol Proto has a single generic parameter Self with a single requrement Self : Proto, this is in fact equivalent to writing:
struct ProtocolInner<Self, U> where Self : Outer, U : Sequence, U.Element == Self { ... }
The only difference is that for some type C conforming to Outer, I write C.Inner<Foo> instead of writing ProtocolInner<C, Foo>.
This also means that referencing the nested type as a member of the protocol itself is not valid! Eg, Outer.Inner<Foo> is not valid because Outer does not conform to itself. Even if the generic requirements of Inner do not reference Self or associated types, the bodies of its methods (or methods defined in extensions) might, so in general we can only allow types nested inside protocols to be seen as members of conforming types, and not of the existential type.
Note that all of the above applies equally well to nested type aliases as well as nested nominal types. We already worked out most of the edge cases for nested type aliases so allowing nominals to be nested inside protocols won't be a big step.
Edit: This is about struct/enum/class declarations nested inside protocols. Protocols nested inside other protocols runs afoul of the "protocol cannot be in generic context" rule, which is more difficult to resolve. Right now we assume a protocol only has a single generic parameter Self, and there are multiple independent ways of generalizing this, all of them rather difficult.