You're right — it's not a literal-expression.
However, I think it's correct to describe it under the primary-expression . self rule and not a (potential) type . self rule.
Something like [Int].self or Int?.self is already shaped like a primary-expression followed by .self, and as @John_McCall said it's reinterpreted as the correct thing by a later compiler decision. These things don't need a new rule.
If the type . self rule existed as you proposed, then something like () -> ().self would be allowed under the rule, since () -> () is a type.
(() -> ()).self works because the parenthesized part is shaped like a primary-expression: it's a binary-expression made primary by the parentheses. Just like the other cases, it's reinterpreted as the correct thing later.