I have a very similar feeling at times, so please let me elaborate on this, and I hope @Karl can clarify his point. This will probably seem long-winded, but please bear with me.
I initially was very excited when Swift was open sourced, and the goal of "Swift conquering the world" as Chris Lattner stated it at the time, made perfect sense. Swift strikes a great balance for me personally and it seemed that improving the cross-platform support would make it a real "language for the next 40 years" But recently, I no longer see enough people taking that goal seriously or looking in the broader context at how to achieve this.
There are also many small frustrations around the ecosystem and the overall process. I've already linked to not so great the experience with server-side Swift other people had above in the thread. The gist is that the developer tools are still not very usable on other platforms. SourceKit-LSP was supposed to fix this, but here we are, 5 years later after the language became publicly available, and it's still not possible to work on more or less real project outside of Xcode.
While working on the WebAssembly port of the compiler I found the compiler codebase very hard to work with, as compared to other compiler codebases, I've previously tinkered with Rust, PureScript, Babel transpilers. The amount of time you need to spend on making even a small contribution to the Swift compiler is currently enormous and requires expensive hardware to work on it comfortably, produce debug builds, set up the environment, dig up the documentation, brush up C++ skills etc.
Important enough: Swift is probably the only major compiler of modern languages that isn't bootstrapped. Scala compiler is written in Scala, Go compiler is written in Go, Rust compiler is written in Rust, Kotlin compiler is written in Kotlin, and only Swift compiler is written in C++.
I bet an important reason for this is the C++ barrier. Many people would like to contribute and participate more in the Swift evolution, but now the requirement is that you have to produce a working patch before a proposal is even considered. I sometimes find myself on Rust Forums and feel overwhelmed by the amount of fundamental activity going on. Rust Generics Working Group, RustWasm Working Group, the infrastructure feels vibrant and moving much faster than what we have for Swift, despite Rust having initially a much smaller mindshare and a very narrow niche.
All in all, the Swift ecosystem for the last few years feels stagnant to me compared to what you can se around. SwiftNIO and Swift4TF were really breakthroughs, but this probably isn't enough. I'm seeing plenty of people around me no longer considering Swift for their mobile apps, primarily because of Flutter and React Native, while Kotlin Native is moving fast too. I understand that there always be a niche for native apps, and there will be people who will use Swift for server-side no matter what. But currently I can only make a compelling argument for using Swift the language, but can't do so for the size of the ecosystem or the quality of the developer tools.
Also, there's the Swift patent thing, which I'm not going to elaborate on. It's been discussed on Swift Forums probably too much at this point, with no progress on that visible.
After all this, it's a bit frustrating to see SwiftUI and Combine, projects with the huge potential for the ecosystem, being released as closed source and with a suprise announcements of new language features that will ship in a few months just for a proprietary framework to be frozen in time for an OS release.
It no longer feels like the language in general has focus. What happened to move-only types? What's the plan for cleaner asynchronous code? I bet there's a lot of under the hood work going on, and I appreciate all of the hard work done by the compiler engineers. I hope this is only a feeling, and a lot of this stuff is easy to fix with more clear and open communication from the Core Team. Obviosly, no one is entitled to any changes in the communication, but I personally think that Swift as an open source project, and the major frameworks released for it, could benefit a lot from more transparency.
These days Microsoft seems to be like a massive open source contributor and I'm jelous of the .NET developer ecosystem, which we unfortunately we don't have in Swift. But I hope we could have it, if major frameworks for Swift were seriously considered to be open sourced. Without it, it all feels like an iOS bubble that's totally dependent on Apple's hardware sales numbers.
I sometimes feel ambivalent about this, but seeing that a lot of people shared similar thoughts in this thread and other threads, I decided to share it, thanks for reading!