I'm not sure that anyone here is saying we should start building a set of libraries right now, in fact, many of the libraries we desire have already been built by multiple different developers, and that's the problem! Adoption of swift packages is slow and fragmented because there is little unity in the swift package community. Different people find and use different libraries for a plethora of reasons. A developer may write his library to do one thing better than this other library that already does basically the same thing and neither library developer works together to find a unified best-approach, so we end up with multiple copies of basically the exact same library all competing to become the most popular which just results in an even more fragmented community.
The core team has said that once one of these libraries "wins" and becomes the de-facto standard, they might consider it for addition to the standard library, or possibly a non-standard library. Swift has been out for years and there are more and more new packages being created and the likelihood of one of them "winning" is becoming slimmer and slimmer. We see this all the time, a developer stops actively working on his library and so someone else writes a new one that's even better. They work on it for a while and then they stop actively working on it. Rinse and repeat.
For example, I know of three libraries for file management/manipulation and each has its pros/cons (there are probably more, but I only know of 3). One has a cleaner interface and significantly larger community, but doesn't support linux. The second supports all OS's but its interfacing isn't quite as nice. The third I haven't ever used because I found the other two before I learned about it and so I haven't tried it. It may actually be better than both! But I'm exhausted from all my investigation into the other two and too heavily invested in the one I'm currently using to justify another switch. This is one of the biggest problems I have with the current state of the swift package community. Packages are difficult to find and there are too many similar packages.
I think that what we want (at least me anyways) is some place where we can work together with a little bit of @core-team oversight to decide on best design approaches and coding practices for some common functionality libraries that already exist anyways. Things like file management, big nums, random functionality ([Proposal] Random Unification), crypto, or whatever it is. This way we could come to a community driven and commonly accepted library, rather than forcing developers to decide between several versions of essentially the same library.
A separate "Libraries" thread here on the swift forums would facilitate this communication. Currently devs are just hoping and praying people find their library through just the right google searches and choose to adopt it rather than write their own. It also opens the door for easier communication backed by this entire community, rather than opening a GitHub issue or pull request and hoping the maintainer(s) are active.