Adding a "Packages" page to Swift.org

[daveverwer]

Hello! @finestructure and I have a PR open on the Swift.org website repository that we would love your feedback on.

As part of running the Swift Package Index, we see the work that the Swift community puts into building a great package ecosystem and want to highlight that work by bringing more package information to the Swift website.

This PR starts to do that by adding a new, top-level Packages page containing lists of packages along with links to the Swift Package Index, where people can find more information. The lists will be generated using a combination of metadata from the Swift Package Index and the curation of community package suggestions and will be updated regularly, allowing plenty of different packages to appear on Swift.org.

Preview/Demo

You can see a live preview of the proposed new page here.

What are the package lists under “Package Ecosystem”?

There are currently six package lists. One is based on community activity (more on that below), one highlights packages that use the new macros feature of Swift 5.9, and four are based on keyword categories from GitHub keywords.

The package “cards” in the lists contain a summary description of the package, along with some metadata and author information. We went through a few designs before arriving here, and there’s a balance between providing enough information to show what the package is about without making the page so long that people only ever see the first category!

Community Showcase

After discussion with both Apple and the Swift Website Workgroup (SWWG), we all feel that it’s important to highlight a wide variety of packages currently being developed by and discussed within the Swift community, and the first list on this page seeks to do that.

This showcase will feature packages sourced from community podcasts, newsletters, blogs, and other community publications. Initially, the SWWG will take responsibility for gathering nominations from those community sources and curating them into the Community Showcase list. The SWWG will pass responsibility for curating this list to the Ecosystem Steering Group once it is operational.

Nominations of packages for this list will be via a public thread on the Swift forums. We will encourage the authors/hosts of community publications/podcasts to subscribe to that thread so that they receive email reminders to submit the packages they recently linked to or discussed. The SWWG will ping that thread asking for nominations one week before each page update.

We are proposing only one guideline (in addition to the Code of Conduct) for inclusion in the showcase, and that’s that each package should come from a conversation about other people’s packages rather than those written by the podcast’s host or blog post/newsletter’s author.

We also considered direct community member submission of packages and some kind of community voting process to determine selections. We may still do that once this process is up and running, but it’s not in scope for now.

Finally, it’s important to note that the showcase is exactly as its name suggests. A broad showcase of packages, and it is not intended to be a “best of the best” or anything like that. It’s an opportunity to find a variety of packages, including new ones.

Other package lists

All other package lists are currently sourced from keyword listings or search results on the Swift Package Index.

One question I’m sure will come up is how the category/keyword list packages are chosen. The order and selection of packages comes from our “package score”, which also powers our search ordering on the Swift Package Index. The current algorithm is available in the source code.

How will the page get updated?

Swift.org is a statically generated site, and this PR doesn’t change that. The package lists are powered by a packages.yml data file that we will update via pull request on a regular schedule.

Your feedback

We’d love feedback on the proposal generally but also on a couple of specific points:

1. What categories and searches should we highlight on the page? We want the top-level lists to cycle over time (especially searches like the “macro target” search, as Swift 5.9 gains adoption). We have an initial set, as you can see in the hosted demo. Are these useful? What others would be good additions?
2. The Community Showcase currently only has packages mentioned on two podcasts, Swift Package Indexing and the Empower Apps podcast. Please subscribe to this thread if you’d like a notification when we start the package nomination process, and please mention below or share this post with any people who run community podcasts/publications.

Thanks for reading!
Dave & Sven

[Sherlouk (James Sherlock)]

@daveverwer

The order and selection of packages comes from our “package score”

I do wonder how as a community we look to manage this moving forward, as it currently stands yourself and Sven (as owners of the SwiftPackageIndex project) essentially act as the sole arbiter of "What is a good package?" - the more your definition is used, the more I feel it requires some level of scrutiny or review. While contributions are accepted, ultimately it's you two who decide.

For instance, I personally think too much weight is given to an "App Store Compatible" license which for many reasons may not be picked up, and indeed something I do value (cross-platform compatibility) is not currently taken into account.

Additionally, I feel like both the scoring and the chosen categories (especially something like 'networking') ends up in what is a rather stale list. Of course the packages themselves are great, widely used, and fantastic examples to show - but they're basically never going to change (except Macros which is constantly changing due to how new it is and how many new contributions there are).

I think one potential solution to this would be simply to select random packages within the category, but require some basic requirements that the community agree on (simply to weed out any spam, incomplete, or, for lack of better word, useless packages). I think this would enable the page to be completely different every two weeks, and allow more of the community to be on display rather than just the top six, ultimately keeping it fresh.

None of this is a strong feeling, but just some food for thought as this moves forward. I think this is a great resource for newcomers to the language to see the breadth of the ecosystem of packages. Awesome work.

[daveverwer]

@Sherlouk

I think one potential solution to this would be simply to select random packages within the category, but require some basic requirements that the community agree on (simply to weed out any spam, incomplete, or, for lack of better word, useless packages). I think this would enable the page to be completely different every two weeks, and allow more of the community to be on display rather than just the top six, ultimately keeping it fresh.

This is an interesting idea. We may have to also tweak the preamble text a little bit to explain, but a random selection from a clipped set of packages by score is interesting. Thank you James!

[daveverwer]

@Sherlouk

For instance, I personally think too much weight is given to an "App Store Compatible" license which for many reasons may not be picked up, and indeed something I do value (cross-platform compatibility) is not currently taken into account.

A couple of points to mention here. We are currently doing some work as part of this year's Swift Mentorship programme to add more indicators to the scoring/ranking system, in fact there's one open as a PR right now. This is an ongoing process, and what we have now is far from perfect, but it is what we have!

On the subject of improving the score itself, I'd love to discuss the pros and cons of specific tweaks to the score, and even better enhancements to it through more indicators, but I think the best place for those is in the project. I've opened up a Discussions thread over there, to hopefully keep that subject slightly separate from the Packages page (although they are related, of course)

Thanks again, James!

[sspringer (Stefan Springer)]

@Sherlouk

… indeed something I do value (cross-platform compatibility) is not currently taken into account.

Just as a short side-note, when speaking about cross-platform compatibility, adding Windows to the compatibility matrix should be a good idea.

[wadetregaskis]

I like the idea in spirit, but…

@daveverwer

Finally, it’s important to note that the showcase is exactly as its name suggests. A broad showcase of packages, and it is not intended to be a “best of the best” or anything like that. It’s an opportunity to find a variety of packages, including new ones.

…is it not inevitably going to be [mis]interpreted as an editorialised list; as something akin to the blessed or "best" packages?

Even just looking at the categorisation, I'm uneasy. e.g. Logging (and to a lesser extent Networking). It's annoying enough that there's two completely independent logging systems just from Apple themselves (swift-log vs oslog), let-alone the myriad third-party ones, many of which predate Swift and aren't built on either of the official logging systems. As an "end-developer" I just wish there was one logging system that everyone used so we could all stop reinventing incompatible wheels. Promoting the mess that is endless logging (and networking) frameworks in Swift is making it worse.

I'm not immediately convinced this can be solved other than by leaning into it and making it an actual, explicitly editorialised collection of officially recommended packages. It's essentially the web search problem, including the facets about king-making, popularity reinforcement loops, etc.

Especially since - I assume - this page is mostly going to be visited by newbies to Swift, who are most impressionable and have the least ability to discern wise recommendations from unwise ones.

[Sherlouk (James Sherlock)]

@wadetregaskis

I'm not immediately convinced this can be solved other than by leaning into it and making it an actual, explicitly editorialised collection of officially recommended packages.

I can see some of your points, but I’m not sure even that solves the problem? The “best package” doesn’t exist because that decision is so nuanced based on your particular project and its needs - and so what are the recommendations based on?

An unwise package in your eyes and for your project might be the perfect fit for me.

This is not new to Swift, or our community, and I do think this page still provides value in highlighting the ever-growing ecosystem of community (and official) packages. Perhaps some change to the wording on the page though could make it clear that these are not recommendations, and that you should find the best solution for you.

[taylorswift]

@Sherlouk

Additionally, I feel like both the scoring and the chosen categories (especially something like 'networking') ends up in what is a rather stale list. Of course the packages themselves are great, widely used, and fantastic examples to show - but they're basically never going to change (except Macros which is constantly changing due to how new it is and how many new contributions there are).

@Sherlouk

I think one potential solution to this would be simply to select random packages within the category, but require some basic requirements that the community agree on (simply to weed out any spam, incomplete, or, for lack of better word, useless packages). I think this would enable the page to be completely different every two weeks, and allow more of the community to be on display rather than just the top six, ultimately keeping it fresh.

i concur with this feedback. in fact, i just don’t feel like package “cards” are the right thing for us to be showing on swift.org.

as a package author and index-site maintainer, i understand why “cards” are appealing - we obviously want to get our packages in front of as many eyeballs as possible, and if that funnels traffic to my website, even better. but i don’t think that’s a great experience for visitors.

generally, i would anticipate swift.org to get two main types of visitors:

1. people who are new to swift, and want to know what swift is capable of, and
2. people who are already using swift, and have a specific problem they are looking for a library to solve.

i don’t think that displaying a bunch of assorted package cards is the best way to target audience #1 - every language with more than a handful of users has a package “ecosystem” of some sort, and people in audience #1 are going to care a lot more about the quality of the packages in the ecosystem than the mere fact that the language has an ecosystem with a bunch of random packages in it.

as for audience #2, i think it’s pretty easy to understand why showing the same list of “editorially featured” packages to different people who all have different, specific things that they are looking for is not going to be all that helpful.

moreover, as others have mentioned, i think “cards” have a lot of complexity (and even controversy) that just isn’t needed here:

@wadetregaskis

…is it not inevitably going to be [mis]interpreted as an editorialised list; as something akin to the blessed or "best" packages?

@Sherlouk

I do wonder how as a community we look to manage this moving forward, as it currently stands yourself and Sven (as owners of the SwiftPackageIndex project) essentially act as the sole arbiter of "What is a good package?" - the more your definition is used, the more I feel it requires some level of scrutiny or review. While contributions are accepted, ultimately it's you two who decide.

on one hand, i think it is really early to be thinking about antitrust issues, and let’s be honest, this community has bigger problems than the possibility of Dave and Sven gatekeeping swift packages. if the SPI grew to the scale of say, dafont.com, where people have commercial interest in their listings and the admins had to craft policy to prevent people from gaming the system, the situation would be different. but we are not there yet.

aside about editorial criteria

i recently noticed that packages with test targets now get a ranking boost on SPI. which i think is totally fine, but i wish it would include executableTarget tests like the ones i prefer to write, because i almost never use the test target format. perhaps a heuristic such as “has at least one target with a name that ends in Tests” might be more appropriate?

on the other hand, i think everyone would be better-served if, instead of cards, swift.org just displayed a list of links to example search queries, such as:

Packages for keyword server-side-swift – Swift Package Index
Packages for keyword logging – Swift Package Index

even if i don’t care about logging and i am looking for say, a JSON encoder instead, it’s a lot more useful to me because i have been directed to a general-purpose search interface and can simply substitute the keywords i am interested in. and of course, this would still direct a lot of traffic (and secondary ad clicks) to SPI and confer onto it a similar number of valuable SEO links.

finally, and this is my chance to do some shameless self-promotion, i think it would be a great idea to link to some other community index-sites such as swiftinit.org, which recently got a major upgrade and now serves integrated docs for some critical multi-target packages like swift-nio that are hard to navigate with a single-target documentation engine. my (biased) view is that giving visitors more choices of index providers to consult would be the best way to bury any editorial/antitrust concerns folks may have.

[wadetregaskis]

@taylorswift expressed what I was getting at better than I did, especially re. the likely audience of this page.

@taylorswift

on the other hand, i think everyone would be better-served if, instead of cards, swift.org just displayed a list of links to example search queries…

And/or perhaps simply embed a searcher in the page, so users can directly enter queries and get live results without having to leave swift.org (until they click on a result, of course, for more details). That way they can really quickly get a feel for what's out there; they might be questioning whether they can / should use Swift for their next project, and thus it'd be super helpful (and good marketing for Swift) if they can quickly find relevant packages for their specific domain(s).

In case I wasn't clear from my earlier response, I definitely do like the direction this is going w.r.t. raising awareness of the SPI and generally making it easier for Swift newbies & explorers to find out about the package community.

[wes1 (Wes)]

This is a great idea, and I really appreciate your work (and Apple's support) for the Swift Package Index.

It's nice to be able to go directly to projects of interest. Some context in the cards helps to choose where to go.

But I think we can do better than just a sampling of packages consider popular or good (in some highly-contended fashion).

Others have mentioned focusing on new users. I agree, but my goal is for them to be encouraged and interested. A list of packages helps mature developers with projects in mind meet specific functional needs, but I'd like to inspire them and draw in newbies.

For orienting users to packages, I would do two things (but not here):

1. I would have some overview of developer maturity cycle in Getting Started, before launching into Install instructions. Start by mentioning picking up the language, creating a project, using and publishing packages, and deploying applications and systems, and the available learning resources. That 1 minute orientation to the lay of the land is more useful than diving into the REPL (instead of swift-fiddle?).

2. Revise the Documentation section of the package manager to better explain packages (bearing in mind that the term is used differently in Linux, Java, etc.)

In this preview/demo, I feel explicit claims about the richness, size, and diversity of the ecosystem can seem to be a bit of signaling that triggers more dispute than insight. It seems more convincing to show than tell, and more welcoming to expressly invite users to try to use or make packages, by showing what's cool.

Weirdly, the best and most popular packages can be the most forbidding, requiring entire workflows or application patterns, or offering sophistication beyond mere mortals.

Curation is more than selection; it's also education about appreciation. I'm interested in learning why a package is of interest.

Weirdly, I'm uncomfortable with the package maintainer names (most are collaborations, and many prefer anonymity), but I'd like the curator to speak in her own name and voice about what she likes and why. And I want to hear from many people with different perspectives. Actual people, actual voices, small insights.

So may I recommend a section with a stream of somebody's and nobody's, curators nominating something of interest, for a telling reason?

One way to do it is just to quote the curator and link to the package and to their blog or the Swift forum discussion for more details. I'd be looking to demonstrate all the weird ways packages can be useful or interesting. Functionally, from connecting devices to orchestrating systems, but also other qualities besides functionality: good code, good testing technique, different domains. Some curators might focus on ML workflows, etc. And a good way to introduce a scary behemoth like the vapor mini-ecosystem might be to just point to one small cool bit of real value.

Visiting curators with different takes is a contrast from selected mavens evolving a ranking system with community input or some search based on gold stars or forks. The normalizing aspect creates a target can devolve to the least objectionable process, while a section of diverse curators with diverse reasons might temper and complement that.

Boosting curators as well as package maintainers I think also helps fills a gap in Swift ecosystem, where experts need more encouragement to help others approach things in a Swifty manner.

It only takes a few good examples (refreshed relatively frequently) to be illuminating and encouraging. ("Look at the output of Swift power asserts!") (While any sampling of top-N packages in M categories invites complaints about what was left out.)

P.S. - With respect, the sidebar link doesn't belong just after Download and above Platform Support, before even the Documentation. It should be in Community section, because it more about surfacing open source packages than orienting users to Swift.

[Jon_Shier]

Personally I think the farthest this should go is a search field and list of example searches users can click on like "networking" or "coreml". That way the issues of package quality or editorialization are sidestepped completely. Could even track popular search terms rather than a fixed list.

[Karl]

I like the idea. But I think it could be improved.

• Firstly, I'm not sure what value the podcast links have (I don't listen to podcasts). Do the podcasts include some kind of tutorial about using the library? How much discussion about the library do they include? And how much is important for a Swift newbie?

• When I click on a package, it takes me to the package's SPI page rather than the package's repository . Not sure how I feel about that. I don't love it.

• Some of the selected packages seem a little strange. For instance, concurrency primitives are a relatively advanced topic so I find it strange that Gwendal's Semaphore library is one of the first things I see on the page. I'm sure it's an excellent package, but imagine somebody new to the Swift language, they want to see what kind of cool libraries are available, and we're leading with... uh, semaphores?

• I think the server section could be better organised. The first item is the Vapor HTTP server, which makes sense (HTTP is the most important thing for servers, and Vapor is clearly the biggest HTTP server library in Swift). Next up is Vapor's fluent sub-library. Then MongoKitten. Then another Vapor sub-library about interacting with CLI tools (doesn't seem particularly server related?). Then yet another Vapor sub-library, fluentkit -- actually, this one seems to be some kind of extension library for the fluent sub-library that was mentioned a couple of places earlier?

  Then, after all of that, Hummingbird. Another HTTP server library which is an alternative to the Vapor HTTP server, the first thing we mentioned.

  If I was new to Swift, this wouldn't really be much help. The arrangement is too chaotic.

• Also, when it comes to the Vapor packages, the experience after the click is poor. For instance, I click on ConsoleKit (the CLI library mentioned previously). As mentioned, it takes me to the SPI page.

  The Readme contains a logo and literally nothing else. Not even a description of what the library does.

  Okay... so, stumbling around a bit, there's a link to go to the documentation. Let's try that.

  Huh. That link takes me to the main Vapor documentation. On the sidebar, I see stuff about Fluent, Leaf, Redis, Security... and nothing at all about ConsoleKit. So already I'm lost

• I wonder if we should have a separate section for "databases"/"working with data"? Programmers often need to work with databases, and a quick search on SPI shows 4 high-quality libraries we could surface:

  • GRDB
  • Fluent (would be moved from the server section)
  • MonoKitten (would be moved from the server section)
  • Realm-Swift
  • (Maybe also Apple's SwiftData? Yes it's an Apple SDK package, but so what?)

• Some simple code snippets would be nice. Almost every library can provide something like that. For instance, on Vapor's website, they have this:

  

  Hummingbird has the same kind of thing:

  

  I included the snippet of text about how the API is very familiar because it expresses the point very well. We want people who have used, say, Express.js to see this and think "ah! I know what this is!" Neither of those snippets show anything particularly groundbreaking, but that's not why they have value.

  Hopefully, that feeling of familiarity makes the idea of writing something in Swift more approachable to the people reading the website.

[wadetregaskis]

@Karl

The Readme contains a logo and literally nothing else. Not even a description of what the library does.

That's a particularly good point - if any packages are promoted on swift.org, they should have excellent documentation. Most popular / good Swift packages do have well-written readmes that cover the essentials (what it does in a nutshell, how to 'install', some simple example usage(s), links to deeper docs) so this requirement won't rule out many packages, but it is essential to providing a good impression.

(personally I wish SPI would just hide or delist all packages with dud readmes like the example @Karl pointed out, but that's perhaps a separate conversation)

[daveverwer]

Thanks so much for the feedback so far, everyone. Lots to take in and discuss.

[ileitch]

Sorry for reviving an old thread, but I didn't find a more appropriate place for providing feedback. First of all, congrats on launching this! It's awesome to see the hard work of the community presented on the official site.

My small piece of feedback is that I think it'd be a nicer experience to link directly to the repositories of the individual projects, rather than the package index page. As mentioned above, the high-profile packages that are likely to be showcased already have high quality readmes, so the package index page feels somewhat redundant.

I also personally dislike that the package index seems intended to sideline the original source, removing the ability for visitors to quickly star the project and completely omitting the sponsors button (instead showing its own prominent “Supporters" link).

Many authors spend a lot of time crafting their readmes as a langing-page-type-experience, and unfortunately the package index overrides this with its own metadata section, leaving the readme itself off-screen. Some of the metadata is insightful, but certainly not what I (as a package author) would choose to lead with when introducing the project to new users.

Is it possible for packages to be showcased on swift.org without first being present on the package index?

[taylorswift]

@ileitch

Many authors spend a lot of time crafting their readmes as a langing-page-type-experience, and unfortunately the package index overrides this with its own metadata section, leaving the readme itself off-screen. Some of the metadata is insightful, but certainly not what I (as a package author) would choose to lead with when introducing the project to new users.

i agree, this was a major reason why we decided not to mirror package readmes on swiftinit. we felt that doing so just wouldn’t add enough value for site visitors, so the swiftinit package pages only contain documentation-specific things.

on the other hand, as an author of many packages, i don’t regard the swift package index as an unwelcome middleman, their build data is valuable to me, and we should provide a shortcut to it. perhaps it would be better for everyone if the visually “primary” link in each package’s entry pointed to its repository, and if the package is listed on the index, it could have a smaller “secondary” link to the build metadata.

[ileitch]

Yeah I think a separate link for the Swift Package Index would work. It'd also make it simpler to introduce links to other package indexes in the future. Right now the 1-1 relationship between swift.org and the Swift Package Index site feels a little exclusionary.

[ileitch]

I've put up a PR to link directly to the package repository, and adds a new section for linking to community driven alternative sites: Link directly to package repositories by ileitch · Pull Request #438 · apple/swift-org-website · GitHub.