Package Feeds

Evolution Pitches
1

Package Feeds

Introduction

This is a proposal for adding support for Package Feeds to SwiftPM. A package feed is a curated list of packages and associated metadata which makes it easier to discover an existing package for a particular use case. SwiftPM will allow users to subscribe to these feeds, search them via the swift package command-line interface, and will make their contents accessible to any clients of libSwiftPM. This proposal is focused on the shape of the command-line interface and the format of configuration data related to package feeds.

Motivation

Currently, it can be difficult to discover packages that fit particular use cases. There is also no standardized way of accessing metadata about a package which is not part of the package manifest itself. We envision educators and community influencers publishing package feeds to go along with course materials or blog posts, removing the friction of using packages for the first time and the cognitive overload of deciding which packages are useful for a particular task. We also envision enterprises using feeds to narrow the decision space for their internal engineering teams, focusing them on a trusted set of vetted packages.

Exposing the data of package feeds via libSwiftPM and the swift package command-line interface will also allow other tools to leverage this information and provide a richer experience for package discovery that is configured by the user in one central place.

Proposed solution

We propose to introduce a new concept called Package Feeds to the Swift package ecosystem. Feeds are authored as static JSON documents and contain a list of packages and additional metadata per package. They are published to a web server or CDN-like infrastructure making them accessible to users. SwiftPM will gain new command-line interface for adding and removing feeds and will index them in the background, allowing users to more easily discover and integrate packages that are included in the feeds.

For example, a course instructor knows they intend to teach with a set of several packages for their class. They can construct a feed JSON file, representing those packages. Then, they can post that JSON file to a GitHub repo or a website, giving the URL to that JSON file to all their students. Students use SwiftPM to add the instructor’s feed to their SwiftPM configuration, and any packages the instructor puts into that feed can be easily used by the students.

Detailed design

We propose to add three new sets of commands to the swift package command-line interface that support the following workflows:

  1. Managing feeds
  2. Querying metadata for individual packages
  3. Searching for packages and modules across feeds

We also propose adding a new per-user SwiftPM configuration file which will initially store the list of feeds a user has configured, but can later be used for other per-user configuration for SwiftPM.

Example

A course instructor shares a feed with packages needed for some assignments. The participants can add this feed to their set of feeds:

$ swift package feed add https://www.example.com/packages.json 
Added "Packages for course XYZ" to your package feeds.

This will add the given feed to the user's set of feeds for querying metadata and search.

One of the assignments requires parsing a YAML file and instead of searching the web, participants can search the curated feed for packages that could help with their task:

$ swift package search --keywords yaml
https://github.com/jpsim/yams: A sweet and swifty YAML parser built on LibYAML.
...

This will perform a string-based search across various metadata fields of all packages, such as the description and name. Results will contain URL and description (if any) of each match.

Once a suitable package has been identified, there will also be the ability to query for more metadata, such as available versions, which will be required to actually depend on the package in practice.

$ swift package info https://github.com/jpsim/yams
Description: A sweet and swifty YAML parser built on LibYAML.
Available Versions: 4.0.0, 3.0.0, ...
Watchers: 14
Readme: https://github.com/jpsim/Yams/blob/master/README.md
Authors: @norio-nomura, @jpsim
--------------------------------------------------------------
Latest Version: 4.0.0
Package Name: Yams
Modules: Yams, CYaml
Supported Platforms: iOS, macOS, Linux, tvOS, watchOS
Supported Swift Versions: 5.3, 5.2, 5.1, 5.0
License: MIT
CVEs: ...

This will list the basic metadata for the given package, as well as more detailed metadata for the latest version of the package. Available metadata for a package can incorporate data from the feed itself, as well as data discovered by SwiftPM. For example, by querying the package's repository or gathering data from the source code hosting platform being used by the package.

Profiles

A user can organise their package feeds into multiple profiles, if desired. When adding a feed, an optional profile can be specified and each command also accepts an optional --profile parameter. This can be useful if the user is part of multiple organisations or is using different sets of feeds for disjunct use cases. There is an implicit default profile that will be used if the --profile parameter is omitted, so the use of profiles is entirely optional. Each of the following commands has an optional --profile parameter.

To list all the profiles the user has created, the profile-list command can be used. Once a profile contains no more feeds, it will not appear in this list anymore.

$ swift package feed profile-list [--json]
Organization 1 Profile
Organization 2 Profile
...

Manage Package Feeds

List

The list command lists all feeds that are configured by the user. The result can optionally be returned as JSON for integration into other tools.

$ swift package feed list [--json] [--profile NAME]
My organisation's packages - https://example.com/packages.json
...

Manual refresh

The refresh command refreshes any cached data manually. SwiftPM will also automatically refresh data under various conditions, but some queries such as search will rely on locally cached data.

$ swift package feed refresh [--profile NAME]
Refreshed 23 configured package feeds.

Add

The add command adds a feed by URL, with an optional order hint, to the user’s list of configured feeds. The order hint will influence ranking in search results and can also potentially be used by clients of SwiftPM to order results in a UI, for example.

$ swift package feed add https://www.example.com/packages.json [--order N] [--profile NAME]
Added "My organisation's packages" to your package feeds.

Remove

The remove command removes a feed by URL from the user’s list of configured feeds.

$ swift package feed remove https://www.example.com/packages.json [--profile NAME]
Removed "My organisation's packages" from your package feeds.

Metadata and packages of a single feed

The info command shows the metadata and included packages for a single feed. This can be used for both feeds that have been previously added to the list of the user’s configured feeds, as well as to preview any other feeds.

$ swift package feed info https://www.example.com/packages.json [--profile NAME]
Name: My organisation's packages
Source: https://www.example.com/packages.json
Description: ...
Keywords: best, packages
Created At: 2020-05-30 12:33
Packages:
    https://github.com/jpsim/yams
    ...

Get metadata for a single package

Note: Feeds will be limited in the number of major and minor versions they store per package. For each major/minor combination that is being stored, only data for the latest patch version will be present.

Metadata for the package itself

The info shows the metadata from the package itself. The result can optionally be returned as JSON for integration into other tools.

$ swift package info [--json] [--profile NAME] https://github.com/jpsim/yams
Description: A sweet and swifty YAML parser built on LibYAML.
Available Versions: 4.0.0, 3.0.0, ...
Watchers: 14
Readme: https://github.com/jpsim/Yams/blob/master/README.md
Authors: @norio-nomura, @jpsim
--------------------------------------------------------------
Latest Version: 4.0.0
Package Name: Yams
Modules: Yams, CYaml
Supported Platforms: iOS, macOS, Linux, tvOS, watchOS
Supported Swift Versions: 5.3, 5.2, 5.1, 5.0
License: MIT
CVEs: ...

Metadata for a package version

When passing an additional --version parameter, the info command shows the metadata for a single package version. The result can optionally be returned as JSON for integration into other tools.

$ swift package info [--json] [--profile NAME] --version 4.0.0 https://github.com/jpsim/yams
Package Name: Yams
Version: 4.0.0
Modules: Yams, CYaml
Supported Platforms: iOS, macOS, Linux, tvOS, watchOS
Supported Swift Versions: 5.3, 5.2, 5.1, 5.0
License: MIT
CVEs: ...

Search

String-based search

The search command does a string-based search when using the --keyword option and returns the list of packages that match the query. The result can optionally be returned as JSON for integration into other tools.

$ swift package search [--json] [--profile NAME] --keywords yaml
https://github.com/jpsim/yams: A sweet and swifty YAML parser built on LibYAML.
...

Module-based search

The search command does a search for a specific module name when using the --module option. The result can optionally be returned as JSON for integration into other tools. Lists the newest version the matching module can be found in. This will display more metadata per package than the string-based search as we expect just one or very few results for packages with a particular module name.

$ swift package search [--json] [--profile NAME] --module yams
Package Name: Yams
Latest Version: 4.0.0
Description: A sweet and swifty YAML parser built on LibYAML.
--------------------------------------------------------------
...

Configuration file

The global configuration file will be expected at this location:

~/.swiftpm/config

This file will be stored in a .swiftpm directory in the user's home directory (or its equivalent on the specific platform SwiftPM is running on).

This file will be managed through SwiftPM commands and users are not expected to edit it by hand. The format of this file is an implementation detail but it will be human readable format, likely JSON in practice.

There could be a supplemental file providing key-value pairs whose keys can be referenced by the main configuration file. This can be used as an override mechanism that allows sharing the main configuration file between different users and machines by keeping user-specific configuration information out of the main configuration file. The use of this additional file will be optional and it will be managed by the user. The format will be based on git's configuration files, described here.

Future direction

This proposal shows an initial set of metadata that package feeds will offer, but the exact metadata provided can be evolved over time as needed. The global configuration files introduced here can be used for future features which require storing per-user configuration data.

This design is the first step in teaching SwiftPM about sets of curated packages, with the goal of allowing the community to build trusted working-sets of packages. Future work to support more dynamic server-based indexes can build on this initial design.

Impact on existing packages

There is no impact on existing packages as this is a discovery feature that's being added to SwiftPM's command-line interface.

Alternatives considered

None considered.

24 Likes
2

I think it would be valuable to lay out how this is expected to interact with the proposals for a package registry that came earlier in the year. Is this a precursor feature? Will its functionality be subsumed? Is the package registry idea no longer moving forward?

16 Likes
3

There is also no information here about the JSON format of the package feed itself. Is there a reason for that omission?

9 Likes
4

I like the idea!

What would happen if a single package is in multiple feeds? Would it be displayed multiple times?

What would happen if someone adds all the packages into a single feed? I bet someone would try to do that, because it's the closest thing we would have to a package registry. I don't have any experience in things like that, so I have no idea if the solution is fast enough for that, or would grind to a halt.

Does that mean swiftpm would now be added to crontab or something like that?

3 Likes
5

First of all, I like the idea of a searchable feed

However, this proposal, dropped out of the blue, raises some questions: Is this about the format only, or does it convey implicit official curated Packages Index feed. Will SPM come with a default feed pre-installed? What feed will Xcode use? Does anyone plan to maintain a vetted list of packages? How does that play with long-time-announced GitHub registry support?

As we know, if SPM will come with a pre-installed feed, it will become de-facto official feed - I'd rather see such feed outside Apple rules (because why otherwise).

For the UX, I'm missing something like swift package feed index [directory] that gets the Package.swift and output index.json suitable for the feed.

5 Likes
6

Package Feeds tell what and where packages that might be relevant to the developer are so SwiftPM can fetch them. Right now package locations are primarily Git repositories, but as part of the Package Registry proposal SwiftPM will gain support for Package Registry URLs as well.

The Package Registry work is still active AFAICT:

This proposal is one way of solving the package discoverability problem that Package Registry doesn't address directly.

There is also no information here about the JSON format of the package feed itself. Is there a reason for that omission?

That's right. The focus of this proposal is the SwiftPM CLI so we didn't include details on the feed format. One can assume the format will be defined and published at some point in time.

2 Likes
7

This is probably when feed ordering comes into play.

Besides conforming to specific JSON format, Package Feeds will have to meet certain criteria to be used. More details will come later separate from this proposal.

1 Like
8

In a world where we have registries, what role would these feeds have? At best they seem like a way to generate a static registry, so perhaps they should embrace that role. Allow feeds to act as container for sets of recommended packages that are easily consumed in package files. Easily consuming a whole set of packages would be great. In fact, a feed could even be another type of artifact published by registries as well as available as a standalone file. I'd certainly rather see registry support first.

9

We really want to make it possible for anyone to publish Package Feeds, and a user can have multiple feeds in case that detail is not clear.

1 Like
10

Does this proposal imply that SwiftPM will have some bundled package feed enabled by default (provided by SwiftPM maintainers or whoever else) as soon as a user installs a Swift toolchain with SwiftPM?

Or to paraphrase it, what's the expected behavior for running swift package search --keywords yaml on a fresh installation, when a user didn't add any package feeds on their own yet? Will it display a warning about no feeds, or will it start searching in some default pre-installed feed?

11

No, this proposal doesn't make any implication on default feed(s).

1 Like
12

Many of my concerns have been brought up by others, but this one is a bit more "bikeshed" in nature - but not entirely.

What's the rationale for putting several of the commands under the package category, when they interact with data that is not specific to a given Swift package, as far as I can tell from the pitch?

All of the swift package subcommands today have to do with the current Swift package in the directory that swift package is invoked from:

SUBCOMMANDS:
  clean                   Delete build artifacts
  completion-tool         Completion tool (for shell completions)
  config                  Manipulate configuration of the package
  describe                Describe the current package
  dump-package            Print parsed Package.swift as JSON
  edit                    Put a package in editable mode
  generate-xcodeproj      Generates an Xcode project
  init                    Initialize a new package
  reset                   Reset the complete cache/build directory
  resolve                 Resolve package dependencies
  show-dependencies       Print the resolved dependency graph
  tools-version           Manipulate tools version of the current package
  unedit                  Remove a package from editable mode
  update                  Update package dependencies

It's a little surprising that the command swift package search would instead query some "feed" data source, rather than the current package's dependency graph.

And every new user would have to understand the difference between swift package info and swift package describe.

6 Likes
13

Can we add some examples that don't assume packages hosted on GitHub or authors with GH usernames? It worries me to see those kinds of assumptions getting baked in, even if only implicitly.

8 Likes
14

Since this question came up several times, we could probably have done a better job articulating the difference between Package Index, Package Feeds and Package Registry. We believe these are three different components with different purposes.

Package Registry is focused on hosting and serving package sources as an alternative to fetching them directly from git. The goal is to provide better immutability, durability and potentially improve performance and security. This initiative is in-progress and governed by [Proposal] Swift Package Registry by mattt · Pull Request #1179 · apple/swift-evolution · GitHub (cc @mattt) .

Package Index is focused for providing a search index for packages. The goal is to improve discoverability of packages that may be hosted anywhere, and provide a rich set of metadata that helps making informed decisions when choosing dependencies. The Index indexes the package core metadata available in Package.swift as well as additional metadata from additional and external sources. An example of a package index is https://swiftpackageindex.com (cc @daveverwer @finestructure)

Package Feeds which are the subject of this pitch are closer to the Package Index than to the Package Registry. Feeds are also designed to make discovery easier, but focused on simpler curation lists that can be easily shared rather than on larger scale indexing and ranking system that requires infrastructure. This design is the first step in teaching SwiftPM about discovery of packages and future work to support Package Indexes can build on this initial design.

15 Likes
15

Both the CLI format and the feed format work together to form the user-facing experience for this feature, particularly as you state that the goal is to let anyone publish a feed, implying that making both publishing and consuming these feeds ergonomic are big parts of what you’re reaching for here.

Therefore, I would expect both the CLI and JSON formats to be reviewed in one piece. Put another way, it would be impossible to deliver any form of this feature if either half were rejected; they do not stand alone.

@tomerd, this is a great explanation; I think it would be very helpful to incorporate this as part of the proposal text and plan of record.

I’d like to echo this concern regarding the proposed CLI interface. Moreover, I would add that I think the deep nesting mars usability (“swift package feed search module” seems an arbitrary ordering of words), and it confuses not only operations on this package versus operations on packages, but also operations on this feed versus operations on feeds.

I think it would be helpful to explore how much of the nesting can be simplified away, and how consistently we can apply the use of singular versus plural words to clarify this distinction:

swift package list-feeds
swift list-package-feeds
swift package-lists # maybe just call them lists?

swift package-lists add
swift package-lists remove
swift package-lists search
swift package-lists -info

Edit: Do other package managers (NPM, Cargo, etc.) have similar features? What’s their CLI design for this? Can we align the syntax so that this becomes a transferrable skill that users can learn once and apply more widely? (And, what’s their JSON format? Can we be interoperable with other language-agnostic package managers’ lists?)

8 Likes
16

Who are the users who are expected to publish a curated lists of packages as a Package Feed instead of adding packages to a Package Index?

Who are the users who are expected to discover packages through Package Feeds instead of the Package Index? And how do those users find those feeds? Will we have later a proposal for an Index of Feeds?

I can think of a few contexts, such as education, or companies that maintain a list of vetted packages. Yet I would appreciate if the proposal would say more about the target audience, and how Package Feeds are a product that match their needs.

2 Likes
17

This is great. Reminds me of nugget feeds. Overview of Hosting Your Own NuGet Feeds | Microsoft Learn

It would be unfortunate if we ended up with multiple feed formats to accommodate registries. Or custom json formats for things like Apple Developer Documentation

If we can align to feeds being just a way to have “private” registries then I’m all for it.

Thank you,
Chéyo

18

I agree with the feedback from @Mordil and @xwu that swift package might not be the right top-level command for these. I initially thought about introducing a new top-level swift feed (or maybe swift package-feed) command, but went with swift package for simplicity's sake. The fact that the new commands don't fit well with the existing once under swift package and the deep nesting convinced me otherwise now.

If we regroup under swift feed, we would end up with these commands:

$ swift feed profile-list [--json]
$ swift feed list [--json] [--profile NAME]
$ swift feed refresh [--profile NAME]
$ swift feed add https://www.example.com/packages.json [--order N] [--profile NAME]
$ swift feed remove https://www.example.com/packages.json [--profile NAME]
$ swift feed info https://www.example.com/packages.json [--profile NAME]
$ swift feed search [--json] [--profile NAME] --keywords yaml
$ swift feed search [--json] [--profile NAME] --module yams

We do end up with a conflict between info for a feed and a package, we could solve this with:

$ swift feed package-info [--json] [--profile NAME] https://github.com/jpsim/yams
$ swift feed package-info [--json] [--profile NAME] --version 4.0.0 https://github.com/jpsim/yams
19

Could we do:

swift feed describe [--json] [--profile NAME] [--version SEMVER] https://github.com/jpsim/yams
  1. This gives some symmetry with the swift package describe command, even though the data returned from the two commands are different
  2. It's shorter
  3. It's easier to explain given that info is more of a noun, matching the result being the noun info of the feed being provided, and describe is a verb, which implies a bit more of work being done to display the info of the given package
    • I know that I originally stated it might be confusing between the two commands, but that's more of an issue with the top category being package as they wildly differ in information source and results, and feeds are going to be inherently "less" frequently used and more "advanced" as a feature
1 Like
20

Would this also enable something such as:

swift package add <packageName>

Where if a matching package is found in one of the feeds it would be added to the dependencies in Package.swift?

I'm thinking to people unfamiliar with SPM: IMHO, while the search command sure would help discovering packages in the feed, it doesn't remove the need for a new user to learn the API of Package.swift in order to actually add this new dependency, which still causes friction today.