Package Feeds

There is also no information here about the JSON format of the package feed itself. Is there a reason for that omission?

9 Likes

I like the idea!

What would happen if a single package is in multiple feeds? Would it be displayed multiple times?

What would happen if someone adds all the packages into a single feed? I bet someone would try to do that, because it's the closest thing we would have to a package registry. I don't have any experience in things like that, so I have no idea if the solution is fast enough for that, or would grind to a halt.

Does that mean swiftpm would now be added to crontab or something like that?

3 Likes

First of all, I like the idea of a searchable feed

However, this proposal, dropped out of the blue, raises some questions: Is this about the format only, or does it convey implicit official curated Packages Index feed. Will SPM come with a default feed pre-installed? What feed will Xcode use? Does anyone plan to maintain a vetted list of packages? How does that play with long-time-announced GitHub registry support?

As we know, if SPM will come with a pre-installed feed, it will become de-facto official feed - I'd rather see such feed outside Apple rules (because why otherwise).

For the UX, I'm missing something like swift package feed index [directory] that gets the Package.swift and output index.json suitable for the feed.

5 Likes

Package Feeds tell what and where packages that might be relevant to the developer are so SwiftPM can fetch them. Right now package locations are primarily Git repositories, but as part of the Package Registry proposal SwiftPM will gain support for Package Registry URLs as well.

The Package Registry work is still active AFAICT:

This proposal is one way of solving the package discoverability problem that Package Registry doesn't address directly.

There is also no information here about the JSON format of the package feed itself. Is there a reason for that omission?

That's right. The focus of this proposal is the SwiftPM CLI so we didn't include details on the feed format. One can assume the format will be defined and published at some point in time.

2 Likes

This is probably when feed ordering comes into play.

Besides conforming to specific JSON format, Package Feeds will have to meet certain criteria to be used. More details will come later separate from this proposal.

1 Like

In a world where we have registries, what role would these feeds have? At best they seem like a way to generate a static registry, so perhaps they should embrace that role. Allow feeds to act as container for sets of recommended packages that are easily consumed in package files. Easily consuming a whole set of packages would be great. In fact, a feed could even be another type of artifact published by registries as well as available as a standalone file. I'd certainly rather see registry support first.

We really want to make it possible for anyone to publish Package Feeds, and a user can have multiple feeds in case that detail is not clear.

1 Like

Does this proposal imply that SwiftPM will have some bundled package feed enabled by default (provided by SwiftPM maintainers or whoever else) as soon as a user installs a Swift toolchain with SwiftPM?

Or to paraphrase it, what's the expected behavior for running swift package search --keywords yaml on a fresh installation, when a user didn't add any package feeds on their own yet? Will it display a warning about no feeds, or will it start searching in some default pre-installed feed?

No, this proposal doesn't make any implication on default feed(s).

1 Like

Many of my concerns have been brought up by others, but this one is a bit more "bikeshed" in nature - but not entirely.

What's the rationale for putting several of the commands under the package category, when they interact with data that is not specific to a given Swift package, as far as I can tell from the pitch?

All of the swift package subcommands today have to do with the current Swift package in the directory that swift package is invoked from:

SUBCOMMANDS:
  clean                   Delete build artifacts
  completion-tool         Completion tool (for shell completions)
  config                  Manipulate configuration of the package
  describe                Describe the current package
  dump-package            Print parsed Package.swift as JSON
  edit                    Put a package in editable mode
  generate-xcodeproj      Generates an Xcode project
  init                    Initialize a new package
  reset                   Reset the complete cache/build directory
  resolve                 Resolve package dependencies
  show-dependencies       Print the resolved dependency graph
  tools-version           Manipulate tools version of the current package
  unedit                  Remove a package from editable mode
  update                  Update package dependencies

It's a little surprising that the command swift package search would instead query some "feed" data source, rather than the current package's dependency graph.

And every new user would have to understand the difference between swift package info and swift package describe.

6 Likes

Can we add some examples that don't assume packages hosted on GitHub or authors with GH usernames? It worries me to see those kinds of assumptions getting baked in, even if only implicitly.

8 Likes

Since this question came up several times, we could probably have done a better job articulating the difference between Package Index, Package Feeds and Package Registry. We believe these are three different components with different purposes.

Package Registry is focused on hosting and serving package sources as an alternative to fetching them directly from git. The goal is to provide better immutability, durability and potentially improve performance and security. This initiative is in-progress and governed by https://github.com/apple/swift-evolution/pull/1179 (cc @mattt) .

Package Index is focused for providing a search index for packages. The goal is to improve discoverability of packages that may be hosted anywhere, and provide a rich set of metadata that helps making informed decisions when choosing dependencies. The Index indexes the package core metadata available in Package.swift as well as additional metadata from additional and external sources. An example of a package index is https://swiftpackageindex.com (cc @daveverwer @finestructure)

Package Feeds which are the subject of this pitch are closer to the Package Index than to the Package Registry. Feeds are also designed to make discovery easier, but focused on simpler curation lists that can be easily shared rather than on larger scale indexing and ranking system that requires infrastructure. This design is the first step in teaching SwiftPM about discovery of packages and future work to support Package Indexes can build on this initial design.

16 Likes

Both the CLI format and the feed format work together to form the user-facing experience for this feature, particularly as you state that the goal is to let anyone publish a feed, implying that making both publishing and consuming these feeds ergonomic are big parts of what you’re reaching for here.

Therefore, I would expect both the CLI and JSON formats to be reviewed in one piece. Put another way, it would be impossible to deliver any form of this feature if either half were rejected; they do not stand alone.

@tomerd, this is a great explanation; I think it would be very helpful to incorporate this as part of the proposal text and plan of record.

I’d like to echo this concern regarding the proposed CLI interface. Moreover, I would add that I think the deep nesting mars usability (“swift package feed search module” seems an arbitrary ordering of words), and it confuses not only operations on this package versus operations on packages, but also operations on this feed versus operations on feeds.

I think it would be helpful to explore how much of the nesting can be simplified away, and how consistently we can apply the use of singular versus plural words to clarify this distinction:

swift package list-feeds
swift list-package-feeds
swift package-lists # maybe just call them lists?

swift package-lists add
swift package-lists remove
swift package-lists search
swift package-lists -info

Edit: Do other package managers (NPM, Cargo, etc.) have similar features? What’s their CLI design for this? Can we align the syntax so that this becomes a transferrable skill that users can learn once and apply more widely? (And, what’s their JSON format? Can we be interoperable with other language-agnostic package managers’ lists?)

8 Likes

Who are the users who are expected to publish a curated lists of packages as a Package Feed instead of adding packages to a Package Index?

Who are the users who are expected to discover packages through Package Feeds instead of the Package Index? And how do those users find those feeds? Will we have later a proposal for an Index of Feeds?

I can think of a few contexts, such as education, or companies that maintain a list of vetted packages. Yet I would appreciate if the proposal would say more about the target audience, and how Package Feeds are a product that match their needs.

2 Likes

This is great. Reminds me of nugget feeds. https://docs.microsoft.com/en-us/nuget/hosting-packages/overview

It would be unfortunate if we ended up with multiple feed formats to accommodate registries. Or custom json formats for things like https://developer.apple.com/documentation/swift_playgrounds/creating_a_subscription

If we can align to feeds being just a way to have “private” registries then I’m all for it.

Thank you,
Chéyo

I agree with the feedback from @Mordil and @xwu that swift package might not be the right top-level command for these. I initially thought about introducing a new top-level swift feed (or maybe swift package-feed) command, but went with swift package for simplicity's sake. The fact that the new commands don't fit well with the existing once under swift package and the deep nesting convinced me otherwise now.

If we regroup under swift feed, we would end up with these commands:

$ swift feed profile-list [--json]
$ swift feed list [--json] [--profile NAME]
$ swift feed refresh [--profile NAME]
$ swift feed add https://www.example.com/packages.json [--order N] [--profile NAME]
$ swift feed remove https://www.example.com/packages.json [--profile NAME]
$ swift feed info https://www.example.com/packages.json [--profile NAME]
$ swift feed search [--json] [--profile NAME] --keywords yaml
$ swift feed search [--json] [--profile NAME] --module yams

We do end up with a conflict between info for a feed and a package, we could solve this with:

$ swift feed package-info [--json] [--profile NAME] https://github.com/jpsim/yams
$ swift feed package-info [--json] [--profile NAME] --version 4.0.0 https://github.com/jpsim/yams

Could we do:

swift feed describe [--json] [--profile NAME] [--version SEMVER] https://github.com/jpsim/yams
  1. This gives some symmetry with the swift package describe command, even though the data returned from the two commands are different
  2. It's shorter
  3. It's easier to explain given that info is more of a noun, matching the result being the noun info of the feed being provided, and describe is a verb, which implies a bit more of work being done to display the info of the given package
    • I know that I originally stated it might be confusing between the two commands, but that's more of an issue with the top category being package as they wildly differ in information source and results, and feeds are going to be inherently "less" frequently used and more "advanced" as a feature
1 Like

Would this also enable something such as:

swift package add <packageName>

Where if a matching package is found in one of the feeds it would be added to the dependencies in Package.swift?

I'm thinking to people unfamiliar with SPM: IMHO, while the search command sure would help discovering packages in the feed, it doesn't remove the need for a new user to learn the API of Package.swift in order to actually add this new dependency, which still causes friction today.

Something like this might be possible in the future, but today we don't have mechanical editing capabilities for the package manifest.

1 Like

I think having both describe and info commands would be very confusing.

That said, I think I slightly prefer describe, so maybe changing the feed-related one to describe-feed is an option. swift feed describe-feed is a bit weird, but I would expect the package-related command to be used more than the feed-related one.

1 Like
Terms of Service

Privacy Policy

Cookie Policy