SE-0291: Package Collections

tomerd · 2020-11-10T20:10:55.555Z

The review of SE-0291, "Package Collections" , begins now and runs through November 18, 2020.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager or direct message in the Swift forums).

What goes into a review of a proposal?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift.

When reviewing a proposal, here are some questions to consider:

What is your evaluation of the proposal?
Is the problem being addressed significant enough to warrant a change to Swift?
Does this proposal fit well with the feel and direction of Swift?
If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Thank you for helping improve the Swift programing language and ecosystem.

Tom Doron
Review Manager

Mordil · 2020-11-10T22:53:45.689Z

I feel like this was prematurely moved from the pitch phase to the review phase when a critical piece of feedback was given and not addressed (still isn't in the current proposal):

Package Feeds

Both the CLI format and the feed format work together to form the user-facing experience for this feature, particularly as you state that the goal is to let anyone publish a feed, implying that making both publishing and consuming these feeds ergonomic are big parts of what you’re reaching for here.

Therefore, I would expect both the CLI and JSON formats to be reviewed in one piece. Put another way, it would be impossible to deliver any form of this feature if either half were rejected; they do not stand alone.

What is your evaluation of the proposal?
In general I'm +1, but we should address the point above.

Is the problem being addressed significant enough to warrant a change to Swift?
This provides a foundation of interacting with more authoritative sources like a registry or central index to aid in the real problem of discoverability of packages.

Does this proposal fit well with the feel and direction of Swift?
SwiftPM has had a direct intention of being decentralized as much as possible and this continues in that philosophy.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
NPM in the JavaScript world and NuGet in C# have a concept of providing additional registries when searching and interacting with the tool aside from the central authority.

This is very similar, but without a default central authority.

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
I read through the original pitch and participated in its discussion.

yim_lee · 2020-11-11T03:07:36.925Z

Thank you for pointing out the missing piece @Mordil.

Agreed that the package collection (feed) format is an important part of the feature, therefore we are preparing a separate pitch for it.

We left the collection format out of this proposal intentionally because evolving the collection format requires no proposal, but evolving the CLI does. We also feel that this proposal has adequately shown the interactions one can do with package collections and the types of information they carry, which should be enough for the review of the CLI commands.

daveverwer · 2020-11-11T08:40:23.350Z

I’m happy to see this move to proposal stage, nice work Tom, Yim, and Boris!

@finestructure and I meant to chime in on the other thread but time got away from us, so I’ll post here instead.

It’s good to see the role of package indexes considered as part of this proposal. Indexes necessarily centralise some aspects of the package ecosystem, but keeping things decentralised wherever possible is an important part of SPM and I’m happy to see this proposal keep that prioritised.

As mentioned, this proposal isn’t about the data format, but @finestructure and I would like to say that we’ll be happy to implement support for this as and when it moves from a proposal into implementation.

What feeds the Swift Package Index will publish are still up for discussion. At a minimum, I could see us producing feeds per package author/organisation and feeds per category (after we implement categories in the index). After that, there are also opportunities to allow people to build collections of packages using the Swift Package Index, and then have those personalised collections exposed as feeds. I won’t say we’ll definitely do that, but it might be something we consider in the future.

Dante-Broggi · 2020-11-11T17:12:36.112Z

The primary question I have is why the global configuration file will be expected at:

~/.swiftpm/config

Although I see rustup uses a similar structure,
I personally prefer the structure used by fish and alacritty in the form of:

~/.config/swiftpm

xwu · 2020-11-11T22:45:25.045Z

yim_lee:

We left the collection format out of this proposal intentionally because evolving the collection format requires no proposal

How come this portion of the feature is not subject to Swift Evolution?

xwu · 2020-11-11T23:09:16.684Z

The commands presented here have incorporated the feedback in the pitch phase, and as such read much better.

I agree with using a similar configuration as git, rustup and other tools as currently proposed. I will not dwell on the bikeshedding here, but if it's an ordinary JSON file, it would be reasonable to have an extension (e.g., config.json). Thought can be given as to whether it might be more in line with Swift conventions not to use abbreviations unnecessarily to call it settings (as rustup does) rather than config.

As to the commands themselves, two points:

Since other commands are mostly verbs (add, describe, remove), it would be more consistent for the command to be called list-profiles as opposed to profile-list.
Regarding describe-collection: it's a bit counterintuitive as the other commands aren't add-collection, remove-collection, etc. It stands to reason that swift package-collections describe would describe a package collection. While reflecting on this issue, do we even need to distinguish the command that describes a collection rather than a package? It doesn't seem like any given URL could be ambiguously both, so the same command could serve both purposes.

typesanitizer · 2020-11-12T01:26:21.955Z

Dante-Broggi:

The primary question I have is why the global configuration file will be expected at:
~/.swiftpm/config

My understanding based on the

(or its equivalent on the specific platform SwiftPM is running on)

in the proposal would be that this respects the XDG Base Directory specification and put the configuration under $XDG_CONFIG_HOME (and $HOME/.config if the former is unset) on Linux.

Dante-Broggi · 2020-11-12T01:41:40.296Z

typesanitizer:

on Linux.

I'm on a Mac, and would still like it to be under ~/.config

ahti · 2020-11-15T11:32:00.474Z

We envision educators and community influencers publishing package collections to go along with course materials or blog posts, removing the friction of using packages for the first time and the cognitive overload of deciding which packages are useful for a particular task. We also envision enterprises using collections to narrow the decision space for their internal engineering teams, focusing them on a trusted set of vetted packages.

Are there further or more concrete motivating examples you could share and/or is there precedent for this sort of concept in other languages/package managers? I feel like this introduces a lot of surface-area and finicky behaviour that wouldn't really be all that useful in the examples mentioned. So far I'd be -1 on this.

For a blog post or course material, as a reader, I would rather have some lines I can copy directly into my package manifest, or just package names and versions, than going through the multi-step process of adding a package collection, getting the details for each package from the cli, filling that into my package manifest, and then possibly removing the package collection again.

As writer, including a list of packages with my content directly also makes sense. It allows me to introduce packages at points in my content where they are relevant (think a list of packages per assignment, rather than a big collection for the course that readers have to find their own way through, which sounds horrible for students in particular; I wouldn't want some confused student to implement some big functionality themselves because they didn't think of the right keyword to search for), I can provide additional information about the packages without breaking the flow of my medium, and it means I don't have to worry about having to somehow produce a JSON file of unknown format (as far as the proposal goes), and—this applies to course materials more than blogs—host that somewhere, hopefully somewhere that doesn't go away and render the content harder to follow in the future, maybe as soon as next semester.

Consequently, I don't really see how introducing a package collection would remove friction in using packages in these examples. Writers and readers alike would need to know both what they currently need to know about how swift packages work, plus how the package collection concepts and commands (and JSON format) work, and the process involves more steps when using a package collection compared to including package lists inline.

The list of vetted packages to be used in an engineering team I think is the strongest example here, although to be a full solution, some more tooling to verify that only packages from a particular collection are used in a companys projects would be required. I also think this example is hurt somewhat by the proposed limitation of only having the latest patch version in each major-minor-combination includable in a collection, because it kind of bypasses the vetting aspect.

Apart from motivation, I too would like to see metadata fields and the format of collection and configuration file mentioned in the proposal, and I think the configuration system is needlessly complicated. Templating of configuration files and provisioning of machines configured for specific users is best handled by one system for all tools rather than by each tool on its own. Plus the choice of the git config file format confuses me. Is this supposed to include things like include and includeIf? onbranch? Does any of the type system from git config apply? Is there good tooling to parse and manipulate git config format files in popular languages? I think something like a simple INI format, or TOML maybe, would be more appropriate, but like I said I don't think this supplemental file is necessary at all. (Also, maybe ~/.config/swift/pm as directory, to have a nice place for other swift toolchain configuration?)

Lastly, the future directions section I think should elaborate on how in particular this proposed feature would integrate into "[f]uture work to support more dynamic server-based indexes". If this is to be a foundation for some bigger feature, I think it should be reviewable in that context (and maybe pitched together with that bigger feature as well).

yim_lee · 2020-11-16T22:51:15.731Z

Thanks for the feedback @ahti. You have made some really good points.

Personally, I think I would find intent-based package collections useful. Suppose I am starting out to build web services using Swift. If someone has published a package collection that includes web framework, database clients, etc., it would help me locate packages that fit my purpose much more quickly and easily. I would also think that packages recommended by others tend to be more popular and well-maintained.

ahti:

Consequently, I don't really see how introducing a package collection would remove friction in using packages in these examples. Writers and readers alike would need to know both what they currently need to know about how swift packages work, plus how the package collection concepts and commands (and JSON format) work, and the process involves more steps when using a package collection compared to including package lists inline.

You're right. Introducing package collections doesn't necessarily solve the most pressing UX issues with using packages. However, we believe it does open up new opportunities--keep in mind that the new SwiftPM CLI commands will be built on top of a new set of libSwiftPM APIs. Clients of libSwiftPM can potentially make use of these APIs to build some of the enhancements you described.

ahti:

I too would like to see metadata fields and the format of collection

We should post the package collection format this week.

ahti:

Plus the choice of the git config file format confuses me.

I want to clarify that we only plan to have syntax similar to that of git config files, not all of the semantics.

yim_lee · 2020-11-17T00:56:28.748Z

For those interested, I created a new post for Package Collection Format.

There will be an OSS tool to help generate package collections adhering to the format.

tomerd · 2020-11-18T19:25:41.801Z

The review for SE-0291 Package Collections ran from November 10 to November 18, 2020.

The core team has decided to return this proposal for revision.

During the review, several community members requested to learn more about the Package Collection data format. The core team felt the proposal should be amended to explicitly call out if the data format is part of the feature specification or not, and provide reasoning for its inclusion or exclusion.

For example, it makes sense for the data format to be included if it is considered a stable API that users may build tools around, and conversely it makes sense to exclude if it is considered a non-stable API that users should not be generated directly.

Thank you for your feedback thus far and helping improve the Swift programing language and ecosystem.

Tom Doron
Review Manager

dabrahams · 2020-11-22T02:44:06.763Z

It's great to see progress on this feature set! Sorry I'm catching up with this topic so late.

I just have one very minor comment at this point: the final s in the command syntax ”swift package-collections …” seems unwarranted. As far as I can tell, none of the commands would read any less well as ”swift package-collection whatever,” and many would read better that way.

Very much looking forward to the next review cycle; thanks to the proposers for all your hard work on this!

Tino · 2020-11-22T10:42:23.388Z

on ~/.swiftpm/
Having a single folder for configuration files is not as bad as putting them directly in the users home directory — but even on Unix there's a better standard nowadays, and macOS even has the well established ~/Library/ to organize such data nicely.
It is a pity that many old programs still rely on a hack to hide their pollution of the home directory, but it would be sad if a modern project needlessly breaks with sensible system standards.

ktoso · 2020-11-22T15:21:20.745Z

Tino:

— but even on Unix there's a better standard nowadays

Honest question: first time I hear about that. No other tools I know of use some other space than $home/.myThingy - what do you mean specifically? Can you provide references to what you’re saying?

I don’t mean Mac, I mean Linux and windows users. What are you suggesting?

Tino · 2020-11-22T15:44:19.087Z

ktoso:

No other tools I know of use some other space than $home/.myThingy - what do you mean specifically? Can you provide references to what you’re saying?

Sure:
https://wiki.gnome.org/Initiatives/GnomeGoals/XDGConfigFolders
https://wiki.archlinux.org/index.php/XDG_Base_Directory#Specification
https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
This might even be inspired by next/mac, so it really feels strange when Apple drops a well thought-out idea for something that's only there because "everybody does it" (but even that changes nowadays).

Actually, I would be fine with simply using ~/Library on every platform — but with the specification, it is even possible to write tools that adhere to system standards on Mac as well as on other operating system.

ahti · 2020-11-22T19:42:45.028Z

Tino:

Actually, I would be fine with simply using ~/Library on every platform

That would be horrible, and only serve to strengthen the (still quite prevalent) perception that Swift is just that Apple language. We should follow platform conventions here, so Library on macOS, XDG dirs on Linux and (I assume, I'm not that familiar with conventions here) AppData on Windows.

tomerd · 2020-12-08T23:05:58.386Z