Package Collection Format

[yim_lee]

Package Collections (SE-0291) are short, curated lists of packages and associated metadata that can be imported by SwiftPM to make package discovery easier. This is a draft of the format that all package collections must adhere to.

Creating a package collection

A package collection is a JSON document that contains a list of packages and metadata per package.

To begin, define the top-level metadata about the collection:

• name : The name of the package collection, for display purposes only.
• overview : A description of the package collection. Optional.
• keywords : An array of keywords that the package collection is associated with. Optional.
• formatVersion : The version of the format to which the package collection conforms. Currently, 1.0 is the only allowed value.
• revision : The revision number of this package collection. Optional.
• generatedAt : The ISO 8601-formatted date-time string when the package collection was generated.
• generatedBy : The author of this package collection. Optional.

{
  "name": "Jane Doe"
}

• packages : An array of package objects.

Add packages to the collection

Each item in the packages array is a package object with the following properties:

• url : The URL of the package. Currently only Git repository URLs are supported.
• summary : A description of the package. Optional.
• keywords : An array of keywords that the package is associated with. Optional.
• readmeURL : The URL of the package's README. Optional.
• versions : An array of version objects representing the most recent and/or relevant releases of the package.

Add versions to a package

A version object has metadata extracted from Package.swift and optionally additional metadata from other sources:

• version : The semantic version string.
• packageName : The name of the package.
• targets : An array of the package version's targets.
  • name : The target name.
  • moduleName : The module name if this target can be imported as a module. Optional.
• products : An array of the package version's products.
  • name : The product name.
  • type : The product type. This must have the same JSON representation as SwiftPM's PackageModel.ProductType .
  • target : An array of the product’s targets.

{
  "name": "MyProduct",
  "type": {
    "library": ["automatic"]
  },
  "targets": ["MyTarget"]
}

• toolsVersion : The tools (semantic) version specified in Package.swift .
• minimumPlatformVersions : An array of the package version’s supported platforms specified in Package.swift . Optional.

{    
  "name": "macOS",    
  "version": "10.15"
}

• verifiedPlatforms : An array of platforms in which the package version has been tested and verified. Valid platform names include macOS , iOS , tvOS , watchOS , Linux , Android , and Windows . This is different from platforms in Package.swift . Optional.

{
  "name": "macOS"
}

• verifiedSwiftVersions : An array of Swift versions that the package version has been tested and verified for. Values must be semantic version strings. This is different from swiftLanguageVersions in Package.swift . Optional.
• license : The package version's license. Optional.
  • name : License name. SPDX identifier ( e.g. , Apache-2.0 , MIT , etc.) preferred.
  • url : The URL of the license file.

Other requirements

• The number of versions included for each package will be limited--the current proposal is at most two major versions and up to three minor versions per major version.

Sample package collection

{
  "name": "Sample Package Collection",
  "overview": "This is a sample package collection listing made-up packages.",
  "keywords": ["sample package collection"],
  "formatVersion": "1.0",
  "revision": 3,
  "generatedAt": "2020-10-22T06:03:52Z",
  "packages": [
    {
      "url": "https://www.example.com/repos/RepoOne.git",
      "summary": "Package One",
      "readmeURL": "https://www.example.com/repos/RepoOne/README",
      "versions": [
        {
          "version": "0.1.0",
          "packageName": "PackageOne",
          "targets": [
            {
              "name": "Foo",
              "moduleName": "Foo"
            }
          ],
          "products": [
            {
              "name": "Foo",
              "type": {
                "library": ["automatic"]
              },
              "targets": ["Foo"]
            }
          ],
          "toolsVersion": "5.1",
          "verifiedPlatforms": [
            { "name": "macOS" },
            { "name": "iOS" },
            { "name": "Linux" }
          ],
          "verifiedSwiftVersions": ["5.1"],
          "license": {
            "name": "Apache-2.0",
            "url": "https://www.example.com/repos/RepoOne/LICENSE"
          }
        }
      ]
    },
    {
      "url": "https://www.example.com/repos/RepoTwo.git",
      "summary": "Package Two",
      "readmeURL": "https://www.example.com/repos/RepoTwo/README",
      "versions": [
        {
          "version": "2.1.0",
          "packageName": "PackageTwo",
          "targets": [
            {
              "name": "Bar",
              "moduleName": "Bar"
            }
          ],
          "products": [
            {
              "name": "Bar",
              "type": {
                "library": ["automatic"]
              },
              "targets": ["Bar"]
            }
          ],
          "toolsVersion": "5.2"
        },
        {
          "version": "1.8.3",
          "packageName": "PackageTwo",
          "targets": [
            {
              "name": "Bar",
              "moduleName": "Bar"
            }
          ],
          "products": [
            {
              "name": "Bar",
              "type": {
                "library": ["automatic"]
              },
              "targets": ["Bar"]
            }
          ],
          "toolsVersion": "5.0"
        }        
      ]
    }
  ]
}

[xwu (Xiaodi Wu)]

@yim_lee

A package collection is a JSON document

Have you considered the use of alternative formats such as TOML (as is used in Cargo) or YAML rather than JSON?

@yim_lee

To begin, define the top-level metadata about the collection:

• title : The name of the package collection, for display purposes only.
• description : A description of the package collection. Optional.
• keywords : An array of keywords that the package collection is associated with. Optional.
• formatVersion : The version of the format to which the package collection conforms. Currently, 1.0 is the only allowed value.
• revision : The revision number of this package collection. Optional.
• generatedAt : The ISO 8601-formatted date-time string when the package collection was generated.
• generatedBy : The author of this package collection. Optional.

Was this format designed de novo, or is it based on some prior art or existing standard (as have been discussed by the community in the past)? How does the format proposed here compare to what's used by other package managers?

@yim_lee

Other requirements

• A package collection can list a maximum of 50 packages.
• Package versions must include at most two major versions and up to three minor version per major version.
• The package collection JSON file must not exceed 100KB in size.

What is the rationale for these limits (and in particular, 50 packages and 100 KB)? How will this be enforced? If a file is 100 KB in one file system but more than that in another, will it be usable on one machine but rejected by another?

[yim_lee]

@xwu

Have you considered the use of alternative formats such as TOML (as is used in Cargo) or YAML rather than JSON?

We believe JSON is the format that is the most straight-forward to use in this case.

@xwu

Was this format designed de novo, or is it based on some prior art or existing standard (as have been discussed by the community in the past)? How does the format proposed here compare to what's used by other package managers?

The format was designed de novo for the use cases described in the package collections proposal. I personally am not aware of previous discussions on this. Do you think there is information missing or shouldn't be included?

@xwu

What is the rationale for these limits (and in particular, 50 packages and 100 KB)? How will this be enforced? If a file is 100 KB in one file system but more than that in another, will it be usable on one machine but rejected by another?

The actual numbers are debatable, but since SwiftPM will be the primary consumer of package collections and clients of libSwiftPM will be able to access the data, there needs to be limits in place to help ensure smooth UX. The limits will be enforced when SwiftPM imports a package collection.

[ktoso (Konrad 'ktoso' Malawski 🐟🏴‍☠️)]

@yim_lee

@xwu

Have you considered the use of alternative formats such as TOML (as is used in Cargo) or YAML rather than JSON?

We believe JSON is the format that is the most straight-forward to use in this case.

To +1 what Yim's saying here:

A different, human readable, format would make sense if humans were to write/edit/read those files -- but they won't, right? It is for tools to generate and consume, thus I don't think formats like YAML/TOML/HOCON add value here. If anything, another binary format could be fun some day to be slightly more efficient, but JSON seems like a good default IMHO

[dabrahams (Dave Abrahams)]

@yim_lee

The actual numbers are debatable, but since SwiftPM will be the primary consumer of package collections and clients of libSwiftPM will be able to access the data, there needs to be limits in place to help ensure smooth UX. The limits will be enforced when SwiftPM imports a package collection.

I beg you to please reconsider having hard limits. It would actually prevent me from using the feature for fear that my collection would someday grow one package beyond the limit. A smooth UX degradation for collections that grow too large can be managed, but falling off a cliff where the system refuses to function cannot.

[scanon (Steve Canon)]

@yim_lee

The actual numbers are debatable, but since SwiftPM will be the primary consumer of package collections and clients of libSwiftPM will be able to access the data, there needs to be limits in place to help ensure smooth UX. The limits will be enforced when SwiftPM imports a package collection.

This doesn't make much sense to me. What about libSwiftPM requires that such limits exist (and why wouldn't we fix that limitation instead of baking a hard limit into the format?)

[yim_lee]

Thanks @dabrahams @scanon. I will leave the size requirements out of the format proposal. I am keeping the limit on the number of versions to include though, because I think that is important to keep the collection data relevant.

[dabrahams (Dave Abrahams)]

@yim_lee

Thanks @dabrahams @scanon. I will leave the size requirements out of the format proposal. I am keeping the limit on the number of versions to include though, because I think that is important to keep the collection data relevant.

Can you say a little more about why that's important? It seems to me that people may have good reasons to select older versions of packages. I assume that would still be possible, somehow; how does keeping older versions out of the collection change the relevance of the collection?

[yim_lee]

@dabrahams

It seems to me that people may have good reasons to select older versions of packages.

Certainly, and nothing with the Package Collections feature will prevent users from using the versions that they want. The intention is that a package collection would contain enough data to introduce users to a package and help them get started with it, and for that listing the latest version should suffice. If users are familiar with the package and think a specific version would work better for them, they should be able to use it even if it's not listed in the collection. In other words, package collection is a discovery mechanism; the proposal doesn't put any restrictions on how dependencies are defined.

Besides, if semantic versioning is followed then minor versions must be backwards compatible, so IMO there is no need for collection to list more than a few minor versions. The only time I would not use the latest minor version is if it contains a bug or if there are underlying changes to the logic that I am not ready to upgrade to yet, but that kind of information is expected to be found elsewhere and not in package collections.

[dabrahams (Dave Abrahams)]

@yim_lee

Besides, if semantic versioning is followed then minor versions must be backwards compatible, so IMO there is no need for collection to list more than a few minor versions. The only time I would not use the latest minor version is if it contains a bug or if there are underlying changes to the logic that I am not ready to upgrade to yet, but that kind of information is expected to be found elsewhere and not in package collections.

All well and good, but why limit the number of major versions?

[yim_lee]

All well and good, but why limit the number of major versions?

It's probably uncommon to be couple major versions behind though?

[dabrahams (Dave Abrahams)]

@yim_lee

It's probably uncommon to be couple major versions behind though?

Sure, it's uncommon. People mostly won't list too many versions. Any hard limits, though, will just limit this tool's flexibility. If you want to discourage listing too many versions, give people a warning, and maybe an interactive prompt to automatically eliminate older major versions when they add a new one.

[yim_lee]

I understand. There will be a tool for generating collections, which by default will choose the latest versions, and on the client side we could ignore the extra versions instead of rejecting the entire collection.

[benrimmington]

You could extend the JSON Feed format. Existing feed readers would be able to display the basics of a package collection, but they'd ignore the "_swift" custom objects.

{
  "version": "https://jsonfeed.org/version/1.1",
  "title": "Sample Package Collection",
  "description": "This is a sample package collection.",
  "authors": [
    {
      "name": "Jane Doe"
    }
  ],
  "_swift": {
    "keywords": ["sample package collection"],
    "formatVersion": "1.0",
    "revision": 3,
    "generatedAt": "2020-10-22T06:03:52Z"
  },
  "items": [
    {
      "id": "https://www.example.com/repos/RepoOne.git",
      "url": "https://www.example.com/repos/RepoOne.git",
      "summary": "Package One",
      "content_text": "...",
      "external_url": "https://www.example.com/repos/RepoOne/README",
      "tags": [
        "keyword"
      ],
      "_swift": {
        "versions": [
          {
            "version": "0.1.0",
            "packageName": "PackageOne"
          }
        ]
      }
    }
  ]
}

[yim_lee]

Thanks for the suggestion @benrimmington. I am hesitant to adopt the JSON Feed format. We changed nomenclature from "package feed" to "package collection" because there are some dissimilarities between it and typical RSS/Atom feed. For one, the concept of updates is different--packages can be added to a collection over time, but other contents such as package versions included, description, etc. could be modified as well, so a package collection should always be looked at as a whole, whereas with a RSS/Atom feed we are more concerned with newly added items and less on edits. Also, the primary consumers of package collections are machines, so feed readers being able to display them is a lower priority.

[yim_lee]

@mattt, in response to your questions:

Version-specific manifest/tag selection was definitely brought up, but we haven't decided what we want to do yet. It's something we will revisit before shipping the feature.

@mattt

What do you think about representing Swift support as a cross-product / matrix of verifiedSwiftVersions and verifiedPlatforms ?

Are you suggesting something like this?

{
  "verifiedSwiftVersions": [
    {
      "name": "5.2",
      "verifiedPlatforms": [{ "name": "macOS" }, { "name": "Linux" }]
    },
    {
      "name": "5.3",
      "verifiedPlatforms": [...]
    }
  ]
}

I think it's reasonable, though I modeled it after how swiftpackageindex.com presents such data (example) and would be curious to see @daveverwer/@finestructure's opinion on this.

@mattt

Should there be a way to communicate compatibility exceptions somehow?

I think this kind of information should be part of the package's README or release notes. In other words, it is the package developers' rather than collection publishers' responsibility to explain why a language version and/or platform is not supported.

@mattt

Could you speak more to the rationale behind the revision value in the JSON payload? Is the intention for that to be used for cache invalidation?

It's mainly a way for collection publishers/users to reference different versions of a collection. We don't intend to interpret it in any meaningful way.

[daveverwer]

@yim_lee

I think it's reasonable, though I modeled it after how swiftpackageindex.com presents such data (example) and would be curious to see @daveverwer/@finestructure's opinion on this.

Certainly, any comprehensive representation of package compatibility needs to be a matrix. There are many real situations where packages would be compatible with a Swift version, but not a platform. Linux is the most common case, where a package includes one of the Apple-specific frameworks like UIKit or AppKit.

The main question of how to represent it comes down to three choices, I think:

• Platform first
• Swift version first
• An array of points inside a matrix

So something like your suggestion Yim:

{
  "verifiedSwiftVersions": [
    {
      "name": "5.2",
      "verifiedPlatforms": [{ "name": "macOS" }, { "name": "Linux" }]
    },
    {
      "name": "5.3",
      "verifiedPlatforms": [...]
    }
  ]
}

Or, platform first:

{
  "verifiedPlatforms": [
    {
      "name": "macOS",
      "verifiedSwiftVersions": [{ "name": "5.2" }, { "name": "5.3" }]
    },
    {
      "name": "iOS"
      "verifiedSwiftVersions": [...]
    }
  ]
}

Or, I wonder if a better way might be as an array of compatibility points:

{
  "compatibility": [
	{
	  "platform": "macOS",
	  "swiftVersion": "5.3.0"
	},
	{
	  "platform": "macOS",
	  "swiftVersion": "5.2.0"
	},
	{
	  "platform": "iOS",
	  "swiftVersion": "5.3.0"
	},
	{
	  "platform": ...,
	  "swiftVersion": ...
	}
  ]
}

I also wonder if we should be storing all Swift version numbers as fully qualified semantic versions? So "5.3.0" rather than "5.3". Everyone refers to Swift 5.3 omitting the patch version, but in a data format such as this, I think it would be better to be explicit.

[daveverwer]

@yim_lee

• versions : An array of version objects representing the most recent and/or relevant releases of the package.

I’m not sure about limiting the number of versions. I’d be happy to see a collection be free to specify any versions without limits, but maybe it’s worth considering some kind of structured version information in addition?

With the Swift Package Index, we track (up to) two significant version numbers for each package:

• Latest stable - The latest valid semantic version that does not have a “pre” part.
• Latest beta - The latest valid semantic version that has a “pre” part. If one exists.

A beta version, especially when it’s a beta of a new major version will quite often have different capabilities and compatibility to the last released stable version, and that’s really relevant information for people consuming packages.

Is it worth including some metadata around “latest stable” and “latest beta” version numbers, but not necessarily making those be limits?

[yim_lee]

@daveverwer

Or, I wonder if a better way might be as an array of compatibility points:

That would work quite nicely I think. So we'd replace verifiedPlatforms and verifiedSwiftVersions with compatibility, which is an array of:

{
  "platform": {
    "name": "macOS"
  }, 
  "swiftVersion": "5.3.0"
}

I suggest we keep platform an object in case we decide to add more details to it in the future.

@daveverwer

I also wonder if we should be storing all Swift version numbers as fully qualified semantic versions?

+1. I will make it explicit.

@daveverwer

I’m not sure about limiting the number of versions.

The number of versions is no longer a hard limit based on previous feedback (i.e., it wouldn't stop SwiftPM from importing a collection). I will reword that section of the doc for they are more like recommendations than requirements.

@daveverwer

but maybe it’s worth considering some kind of structured version information in addition?

Are you suggesting we add latestRelease and latestPreRelease along side versions (with the metadata being kept in versions)?

{  
  "packages": [
    {
      "latestRelease": "0.5.2",
      "latestPreRelease": "1.0.0-alpha.1",
      "versions": [
        {
          "version": "1.0.0-alpha.1",
          ...
        },
        {
          "version": "0.5.2",
          ...
        },
        {
          "version": "0.4.1",
          ...
        }
      ],
      ...      
    }
  ]
}

Or latestRelease and latestPreRelease will contain metadata and be excluded from versions to avoid duplicating data?

{  
  "packages": [
    {
      "latestRelease": {
        "version": "0.5.2",
        ...
      },
      "latestPreRelease": {
        "version": "1.0.0-alpha.1",
        ...
      },
      "versions": [
        {
          "version": "0.4.1",
          ...
        }
      ],
      ...      
    }
  ]
}

The Package Collections CLI commands already have the concept of "latest version" (see sample outputs in the proposal), but it's computed using versions. It might make sense to build that into the collection itself as you are suggesting here.

[NeoNacho (Boris Buegling)]

@daveverwer

I also wonder if we should be storing all Swift version numbers as fully qualified semantic versions? So "5.3.0" rather than "5.3". Everyone refers to Swift 5.3 omitting the patch version, but in a data format such as this, I think it would be better to be explicit.

Do we expect the patch version to matter? As in, would there be the possibility of compatibility only with e.g. "5.3.1"? I think additionally, the question would also be if "5.3.0" means 5.3.0 but not .1 etc or .0 and later versions.