Are SE-0292 scoped package identifiers still relevant?

Evolution Discussion
1

a long time ago in a galaxy far away, SE-0292 defined a scoped format for package identifiers, where a package id would be allowed to contain up to one dot.

the intent was likely for people to use the scope as a namespace, so you might have apple.package-a, apple.package-b, tayloraswift.package-a, etc.

this of course, wasn’t what happened. many people seem to have started using the dots artistically, and now we have packages named progress.swift, grdb.swift, etc. SPM now interprets progress as the scope and swift as the local package name, which completely negates the usefulness of the scoped identifier format.

are SE-0292 scoped package identifiers still relevant today?

2 Likes
2

They're definitely still relevant.

It feels like a bug that we're treating URL-based packages as if they were registry identifiers? I think only if package(id:from:) and such are being used we should make that assumption.

cc @tomerd

3

when i tried to build docs for @hassila ’s package-benchmark, i found a pin named

    {
      "identity" : "progress.swift",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/ordo-one/Progress.swift",
      "state" : {
        "revision" : "29dc5dc29d8408f42878b832c7aae38a35ff26ee",
        "version" : "1.0.3"
      }
    },

i’m not sure if the identifiers in Package.resolved are full identifiers or just local identifiers. but based on the relevant section of SE-0292, it suggests that we should be parsing progress as the scope and swift as the local identifier.

1 Like
4

I think it should be conditional on kind if that happens? Are you seeing a concrete issue with this?

5

yes, i am trying to combine docs for packages on the swiftinit index, and i was considering using the owner’s GitHub username as the scope within the package database. this implies SPM must enforce the rules for allowed package name characters defined in SE-0292:

A package's name uniquely identifies a package in a scope. A package name consists of alphanumeric characters, underscores, and hyphens. Hyphens and underscores may not occur at the beginning or end, nor consecutively within a name. The maximum length of a package name is 100 characters. A valid package scope matches the following regular expression pattern:

\A[a-zA-Z0-9](?:[a-zA-Z0-9]|[-_](?=[a-zA-Z0-9])){0,99}\z

(i am assuming here that package scope here is a typo, since there is an analogous grammar in the paragraph above it.)

the error i got was that the documentation compiler encountered a remoteSourceControl package that already has a scope, and that is unexpected. if we automatically strip the superfluous scope, we get a package named swift, which is not a very useful package name.

1 Like
6

Ah I see.

I don't think the rules for a package identifiers map to the rules for GH usernames and repositories (which I think is the problem you're facing).

Also note that package identifiers aren't globally unique, they're unique per registry. So if swiftinit is a global database of sorts, using package identifiers won't necessarily work since they can be duplicated across registries.

1 Like
7

SPM always uses the last URL path component as the package name for assigning pin identifiers. what seems to have happened is that because GitHub permits dots in repository names, and the repository name is the last URL path component in a GitHub repo dependency, that SPM (transitively) allows dots to appear in package names.

this means there exists a significant population of swift packages that have no legal name today under SE-0292, which is why i asked if SE-0292 is still relevant today, since these packages obviously aren’t going anywhere and we have to support them.

one possible, very long term, direction for the site is for it to become a package registry.

1 Like
8

Again, SE-0292 was never meant to add restrictions onto URLs of git-based packages or local dependencies, it is only supposed to govern the scheme of identifiers as used by .package(id:from:) and similar APIs. This may or may not be 100% reflected in the text of the proposal, but that was the intent.

9

is it accurate to say then that the id field in Package.resolved is not meaningful for remoteSourceControl dependencies?

what is the recommended way to obtain an SE-0292-compliant package name from SPM?

10

No, the ID should be usable. It just doesn't follow the rules of SE-0292 and you shouldn't think of it as "globally unique" in any way.

We do have this thing called CanonicalPackageLocation which could maybe help to identify packages uniquely on a broader scale.

1 Like
11

it looks like CanonicalPackageLocation encodes the full domain/path, is this what SPM uses internally to identify packages?

12

It depends, for the most part, we're just using the identity to disambiguate inside a particular package graph. We have CanonicalPackageLocation and CanonicalPackageURL as identifiers for situation where we need something more "global", e.g. for the shared cache of git checkouts.

13

i suppose that could work in the short term, but i could envision some issues in the long term with URL identities as packages migrate between organizations and people change their GitHub usernames.

14

Right, the way registries are supposed to handle this is that they would know a number of aliases for one logical identity.

So e.g. apple.swift-nio would also map to https://github.com/apple/swift-nio and any number of alternative URLs and identities.

This does hinge on there being some sort of administrative side of things where users can claim scopes and relevant URLs with some kind of human validation to prevent abuse.

1 Like
15

i was afraid it was going to involve something like this…

my goal for swiftinit is to index package documentation in a way that is durable over time. this requires packages to have stable identities, so the dependency graphs don’t rot as people rename things. but implementing a package registry is very far out of the current scope of the documentation index.

anyway, thanks for the clarifications. i’m going to need to come up with a different way to persist dependency graphs that doesn’t rely on concepts of package identity.

16

I wonder if the GH API exposes the various old URLs for a repository? I know that they keep redirecting across renames so the information must be there at least.

17

the challenge is more so with the internal consistency of the package database, because i am using the package ID as SPM defines it as a primary key in the database.

swiftinit doesn’t build docs directly from source, it stores binary symbol graphs on the server-side. building docs for a package involves compiling a binary symbol graph and then linking it against the symbol graphs for its dependencies. the dependencies are loaded by looking up packages by name, as that is how the dependency tree is currently encoded within the symbol graph metadata.

1 Like
18

Literally my first day trying to develop with swift, so still trying to figure out up from down and learning about the community, but felt like I might be able to positively contribute to the thread :sweat_smile:

I believe GH will redirect until someone else claims the previous username/reponame or blocks the creation of the old_username/reponame combination under certain conditions[1]

This was recently (2023) implemented by GH to mitigate "repojacking" vulnerabilities described in one this years DEFCON presentations[2]

Coming devops/platform engineering background and needing survive dealing with the sharp edges of python[3]/golang/ruby/terraform/cloudformation packages/registries and respective methods of dependency management, attempting to learn how Swift Packages/Namespacing worked has been one of my first steps into the swift ecosystem and how I stumbled across this thread.

Looking forward to future improvements in this area :slight_smile:

  1. https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-user-account-settings/changing-your-github-username ↩︎

  2. Youtube link

    DEF CON 31 - The GitHub Actions Worm - Asi Greenholts


    Watch the video     ↩︎

  3. fun with python package/filename normalization and registries :upside_down_face:
    Change in PyPI upload behavior. Intentional, accidental, pebkac? - Packaging - Discussions on Python.org ↩︎

2 Likes