RFC: `@package` imports, dependency scanning, and explicit modules

Development Package Manager
1

Hello! I wanted to check in and see if there's been any implementation efforts or design discussion around the concept of "@package imports" recently. I'm asking because I've run some experiments leveraging the awesome new fast dependency scanner which seem promising, and I think explicit module builds could be a good path forward for this feature if there aren't already plans in place. Feel free to ignore all of the following if someone is already working on this though!

What I had in mind was:

  1. The swift compiler frontend defines syntax for an @package attribute on imports. The contents of the attribute need to parse as a valid argument list, but otherwise no semantic validation occurs in the compiler to avoid baking in too much knowledge of the package description API.

  2. The fast dependency scanner is extended to collect the module name and @package attribute contents and output it alongside the other module dependencies. e.g. @package(url: "https://github.com/jpsim/Yams.git", .branch("releases/1.0")) import Foo is reported as something like the following by the fast dependency scanner.

"packageDependencies": [
            {
              "module": "Foo",
              "packageDependencyDescription": "(url: \"https://github.com/jpsim/Yams.git\", .branch(\"releases/1.0\"))",
              "targetInfo": ...
            }
]

  1. swift-driver is extended to invoke SPM to build these package imports in a known location instead of managing the build directly. The driver passes on the dependency scanner output. This creates kind of a weird circular dependency between SPM and the driver, but it should be manageable.

  2. SPM consumes the dependency scanner output and fetches & builds dependencies. I don't have a clear idea of how this should work yet, but it seems feasible and keeps SPM responsible for most of the details of the package API, etc. This is the point at which the contents of the attribute are semantically validated, possibly by pasting the argument list into SPM's Package.Dependency API or something similar.

  3. swift-driver includes the built package dependencies as explicit modules using the build paths it provided SPM.

I've put together an early proof-of-concept for points 1 & 2 at [Experiment/DNM] @package imports found via fast dependency scanning by owenv · Pull Request #32366 · apple/swift · GitHub which seems viable, but I'd be interested in hearing from SPM experts as to whether this is a good direction and, if so, how it might be implemented it in SPM. If there's interest, I'll look into putting together a production-ready version of the compiler PR to allow prototyping the driver and SPM components.

As always, feedback/comments/questions are welcome! This is just a tiny experiment inspired by some of the other explicit modules work that's been going on, so it might very well end up not being the greatest idea :slight_smile:

1 Like
2

I've been exploring this idea a bit more recently now that explicit module builds are more mature, following a slightly different approach to what I proposed above:

  1. In the prototype at [Experiment/DNM] @package imports found via fast dependency scanning by owenv · Pull Request #32366 · apple/swift · GitHub, if a module is imported by a statement with the @_package attribute, the dependency scanner will not attempt to load it. Instead, it'll be reported to the driver as a "Swift package product" module dependency, with the product description being the attribute arguments.

  2. swift-driver will resolve package dependencies similar to how it resolves placeholder dependencies in the explicit module build graph. Before resolving placeholder dependencies, swift-driver will invoke SPM with the collected package product descriptions, and SPM will resolve & build package deps if needed and return a JSON-encoded list of the intermodule dependency graphs for the built package products and all of their transitive dependencies. The driver could then merge these into its own module dependency graph as precompiled modules.

There are still a few open questions with this approach:

  • What interface should SPM provide to the driver to accept a list of package products, resolve and build them, and return the dependency graph of compiled modules to the driver? I've been thinking this should be a command line interface of some sort that minimizes the driver's dependency on SPM, but I'm open to other ideas. I'm also not sure if the driver should attempt to construct some kind of fake manifest from the contents of @_package attributes or if SPM should parse them internally and define the API.
  • Could this approach to package imports ever be extended to work in the REPL? (Immediate/script mode might not use the full functionality of explicit module builds, but it should still be able to use the dependency scanner to discover package dependencies.)
  • Should build systems have a way of flagging package imports as unsupported when invoking the driver? For example, this might be needed to ban their use in SPM packages which will need to continue to declare all of their dependencies in the manifest, and to handle platforms where SPM isn't supported at all.

Overall I think this feature is probably viable so long as the SPM integration doesn't run into any major issues. Long term maintenance of a dedicated SPM mode to support it might be annoying though, I'm not sure if anyone has thoughts on that or ideas about alternate approaches.

1 Like