Get a list of system libraries from the dependency graph

Development Package Manager
,
1

Hi Swift Community,

I'm working on a program or script that can analyze a Swift Package Manager (SPM) dependency graph and return all the system libraries declared in .systemLibrary targets across the entire graph.

For example, given a Package.swift like this:

import PackageDescription

let package = Package(
    name: "libNFCTest",
    dependencies: [
        // Dependencies declare other packages that this package depends on.
    ],
    targets: [
        .systemLibrary(
            name: "libnfc", 
            pkgConfig: "libnfc",
            providers: [
                .apt(["libnfc-dev", "libopenssl", "libsqlite-dev"]), 
                .brew(["libnfc", "libopenssl", "libsqlite-dev"])
            ]
        ),
        .target(
            name: "libNFCTest",
            dependencies: ["libnfc"]
        ),
        .testTarget(
            name: "libNFCTestTests",
            dependencies: ["libNFCTest"]
        ),
    ]
)

I want to generate a structure like this:

[
   "libnfc": 
            [
                "apt": ["libnfc-dev", "libopenssl", "libsqlite-dev"],
                "brew": ["libnfc", "libopenssl", "libsqlite-dev"]
            ]
   "libfoo": 
            [
                "apt": ["libbar", "liboo", "libsdl"]
            ]
]

This map should include system libraries from all packages in the dependency graph, not just the current package. Each provider (apt, brew, choco, etc.) would serve as the key, and its corresponding value would be an array of system library or package names.

Goal:

I aim to use this information to create a program that can:

  1. Prompt the user during deployment to install missing system libraries or system packages.
  2. Or simply just informationally listing all the system libraries and their respective providers
  3. Handle the dependency graph holistically, extracting and consolidating all system libraries across all dependencies.

Question:

  1. How can I effectively traverse the SPM dependency graph to collect all .systemLibrary targets across all packages?
  2. Is there a recommended way or existing tooling to parse and extract the providers metadata programmatically?
  3. Any pitfalls or best practices to be aware of when implementing such a program?

Unfortunately, I've tried:

swift package show-dependencies --format json

and looking at

Package.resolved

However, it doesn't list out systemLibrary values.

I'm open to ideas, suggestions, or even pointers to similar implementations! Your insights would be greatly appreciated.

Thanks in advance! :blush:

1 Like
2

Disclaimer: this isn't necessary an ideal solution, but if no other solutions come up it’s at least doable. I’m also not at my laptop right now so some details may be slightly incorrect (haven’t verified).

The swift package dump-package command dumps a package’s package manifest as JSON. With the you should be able to extract immediate system library dependencies along with dependencies on other packages. From there you’d have to locate the checkouts produced by SwiftPM for each of the package dependencies and recurse.

The main downsides I see are that the package dump JSON format is probably not guaranteed to be stable, and you’ll probably also have to deal with a few edge cases when locating checkouts for local package dependencies and registry based dependencies.

3

Thank you for this! But I didn't seem to have much luck. I just tried this:

My Top Level Package looks like so:

import PackageDescription

let package = Package(
    name: "test-csqlite",
    dependencies: [
        .package(url: "https://github.com/sbooth/CSQLite.git", from: "3.47.2"),
        .package(url: "https://github.com/ctreffs/SwiftSDL2.git", from: "1.4.0"),
        .package(url: "https://github.com/Kitura/OpenSSL.git", from: "2.3.1"),
        .package(url: "https://github.com/pureswift/bluetooth.git", from: "7.1.1"),
        .package(url: "https://github.com/migueldeicaza/SkiaKit.git", from: "1.2.5")
    ],  
    targets: [
        .executableTarget(
            name: "test-csqlite",
            dependencies: [
                .product(name: "CSQLite", package: "CSQLite"),
                .product(name: "SDL", package: "SwiftSDL2"),
                .product(name: "OpenSSL", package: "OpenSSL"),
                .product(name: "Bluetooth", package: "Bluetooth"),
                .product(name: "SkiaKit", package: "skiakit")
            ]),
    ]
)

Each child dependency has multiple references to .apt

Running swift package dump-package

Seems to show the following:

{
  "cLanguageStandard" : null,
  "cxxLanguageStandard" : null,
  "dependencies" : [
    {
      "sourceControl" : [
        {
          "identity" : "csqlite",
          "location" : {
            "remote" : [
              {
                "urlString" : "https://github.com/sbooth/CSQLite.git"
              }
            ]
          },
          "productFilter" : null,
          "requirement" : {
            "range" : [
              {
                "lowerBound" : "3.47.2",
                "upperBound" : "4.0.0"
              }
            ]
          }
        }
      ]
    },
    {
      "sourceControl" : [
        {
          "identity" : "swiftsdl2",
          "location" : {
            "remote" : [
              {
                "urlString" : "https://github.com/ctreffs/SwiftSDL2.git"
              }
            ]
          },
          "productFilter" : null,
          "requirement" : {
            "range" : [
              {
                "lowerBound" : "1.4.0",
                "upperBound" : "2.0.0"
              }
            ]
          }
        }
      ]
    },
    {
      "sourceControl" : [
        {
          "identity" : "openssl",
          "location" : {
            "remote" : [
              {
                "urlString" : "https://github.com/Kitura/OpenSSL.git"
              }
            ]
          },
          "productFilter" : null,
          "requirement" : {
            "range" : [
              {
                "lowerBound" : "2.3.1",
                "upperBound" : "3.0.0"
              }
            ]
          }
        }
      ]
    },
    {
      "sourceControl" : [
        {
          "identity" : "bluetooth",
          "location" : {
            "remote" : [
              {
                "urlString" : "https://github.com/pureswift/bluetooth.git"
              }
            ]
          },
          "productFilter" : null,
          "requirement" : {
            "range" : [
              {
                "lowerBound" : "7.1.1",
                "upperBound" : "8.0.0"
              }
            ]
          }
        }
      ]
    },
    {
      "sourceControl" : [
        {
          "identity" : "skiakit",
          "location" : {
            "remote" : [
              {
                "urlString" : "https://github.com/migueldeicaza/SkiaKit.git"
              }
            ]
          },
          "productFilter" : null,
          "requirement" : {
            "range" : [
              {
                "lowerBound" : "1.2.5",
                "upperBound" : "2.0.0"
              }
            ]
          }
        }
      ]
    }
  ],
  "name" : "test-csqlite",
  "packageKind" : {
    "root" : [
      "/Users/max/github/test-csqlite"
    ]
  },
  "pkgConfig" : null,
  "platforms" : [

  ],
  "products" : [

  ],
  "providers" : null,
  "swiftLanguageVersions" : null,
  "targets" : [
    {
      "dependencies" : [
        {
          "product" : [
            "CSQLite",
            "CSQLite",
            null,
            null
          ]
        },
        {
          "product" : [
            "SDL",
            "SwiftSDL2",
            null,
            null
          ]
        },
        {
          "product" : [
            "OpenSSL",
            "OpenSSL",
            null,
            null
          ]
        },
        {
          "product" : [
            "Bluetooth",
            "Bluetooth",
            null,
            null
          ]
        },
        {
          "product" : [
            "SkiaKit",
            "skiakit",
            null,
            null
          ]
        }
      ],
      "exclude" : [

      ],
      "name" : "test-csqlite",
      "packageAccess" : true,
      "resources" : [

      ],
      "settings" : [

      ],
      "type" : "executable"
    }
  ],
  "toolsVersion" : {
    "_version" : "6.0.0"
  }
}
4

Yeah that command only dumps the contents of the package, so you’ll only see immediate system library dependencies that are defined within the package. The second step of my approach, recursing into the packages depended upon by the root package, is required to discover the system library dependencies defined by the child packages.

5

That is what I was thinking.

I assume that just looking under the .build directory for all the resolved package.swift files is all we need to do?

Is there some caveat that I should be aware of?

6

The most reliable way I’ve found to parse the package graph with a good level of fidelity (and, notably, stability) is a hybrid approach:

  1. List the dependencies with swift package show-dependencies --format json
  2. For each dependency (traversing the dep graph using your favourite graph traversal algorithm), dump its package metadata with swift package describe --type json

I think describe is a bit less detailed than dump-package but unlike the latter, the former is guaranteed to have a stable output.

You can see an example of this approach here:

1 Like
7

Very cool,

Using this

// swift-tools-version: 6.0
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
  name: "Depth4",
  products: [
    // Products define the executables and libraries a package produces, making them visible to other packages.
    .library(
      name: "Depth4",
      targets: ["Depth4"])
  ],
  targets: [
    // Targets are the basic building blocks of a package, defining a module or a test suite.
    // Targets can depend on other targets in this package and products from dependencies.
    .systemLibrary(
      name: "opencv",
      pkgConfig: "opencv4",
      providers: [
        .apt(["opencv-dev"]),
        .brew(["opencv"]),
      ]
    ),
    .target(
      name: "Depth4"),
    .testTarget(
      name: "Depth4Tests",
      dependencies: ["Depth4"]
    ),
  ]
)

I wasn't able to see any JSON output from dump-package regarding the .systemLibrary targets... It seems completely omitted.