Strategies for organizing/navigating “very” large SwiftPM packages?

i have a (few) SPM project(s) that have a “very”[1] large number of targets (50+), and the lack of any hierarchical organization capabilities in SPM is making it increasingly difficult to manage and navigate these projects.

my problem is essentially such:

SPM has no support for sub-Sources directories, or sub-manifests, so all targets are pooled together in one monster manifest and one monster Sources directory. and this presents problems because logically-related targets do not necessarily appear near one another in alphabetical order.

i’ve experimented with a couple workarounds and found them all unsatisfactory:

  1. breaking up Sources itself into many directories, and using custom target paths to map the custom hierarchical structure back to SPM’s flat manifest model. but this doesn’t make the manifest itself any easier to edit, and in the process of normal development (renaming modules, moving modules, splitting and fusing modules, refactoring dependencies, etc.) i always felt like i was going “against the grain” of SPM’s design and i was trying to retrofit a directory layout on a build system that really wants to use a flat layout.

  2. adopting a latin-style naming scheme where adjectives come after nouns so related targets appear near each other in alphabetical order. but english is not latin and this quickly got really awkward because instead of - for example - BrickBuilding, ConcreteBuilding, WoodenBuilding, MetalBuilding, etc., you now have BuildingBrick, BuildingConcrete, BuildingWooden, BuildingMetal, etc. which are just nonsensical choices of names in isolation.

any better ideas for organizing a large SPM project?


[1] i say “very” because coming from the C/C++ world, fifty targets is nothing!

2 Likes

Would it make sense to spit your package into smaller packages? I think the Swift extension for VSCode allows editing different packages nested under a workspace directory.

i am not sure if this has changed, but as far as i remember, SPM will only recognize Package.swift manifests at the top-level of the source-control repository.

You can set a target's source path with path: "my/path/here"

1 Like

I have a few mono-repospackages and I agree with both of your approaches, combining custom paths and prefix-oriented naming (for discoverability in directories and in content-assist).

What makes it manageable for me in Package.swift is to consolidate naming, location, and resource logic in code that works from a specification for each node, and then consolidate project-specific conventions in some local builder functions. This makes the package declaration not only more writable/readable, but also changeable: I've altered directory conventions without changing the package declaration itself, which makes experimenting easier. The PackageSpec ends up embodying whatever conventions are commonly adopted.

Stepwise, when building a package, I declare and use a struct PackageSpec for each dependency node, to consolidate logic (and minimally to avoid duplicating names in strings).

Then in a package builder factory/method, for per-package behavior I implement local factory methods closing over common nodes, and finally declare the package.

Thus, overall Package.swift top-level structure and factory method organization looks like...

let package = P.factory()

fileprivate enum P {
  static func factory() -> Package {
    // 1. Nodes as PackageSpec instances...

    // 2. Local factory functions wrapping common/core nodes...

    // 3. Build Package using Package API's, PackageSpec, and local functions
  }
}

/// Manages conventions, common declarations...
fileprivate struct PackageSpec {
  typealias PackageDep = PackageDescription.Package.Dependency
  typealias TargetDep = Target.Dependency
  static let ours = "github-username"
  let user: String
  let package: String
  let product: String
  let version: Version
  let urlAccess: UrlAccess // ssh or https
  let current: Bool // being declared in this package (vs external)

  // PackageSpec factories, utilities common to all projects
  func asLibrary() .. {}
  func asPackDepWith() _ others: [PackageSpec]) -> [PackageDep] {..}
  func asTargDepWith(_ others: [PackageSpec]) -> [TargetDep] {
  func testName() -> String {...}
  ...
}

The Package factory function below shows the three phases:

  1. Common nodes
  2. Local functions closing over the common nodes
  3. The package declaration
static func makePackage() -> Package {

// 1. Current package and its products
let bird = PackageSpec("Bird", .init(0,0,1), current: true)
let birdNest = bird.product("BirdNest")
let demo = bird.product("Demo")
let localTree = bird.product("LocalTree")

// External dependencies: 
let log = PackageSpec(
  package: "swift-log",
  "Logging",
   by: "apple",
   Version(1, 5, 2))

// 2. project-specific logic closing over common nodes

// Declare target that only depends on Bird (in Sources/\(spec.product)/)
func birdLibTarget(_ spec: PackageSpec) -> Target {
  PackageDescription.Target.target(
    name: spec.product,
    dependencies: [bird.asTargetDep()]
  )
}

// Declare test target that depends on its peer module and the "fix" Test library
func fixTestTarget(_ spec: PackageSpec, _ otherDeps: [PackageSpec] = []) -> Target {...}

/// 3. Then building Package is concise if lengthy:

let package = Package(
  name: bird.package,
  platforms: [...],
  products: [
    bird.asLibrary(),
    birdNest.asLibrary(),
    places.asExecutable(),
    ..
    ],
  dependencies: log.asPackDepWith([atomics, markdown, docc]),
  targets: [
    // MARK: library targets
    .target(name: places.product), // direct API
    birdLibTarget(birdBits),  // method-local API

    // MARK: test targets
    fixTestTarget(badBird),
   ...

  return package
}

I re-use PackageSpec by copying across projects; common conventions are implemented with common PackageSpec, and those projects kinda look like each other. I use one in particular enough that I have a script complain when they differ (for lack of a true common dependency mechanism).

1 Like

To be clear I’m referring to this part of the extension’s announcement, but I haven’t tried it so it’s possible I misunderstood.

You can set a target's source path with path: "my/path/here"

One most definitely can, yet the bloat in manifest file only grows. I have like 20-something products defined and my manifest is already 600 lines long :/

No, this has not changed. What's worse is if you want to share your code as swift package you have to use the top-level directory for manifest location because swiftpm can not use https://github.com/something/foo/bar/baz as a valid package url