When you add a dependency to a package you need to declare that dependency in two places. The first is the package dependency array - that can only specify the whole package. The second is the dependency array for each target that depends on the package you're pulling in. In that part you need to declare the product that target depends on if a package vends more that one target. E.g. vapor/Package.swift at main · vapor/vapor · GitHub
I can't remember the current state of SwiftPM but IIRC it's clever enough to only download the transitive dependencies it needs and won't build products that aren't needed.