# \[Pitch\] Restrict testability to direct in-package source module dependencies of test targets

**URL:** <https://forums.swift.org/t/pitch-restrict-testability-to-direct-in-package-source-module-dependencies-of-test-targets/73921>\
**Category:** Pitches\
**Tags:** packagemanager\
**Created:** [August 12, 2024, 9:13pm UTC](https://forums.swift.org/t/pitch-restrict-testability-to-direct-in-package-source-module-dependencies-of-test-targets/73921 "2024-08-12T21:13:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rauhul](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/rauhul/32/35207_2.png) [@rauhul](https://forums.swift.org/u/rauhul)\
**Post date:** [August 12, 2024, 9:13pm UTC](https://forums.swift.org/t/pitch-restrict-testability-to-direct-in-package-source-module-dependencies-of-test-targets/73921/1 "2024-08-12T21:13:38Z")

</div>

SwiftPM's current behavior around `@testable`/testability can cause confusion and build performance issues.

* * *

When building in debug mode, _all_ modules are built with testability. This means programmers can `@testable` import other modules from any package, producing code that violates public API contracts. For example, consider the following modules in two distinct packages:

```swift
// Package.swift
.target(
  name: "MyModule",
  dependencies: [.product(name: "OtherModule", package: "OtherPackage")])
.excutableTarget(
  name: "MyExecutable",
  dependencies: ["MyModule"]),

// OtherModule
let x = 1

// MyModule
@testable import OtherModule
let y = x + 1 // ok in debug mode!

// MyExecutable
@testable import MyModule
print(y) // ok in debug mode!

```

This code seems like a pretty clear misuse of the testability feature, but compiles and runs "correctly" in debug mode. However in release mode the developer will hit a compilation error.

* * *

The benefit of building with testability by default in debug mode (as SwiftPM currently does), is that modules do not need to be rebuilt when running `swift test`. For example, the following test target does _not_ require MyModule to be rebuilt when running `swift test` after `swift build`:

```swift
// Package.swift
.testTarget(
  name: "MyTest",
  dependencies: ["MyModule"]),

// MyTest
@testable import MyModule
#expect(y == 2)

```

However this "no-rebuild" benefit only applies in debug mode. In release mode, modules built during `swift build -c release` do not have testability enabled, meaning when `swift test -c release` is run, all modules must be rebuilt.

* * *

If we consider a package vending a macro, we can see how the interaction of rebuilding the entire package graph in release mode with testabillity can really hamper developer productivity.

```swift
// Package.swift
.target(
  name: "MyLibrary",
  dependencies: ["MyMacros"]),

.macro(
  name: "MyMacros",
  dependencies: [
    .product(name: "SwiftCompilerPlugin", package: "swift-syntax"),
    .product(name: "SwiftDiagnostics", package: "swift-syntax"),
    .product(name: "SwiftOperators", package: "swift-syntax"),
    .product(name: "SwiftSyntax", package: "swift-syntax"),
    .product(name: "SwiftSyntaxBuilder", package: "swift-syntax"),
    .product(name: "SwiftSyntaxMacroExpansion", package: "swift-syntax"),
    .product(name: "SwiftSyntaxMacros", package: "swift-syntax"),
  ]),

.testTarget(
  name: "MyMacrosTests",
  dependencies: [
    "MyMacros",
    .product(name: "SwiftSyntax", package: "swift-syntax"),
    .product(name: "SwiftSyntaxMacros", package: "swift-syntax"),
    .product(name: "SwiftSyntaxMacrosTestSupport", package: "swift-syntax"),
  ]),

```

This package produces a target called "MyLibrary" which acts as the entry point the macros vended by the "MyMacros" target. Additionally, the package tests their macros' implementations in a test target called "MyMacrosTests".

1. The developer first runs `swift build -c release`.

SwiftPM builds "MyLibrary", "MyMacros", and all dependent modules in release mode as expected. This can take a while because release builds are slow to begin with and swift-syntax is a very large project on top of that.

1. The developer then runs `swift test -c release` to confirm their macro works.

SwiftPM builds "MyMacrosTests" for the first time and rebuilds _entire_ dependency tree of "MyMacrosTests" with testability.

1. The developer runs `swift build -c release` again to fix a bug uncovered by testing.

All dependent modules are rebuilt once _again_ with testability disabled.

This pattern effectively doubles (or triples) the already long build time of swift-syntax.

What's most important, however, is that the developer isn't even trying to test the implementation of swift-syntax! "MyMacrosTests" only `@testable` imports "MyMacros" and (IMO) it should only ever be able to `@testable` import modules from the same package.

This is a problem I personally face frequently when developing swift-mmio. Swift-mmio is pretty tiny library, but CI can take over 20 minutes to complete (per swift version) due to repeated re-builds of swift-syntax (and other dependencies).

* * *

I propose we change SwiftPM to restrict testability to only _direct_ in-package source module dependencies of test targets. With this hcange can we can avoid rebuilding most of the tests' dependencies (e.g. swift-syntax in this example). Additionally, this change would resolve the bug of being able to `@testable` import modules in non-test targets.

This behavior change would of course be gated on swift-tool-version: next to avoid breaking an existing packages.

Thoughts?

---

<div class="post-metadata">

**Author:** ![xedin](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/xedin/32/309_2.png) [@xedin](https://forums.swift.org/u/xedin)\
**Post date:** [August 12, 2024, 9:29pm UTC](https://forums.swift.org/t/pitch-restrict-testability-to-direct-in-package-source-module-dependencies-of-test-targets/73921/2 "2024-08-12T21:29:54Z")

</div>

I think this is a reasonable proposal that fits into what I'd expect the semantics would be. Making this work would require some extensive changes to underlying infrastructure in order to split modules into two representations (with and without testing enabled).

---

<div class="post-metadata">

**Author:** ![aetherealtech](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/aetherealtech/32/36879_2.png) [@aetherealtech](https://forums.swift.org/u/aetherealtech)\
**Post date:** [August 13, 2024, 10:47am UTC](https://forums.swift.org/t/pitch-restrict-testability-to-direct-in-package-source-module-dependencies-of-test-targets/73921/3 "2024-08-13T10:47:49Z")

</div>

I have abandoned `swift-format` and `swift-lint` as package plugins and opted instead for manually bundling pre-built binaries into the repo for exactly the reason of slow build times. In a greenfield project, simply depending on `swift-lint` caused the build time to jump from a couple of seconds to several minutes, and it spent most of the time rebuilding `swift-syntax`. I tried to figure out what made it sometimes reuse previously built artifacts of third party packages, but it seemed random to me, and it didn't happen very often.

That's interesting that it might have something to do with testability. However to me those don't seem _essentially_ related. I agree that not even tests should be able to access internals of third party modules (when a colleague of mine who mostly works on other stacks like .NET worked with me on tests and I showed him `@testable`, he commented, "oh, so you can friend yourself!" LOL).

But even if you can, or under the hood all packages are built as testable (perhaps the restriction would get applied higher up while compiling tests), I still don't know why SPM _ever_ rebuilds externally consumed packages. They don't even pick up changes in the branch being pulled down until a manual update. What is the purpose of ever building a specific snapshot of a package more than _once_ (per build flavor)?

Beyond the example I gave above, I've seen projects that pull in lots of external packages suffer long compile times for the same reason that external packages are constantly rebuilt. Even after cleaning this should not be happening. The build system should compile those packages once the first time a build depending on them is executed and never again unless you go in and clean out the SPM cache. Manually setting up a system to depend on pre-built binaries is accomplishing what SPM should be doing automatically.

I consider that a critical problem SPM needs to fix, even if fixing it doesn't patch the hole of being able to access third party internals in tests.