[Pitch] Restrict testability to direct in-package source module dependencies of test targets

Evolution Pitches
1

SwiftPM's current behavior around @testable/testability can cause confusion and build performance issues.

When building in debug mode, all modules are built with testability. This means programmers can @testable import other modules from any package, producing code that violates public API contracts. For example, consider the following modules in two distinct packages:

// Package.swift
.target(
  name: "MyModule",
  dependencies: [.product(name: "OtherModule", package: "OtherPackage")])
.excutableTarget(
  name: "MyExecutable",
  dependencies: ["MyModule"]),

// OtherModule
let x = 1

// MyModule
@testable import OtherModule
let y = x + 1 // ok in debug mode!

// MyExecutable
@testable import MyModule
print(y) // ok in debug mode!

This code seems like a pretty clear misuse of the testability feature, but compiles and runs "correctly" in debug mode. However in release mode the developer will hit a compilation error.

The benefit of building with testability by default in debug mode (as SwiftPM currently does), is that modules do not need to be rebuilt when running swift test. For example, the following test target does not require MyModule to be rebuilt when running swift test after swift build:

// Package.swift
.testTarget(
  name: "MyTest",
  dependencies: ["MyModule"]),

// MyTest
@testable import MyModule
#expect(y == 2)

However this "no-rebuild" benefit only applies in debug mode. In release mode, modules built during swift build -c release do not have testability enabled, meaning when swift test -c release is run, all modules must be rebuilt.

If we consider a package vending a macro, we can see how the interaction of rebuilding the entire package graph in release mode with testabillity can really hamper developer productivity.

// Package.swift
.target(
  name: "MyLibrary",
  dependencies: ["MyMacros"]),

.macro(
  name: "MyMacros",
  dependencies: [
    .product(name: "SwiftCompilerPlugin", package: "swift-syntax"),
    .product(name: "SwiftDiagnostics", package: "swift-syntax"),
    .product(name: "SwiftOperators", package: "swift-syntax"),
    .product(name: "SwiftSyntax", package: "swift-syntax"),
    .product(name: "SwiftSyntaxBuilder", package: "swift-syntax"),
    .product(name: "SwiftSyntaxMacroExpansion", package: "swift-syntax"),
    .product(name: "SwiftSyntaxMacros", package: "swift-syntax"),
  ]),

.testTarget(
  name: "MyMacrosTests",
  dependencies: [
    "MyMacros",
    .product(name: "SwiftSyntax", package: "swift-syntax"),
    .product(name: "SwiftSyntaxMacros", package: "swift-syntax"),
    .product(name: "SwiftSyntaxMacrosTestSupport", package: "swift-syntax"),
  ]),

This package produces a target called "MyLibrary" which acts as the entry point the macros vended by the "MyMacros" target. Additionally, the package tests their macros' implementations in a test target called "MyMacrosTests".

  1. The developer first runs swift build -c release.

SwiftPM builds "MyLibrary", "MyMacros", and all dependent modules in release mode as expected. This can take a while because release builds are slow to begin with and swift-syntax is a very large project on top of that.

  1. The developer then runs swift test -c release to confirm their macro works.

SwiftPM builds "MyMacrosTests" for the first time and rebuilds entire dependency tree of "MyMacrosTests" with testability.

  1. The developer runs swift build -c release again to fix a bug uncovered by testing.

All dependent modules are rebuilt once again with testability disabled.

This pattern effectively doubles (or triples) the already long build time of swift-syntax.

What's most important, however, is that the developer isn't even trying to test the implementation of swift-syntax! "MyMacrosTests" only @testable imports "MyMacros" and (IMO) it should only ever be able to @testable import modules from the same package.

This is a problem I personally face frequently when developing swift-mmio. Swift-mmio is pretty tiny library, but CI can take over 20 minutes to complete (per swift version) due to repeated re-builds of swift-syntax (and other dependencies).

I propose we change SwiftPM to restrict testability to only direct in-package source module dependencies of test targets. With this hcange can we can avoid rebuilding most of the tests' dependencies (e.g. swift-syntax in this example). Additionally, this change would resolve the bug of being able to @testable import modules in non-test targets.

This behavior change would of course be gated on swift-tool-version: next to avoid breaking an existing packages.

Thoughts?

11 Likes
2

I think this is a reasonable proposal that fits into what I'd expect the semantics would be. Making this work would require some extensive changes to underlying infrastructure in order to split modules into two representations (with and without testing enabled).

1 Like
3

I have abandoned swift-format and swift-lint as package plugins and opted instead for manually bundling pre-built binaries into the repo for exactly the reason of slow build times. In a greenfield project, simply depending on swift-lint caused the build time to jump from a couple of seconds to several minutes, and it spent most of the time rebuilding swift-syntax. I tried to figure out what made it sometimes reuse previously built artifacts of third party packages, but it seemed random to me, and it didn't happen very often.

That's interesting that it might have something to do with testability. However to me those don't seem essentially related. I agree that not even tests should be able to access internals of third party modules (when a colleague of mine who mostly works on other stacks like .NET worked with me on tests and I showed him @testable, he commented, "oh, so you can friend yourself!" LOL).

But even if you can, or under the hood all packages are built as testable (perhaps the restriction would get applied higher up while compiling tests), I still don't know why SPM ever rebuilds externally consumed packages. They don't even pick up changes in the branch being pulled down until a manual update. What is the purpose of ever building a specific snapshot of a package more than once (per build flavor)?

Beyond the example I gave above, I've seen projects that pull in lots of external packages suffer long compile times for the same reason that external packages are constantly rebuilt. Even after cleaning this should not be happening. The build system should compile those packages once the first time a build depending on them is executed and never again unless you go in and clean out the SPM cache. Manually setting up a system to depend on pre-built binaries is accomplishing what SPM should be doing automatically.

I consider that a critical problem SPM needs to fix, even if fixing it doesn't patch the hole of being able to access third party internals in tests.

2 Likes