SwiftPM's current behavior around @testable
/testability can cause confusion and build performance issues.
When building in debug mode, all modules are built with testability. This means programmers can @testable
import other modules from any package, producing code that violates public API contracts. For example, consider the following modules in two distinct packages:
// Package.swift
.target(
name: "MyModule",
dependencies: [.product(name: "OtherModule", package: "OtherPackage")])
.excutableTarget(
name: "MyExecutable",
dependencies: ["MyModule"]),
// OtherModule
let x = 1
// MyModule
@testable import OtherModule
let y = x + 1 // ok in debug mode!
// MyExecutable
@testable import MyModule
print(y) // ok in debug mode!
This code seems like a pretty clear misuse of the testability feature, but compiles and runs "correctly" in debug mode. However in release mode the developer will hit a compilation error.
The benefit of building with testability by default in debug mode (as SwiftPM currently does), is that modules do not need to be rebuilt when running swift test
. For example, the following test target does not require MyModule to be rebuilt when running swift test
after swift build
:
// Package.swift
.testTarget(
name: "MyTest",
dependencies: ["MyModule"]),
// MyTest
@testable import MyModule
#expect(y == 2)
However this "no-rebuild" benefit only applies in debug mode. In release mode, modules built during swift build -c release
do not have testability enabled, meaning when swift test -c release
is run, all modules must be rebuilt.
If we consider a package vending a macro, we can see how the interaction of rebuilding the entire package graph in release mode with testabillity can really hamper developer productivity.
// Package.swift
.target(
name: "MyLibrary",
dependencies: ["MyMacros"]),
.macro(
name: "MyMacros",
dependencies: [
.product(name: "SwiftCompilerPlugin", package: "swift-syntax"),
.product(name: "SwiftDiagnostics", package: "swift-syntax"),
.product(name: "SwiftOperators", package: "swift-syntax"),
.product(name: "SwiftSyntax", package: "swift-syntax"),
.product(name: "SwiftSyntaxBuilder", package: "swift-syntax"),
.product(name: "SwiftSyntaxMacroExpansion", package: "swift-syntax"),
.product(name: "SwiftSyntaxMacros", package: "swift-syntax"),
]),
.testTarget(
name: "MyMacrosTests",
dependencies: [
"MyMacros",
.product(name: "SwiftSyntax", package: "swift-syntax"),
.product(name: "SwiftSyntaxMacros", package: "swift-syntax"),
.product(name: "SwiftSyntaxMacrosTestSupport", package: "swift-syntax"),
]),
This package produces a target called "MyLibrary" which acts as the entry point the macros vended by the "MyMacros" target. Additionally, the package tests their macros' implementations in a test target called "MyMacrosTests".
- The developer first runs
swift build -c release
.
SwiftPM builds "MyLibrary", "MyMacros", and all dependent modules in release mode as expected. This can take a while because release builds are slow to begin with and swift-syntax is a very large project on top of that.
- The developer then runs
swift test -c release
to confirm their macro works.
SwiftPM builds "MyMacrosTests" for the first time and rebuilds entire dependency tree of "MyMacrosTests" with testability.
- The developer runs
swift build -c release
again to fix a bug uncovered by testing.
All dependent modules are rebuilt once again with testability disabled.
This pattern effectively doubles (or triples) the already long build time of swift-syntax.
What's most important, however, is that the developer isn't even trying to test the implementation of swift-syntax! "MyMacrosTests" only @testable
imports "MyMacros" and (IMO) it should only ever be able to @testable
import modules from the same package.
This is a problem I personally face frequently when developing swift-mmio. Swift-mmio is pretty tiny library, but CI can take over 20 minutes to complete (per swift version) due to repeated re-builds of swift-syntax (and other dependencies).
I propose we change SwiftPM to restrict testability to only direct in-package source module dependencies of test targets. With this hcange can we can avoid rebuilding most of the tests' dependencies (e.g. swift-syntax in this example). Additionally, this change would resolve the bug of being able to @testable
import modules in non-test targets.
This behavior change would of course be gated on swift-tool-version: next to avoid breaking an existing packages.
Thoughts?