Pitch: [SwiftPM] @testable build setting

Evolution Pitches
, ,
1

Hello, Swift Evolution!

I'd like us to consider the following addition to the Swift Package Manager APIs.

Introduction

The current Swift Package Manager build system is currently hardcoded to pass the -enable-testing flag to the Swift compiler to enable @testable import when building in debug mode.

Swift-evolution thread: Pitch: [SwiftPM] @testable build setting

Motivation

Not all targets in a given package make use of the @testable import feature (or wish to use it at all), but all targets are presently forced to build their code with this support enabled regardless of whether it's needed.

Developers should be able to disable @testable import when it's not needed or desired, just as they're able to do so in Xcode's build system.

On Windows in particular, where a shared library is limited to 65k exported symbols, disabling @testable import provides developers an option to significantly reduce the exported symbol count of a library by hiding all of the unnecessary internal APIs. It can also improve debug build performance as fewer symbols exported from a binary can result in faster linking.

Proposed solution

Add a new Swift target setting API to specify whether testing should be enabled for the specified target, falling back to the current behavior by default.

Detailed design

Add a new enableTesting API to SwiftSetting limited to manifests >= 6.1:

public struct SwiftSetting {
  // ... other settings
  
  @available(_PackageDescription, introduced: 6.1)
  public static func enableTesting(
      _ enable: Bool,
      _ condition: BuildSettingCondition? = nil
  ) -> SwiftSetting {
     ...
  }
}

The existing --enable-testable-imports / --disable-testable-imports command line flag to swift-test currently defaults to --enable-testable-imports. It will be changed to default to "unspecified" (respecting any target settings), and explicitly passing --enable-testable-imports or --disable-testable-imports will force all targets to enable or disable testing, respectively.

Security

New language version setting has no implications on security, safety or privacy.

Impact on existing packages

Since this is a new API, all existing packages will use the default behavior - testing will be enabled when building for the debug configuration, and disabled when building for the release configuration.

Alternatives considered

None.

3 Likes
2

I prepared the swift-evolution PR and a draft implementation.

1 Like
3

Looks like a good addition without any drawbacks in my opinion. +1

Don’t have much else to say as I don’t plan to use this feature but the use cases outlined in your pitch sound solid.

4

FWIW I believe the long-term direction for engineers will be to move away from testable.

5

+1. @testable is a hack and we really shouldn't do anything to give it more "first-class" support. The opposite really; we should be driving users far away from it with better options that let them test the thing they're actually releasing, not test something built in a different mode that busts a hole in access control.

I personally hope to see @testable deprecated and eventually removed. Fortunately, there are a number of better approaches that are available now or likely on the horizon:

  • package-level access is likely a suitable replacement for what most people use @testable for today.
  • swift-testing uses runtime discovery that can find tests in any binary image, including the library under test; they don't have to exist in separate targets. That could be used to let tests for internal/private declarations live right next to that code. There's work to be done there to make that realistic, though (it needs to be possible to strip such tests completely from release builds of those libraries).
4 Likes
6

+1 from the peanut gallery (having stumbled over this before for other reasons)

You likely have a strong immediate need to support large modules on windows. The community has a longer-term ambitions to migrate away from @testable, but without immediate solutions.

One strategy for both would be a setting now that could be augmented for later implementations. The current proposal of surfacing a boolean flag might complicate later settings, as would preserving the name/concept of "testable".

The name/concept could be some variant of "test-access".

The setting could be an enum, with e.g., values (some future):

  • global (current default)
  • external (no special access/disabled (future default))
  • module
  • package
  • specified type(s)

Another question is where/when to apply configuration (command-line, package-level, any targets, only test targets, source annotations/macros...), considering existing practice and future needs.

I tend to prefer the most local (indeed, it would be ideal if the presence of @testable were enough, and restricted to the target (subject to @availability?)), but the need to handle platform variations on Windows and Linux suggest that flags and/or broader scope are warranted.

It would be nice to track best practice here. Commonly in Package.swift people set up one [SwiftSetting] variable and use it in multiple contexts, so supporting that is probably the default.

7

Just to be clear, is this commentary on @testable in general meant to be in favor of, or opposed to the pitch at hand? I don't see a conflict between giving people the ability to turn testability off and having a bigger picture vision of de-emphasizing and maybe eventually deprecating @testable import. On the contrary, it actually seems like a step in that direction, since it gives package owners a way to assert explicitly that they do not want to use testability.

I'm in favor of the pitch, as it seems like a no-brainer to me for SwiftPM to have parity with respect to control over this build behavior. I fear this thread could easily be dominated by litigation of @testable in general, so I'd ask us to consider using a dedicated thread for that discussion because I think it's a pretty deep and potentially polarizing topic. I think that we should be able to decouple these discussions because whatever the future of @testable is, it's likely that it needs to be supported for a while either way, and in the interim having more control over its impact makes sense.

4 Likes
8

That's a fair point. I'm looking at it through the lens of "If there's any doubt, I would like for us to decide what the future of this thing is before we invest further in it/make it appear to be even more supported". But looking at it through the lens of "this thing already exists in Swift today and the proposed capability provides additional control to users who need it", I can't find much fault with it.

3 Likes
9

As a user of @testable that actually doesn't hate it, I dont agree with placing this setting on the target itself.

I think testability is a function of how a target is used, not the target itself. By default all targets should not have testability enabled at all, but instead it should be opt-in as part of the dependency of a test target.

something like:

.target(name: "Foo"),
.testTarget(
  name: "FooTests",
  dependencies: [.target(name: "Foo", testable: true)]),

Obviously I'm biased, but I would like to see more of the idea pitched here incorporated into this change: [Pitch] Restrict testability to direct in-package source module dependencies of test targets

1 Like
10

This may be worth bringing to the attention of the Testing Steering Working Group once we've got it set up.

1 Like
11

That approach is interesting, but I think it would add significantly more complexity in needing to potentially create multiple variants of a target with testability enabled or disabled, and figuring out the right dependencies between different variations.

12

This is something that is rather well timed for that complexity though. I think that @dschaefer2 is going to have to deal with a similar problem for the DLL storage handling. The shared or static nature of the build is determined by the consumer, not the producer. The only way to avoid that is to build both variants. This would be a similar situation - we either build two variants of each target or we back propagate the build type information in the graph.

1 Like