Release build performance tests in SwiftPM

Development Package Manager
1

Hi,

Currently SwiftPM builds tests in debug configuration with testability
enabled i.e. @testable import Module which gives tests internal level
access to that module. This works ok for normal unit tests but there is no
way to run performance tests in release mode right now.

SwiftPM can enable a release config with testability for building and
running tests but enabling testability will remove some
compiler optimisations and will not be the final release code which will be
shipping and users would almost always want to run perf tests on the final
release code. Not enabling testability in release mode tests will make the
code that uses @testable import fail to compile.

There are two potential solutions which I can think of:
1. let testability be enabled in release mode tests and not care about the
difference.

2. Have another convention especially for performance tests. A PerfTests
directory besides Tests directory which will always test in release mode
with testability off and user is expected not to use the @testable imports
there.

Thoughts?

···

--
Ankit

1 Like
2

FWIW, and this isn't really an answer, but the convention I have personally adopted (in Xcode, not SwiftPM) to deal with this problem is that performance tests always go into their own module, so it can have a different set of build dependencies, flags, workflows, etc.

This is a workable model, but it feels unfortunate to me because it is partly in response to both (a) how testability is currently defined, and (b) how XCTest defines performance tests. I have dreams we will eventually improve (a), which makes me wonder if adding a convention specifically for performance tests is the right direction.

On the other hand, I also believe performance tests end up having very different workflow requirements than regular tests. That may mean they deserve a special case independent of the @testability problem.

I am very strongly opposed to any workflow which only works by building with @testability enabled in release mode. Those workflows don't work well for very performance sensitive code which depend critically on compiler optimization, and which need that behavior to be tracked by performance tests to defend against regressions.

BTW, this is somewhat related to the topic Brian raised recently:
  [swift-build-dev] Performance testing via SwiftPM and XCTest

Also, this is basically part of:
  [SR-1354] [SwiftPM] Improve support for performance tests · Issue #5298 · apple/swift-package-manager · GitHub

I think it would be worth brainstorming what a performance test "convention" would look like, and then see if the proposal has sufficient interest and can justify its own existence versus alternatives.

- Daniel

···

On Aug 1, 2016, at 11:33 AM, Ankit Agarwal via swift-build-dev <swift-build-dev@swift.org> wrote:

Hi,

Currently SwiftPM builds tests in debug configuration with testability enabled i.e. @testable import Module which gives tests internal level access to that module. This works ok for normal unit tests but there is no way to run performance tests in release mode right now.

SwiftPM can enable a release config with testability for building and running tests but enabling testability will remove some compiler optimisations and will not be the final release code which will be shipping and users would almost always want to run perf tests on the final release code. Not enabling testability in release mode tests will make the code that uses @testable import fail to compile.

There are two potential solutions which I can think of:
1. let testability be enabled in release mode tests and not care about the difference.

2. Have another convention especially for performance tests. A PerfTests directory besides Tests directory which will always test in release mode with testability off and user is expected not to use the @testable imports there.

Thoughts?

--
Ankit

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

3

A workable convention could be to detect classes which end with
"*PerformanceTests" and when building, create their own test module that
doesn't have testability enabled. So we'd end up with building two
underlying test modules and running them both under the hood. All that
users would need to do is 1) name some of their test classes with
PerformanceTests at the end and 2) not use @testable import in those test
classes.

Might be too much magic, but I'm just building on top of *Tests naming
conventions that we're already using.

Honza

···

On Mon, Aug 1, 2016 at 9:33 PM Daniel Dunbar via swift-build-dev < swift-build-dev@swift.org> wrote:

FWIW, and this isn't really an answer, but the convention I have
personally adopted (in Xcode, not SwiftPM) to deal with this problem is
that performance tests always go into their own module, so it can have a
different set of build dependencies, flags, workflows, etc.

This is a workable model, but it feels unfortunate to me because it is
partly in response to both (a) how testability is currently defined, and
(b) how XCTest defines performance tests. I have dreams we will eventually
improve (a), which makes me wonder if adding a convention specifically for
performance tests is the right direction.

On the other hand, I also believe performance tests end up having very
different workflow requirements than regular tests. That may mean they
deserve a special case independent of the @testability problem.

I am very strongly opposed to any workflow which only works by building
with @testability enabled in release mode. Those workflows don't work well
for very performance sensitive code which depend critically on compiler
optimization, and which need that behavior to be tracked by performance
tests to defend against regressions.

BTW, this is somewhat related to the topic Brian raised recently:

[swift-build-dev] Performance testing via SwiftPM and XCTest

Also, this is basically part of:
  [SR-1354] [SwiftPM] Improve support for performance tests · Issue #5298 · apple/swift-package-manager · GitHub

I think it would be worth brainstorming what a performance test
"convention" would look like, and then see if the proposal has sufficient
interest and can justify its own existence versus alternatives.

- Daniel

On Aug 1, 2016, at 11:33 AM, Ankit Agarwal via swift-build-dev < > swift-build-dev@swift.org> wrote:

Hi,

Currently SwiftPM builds tests in debug configuration with testability
enabled i.e. @testable import Module which gives tests internal level
access to that module. This works ok for normal unit tests but there is no
way to run performance tests in release mode right now.

SwiftPM can enable a release config with testability for building and
running tests but enabling testability will remove some
compiler optimisations and will not be the final release code which will be
shipping and users would almost always want to run perf tests on the final
release code. Not enabling testability in release mode tests will make the
code that uses @testable import fail to compile.

There are two potential solutions which I can think of:
1. let testability be enabled in release mode tests and not care about the
difference.

2. Have another convention especially for performance tests. A PerfTests
directory besides Tests directory which will always test in release mode
with testability off and user is expected not to use the @testable imports
there.

Thoughts?

--
Ankit

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev