Draft proposal: "Test Executables"


(George King) #1

Hello, here is a rough draft proposal, inspired by a conversation with Daniel Dunbar last week on the Slack channel. I realize it is light on details, but hopefully this will be a starting point for figuring out how to better support building dedicated test executable targets. Let me know what you think, and if there is anything else I can do to move this forward. Thanks, -George

Test Executables

Proposal: SE-NNNN <applewebdata://380819B4-D77C-4970-B9BD-49646BA9A819/NNNN-filename.md>
Author: George King <https://github.com/gwk>
Status: Awaiting review
Review manager: TBD
Introduction

The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables directly in the package manager would make command line testing of Swift packages easier and more robust.

Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.

XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing,and GUI testing, but not for testing the basic IO behavior of a process. Since the swift package manager supports building command-line executables, it should support testing them as well.

Consider the existing features of swiftpm as two orthogonal axes:

Build, Test
Library, Executable
Of the four combinations, only "Test Executable" is missing.

One preliminary step towards process-based testing would be to add conventions for building executables that are meant strictly for testing, rather than for public usage. These would be exercised by some external testing harness; adding such capabilities to XCTest is beyond the scope of this proposal.

Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:

Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
For simple projects that produce a single library module, adding a test executable no longer necessitates moving the library code from Sources/ to Sources/[Module] to allow for the second target.
Test executables would only be built by swift test, not swift build.
Proposed solution

swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.

The test command should, upon encountering a test executable directory, build the executable.

If the test executable build fails, test should report a test failure and return a nonzero exit code.

If the build succeeds, test should report a test success and clearly output the path to the resulting test executable, for consumption by external test harness scripts.

Future improvements could include invoking a test command specified in Package.swift upon successful build.

Detailed design

I am unclear on the exact semantics of @testable import (I see no difference when I remove the@testable modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.

swift test should output a list of successfully built executable test paths, for consumption by an external test harness. This should be a text file listing one path per line, so that simple shell scripts can run the tests without having to parse formats like json or yaml. The list should contain only executables that built successfully (or where deemed unchanged by dependency calculation), so that nonexistant and/or stale tests do not get executed. If a use case emerges that requires a list of all tests, then perhaps a complete listing could be produced separately.

Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.

Alternatives considered

No action

See the list of problems with treating test executables as regular targets in the "Motivation" section above.

Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.


(Daniel Dunbar) #2

Hi George!

Generally speaking, I like the direction for SwiftPM of making unit tests
consistent with the other kinds of targets, and so the "is a test" becomes
just a flag for each target rather than a somewhat different kind of target.

Your proposal calls out a couple things that there aren't concrete details
on: why exactly do we need the output file with a list of test executables?
Where and when would it be written, and how would other targets find it? I
suspect this is something we should consider independently, and I would
recommend exploring whether it is something we should expose as an API
targets could use instead of a build artifact.

The big part of this proposal is a change to the conventions for test
targets, and I'm not sure exactly what convention you are suggesting --
that `main.swift` inside a module within Tests just become an executable
target? I am tempted towards a slight different direction:

- Rename test targets to `<NAME>Tests`, which matches the Xcode convention
and solves an existing problem where the module name of the test module
isn't apparent in the file system (despite being required to be named in
the source code sometimes).
- Accept all target forms under Tests/, so `Tests/foo/main.swift` would
declare a new executable `foo`, just like it would in Sources. The only
special behavior of these targets would be that they wouldn't automatically
be built by downstream package dependencies.

Are there other reasons why we need a separate "test executable" concept
other than just putting another executable into `Sources/`, if SwiftPM
itself isn't going to tie custom behaviors to them? Currently I can think
of two reasons: organizational clarity and the inability to specify
dependencies of test modules.

Bear in mind that the perspective I am coming from is that even if you want
to do integration tests of test executables, the "driver" of those
executables would still be an XCTest module adapting them, at least until
we had third-party testing support.

- Daniel

···

On Thursday, July 7, 2016, George King via swift-build-dev < swift-build-dev@swift.org> wrote:

Hello, here is a rough draft proposal, inspired by a conversation with
Daniel Dunbar last week on the Slack channel. I realize it is light on
details, but hopefully this will be a starting point for figuring out how
to better support building dedicated test executable targets. Let me know
what you think, and if there is anything else I can do to move this
forward. Thanks, -GeorgeTest Executables

   - Proposal: SE-NNNN <http://NNNN-filename.md>
   - Author: George King <https://github.com/gwk>
   - Status: *Awaiting review*
   - Review manager: TBD

Introduction

The package manager currently supports testing via XCTest, but does not
provide support for other testing methodologies. In particular, facilities
for building test executables directly in the package manager would make
command line testing of Swift packages easier and more robust.
Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is,
broadly speaking, the practice of testing multiple software components such
as functions and classes (which may be "unit tested" individually) in
aggregate. Integration tests can reveal bugs arising from interactions
between components, can validate the intended usage patterns for individual
components, and serve as references for that intended usage.

XCTest offers facilities
<https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for
unit testing, performance testing,and GUI testing, but not for testing the
basic IO behavior of a process. Since the swift package manager supports
building command-line executables, it should support testing them as well.

Consider the existing features of swiftpm as two orthogonal axes:

   - Build, Test
   - Library, Executable

Of the four combinations, only "Test Executable" is missing.

One preliminary step towards process-based testing would be to add
conventions for building executables that are meant strictly for testing,
rather than for public usage. These would be exercised by some external
testing harness; adding such capabilities to XCTest is beyond the scope of
this proposal.

Currently, test executables can simply be placed in the Sources, where
they are compiled as regular executable targets. Distinguishing them as
test executables would be helpful for several reasons:

   - Putting all test code in the Tests directory will prevent test
   executables from being exposed to library consumers, and clarifies
   developer intent.
   - For simple projects that produce a single library module, adding a
   test executable no longer necessitates moving the library code from
   Sources/ to Sources/[Module] to allow for the second target.
   - Test executables would only be built by swift test, not swift build.

Proposed solution

swift test should distinguish between unit test directories and
executable directories, just as swift build distinguishes between libary
and executable directories.

The test command should, upon encountering a test executable directory,
build the executable.

If the test executable build fails, test should report a test failure and
return a nonzero exit code.

If the build succeeds, test should report a test success and clearly
output the path to the resulting test executable, for consumption by
external test harness scripts.

Future improvements could include invoking a test command specified in
Package.swift upon successful build.
Detailed design

I am unclear on the exact semantics of @testable import (I see no
difference when I remove the@testable modifier from a test case), but I
assume it has to do with linking a unit test against the library under
test. Superficially, it makes sense that a test executable would have the
same requirements and use the same syntax.

swift test should output a list of successfully built executable test
paths, for consumption by an external test harness. This should be a text
file listing one path per line, so that simple shell scripts can run the
tests without having to parse formats like json or yaml. The list should
contain only executables that built successfully (or where deemed unchanged
by dependency calculation), so that nonexistant and/or stale tests do not
get executed. If a use case emerges that requires a list of all tests, then
perhaps a complete listing could be produced separately.
Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that
do not contain XCTest-based unit tests.
Alternatives consideredNo action

See the list of problems with treating test executables as regular targets
in the "Motivation" section above.
Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a
solution would necessitate the essentials of this proposal anyway.


(George King) #3

Hi list,
Here is an updated draft of my "Test Executables" proposal. This draft drops the less well defined ideas, and tightens up some of the motivation. Let me know what you think, and what more needs to be done.
Thanks,
George

Test Executables

Proposal: SE-NNNN <applewebdata://CA2DA431-9CC7-418E-8DC8-FA9AB46534B8/NNNN-filename.md>
Author: George King <https://github.com/gwk>
Status: Awaiting review / DRAFT
Review manager: TBD
Introduction

The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables (that is, executables meant strictly for testing and not as an end product) in the Swift Package Manager would make command line testing of Swift packages easier and more robust. We propose to add to the package manager the ability to specify and build test-only executable targets that are distinct from regular executables.

Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.

XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing, and GUI testing, but not for testing the basic IO behavior of a process. To be clear, this proposal does not address XCTest at all; while adding whole process, IO oriented APIs to XCTest sounds desirable, it is a larger project and out of scope for this proposal.

Instead, this proposal simply asks for the ability to build executables specifically for testing, rather than for public usage. The intended use case is execution via an external test harness, or from within an XCTest case via NSTask/Process.

Consider the existing features of swiftpm as two orthogonal axes:

Build, Test
Library, Executable
Of the four combinations, only "Test Executable" is missing.

Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:

Test executables would only be built by swift test, not swift build.
Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
For simple projects that produce a single library module, adding a test executable will no longer necessitate moving the library code from Sources/ to Sources/[Module] to allow for the second target.
Proposed solution

swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.

The swift test command should, upon encountering a test executable directory, build the executable.

If the test executable build fails, swift test should report a test failure and return a nonzero exit code. An external test harness can then run the test executables as it sees fit. Note that for an XCTest case to invoke the test executable, test executables would need to be built prior to running test cases.

Future improvements could include invoking an external test command specified in Package.swift upon successful build.

Detailed design

I am unclear on the exact semantics of @testable import (I see no difference when I remove the @testable modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.

Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.

Alternatives considered

No action

See the list of problems with treating test executables as regular targets in the "Motivation" section above.

Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.

···

On Jul 7, 2016, at 7:07 PM, George King <george.w.king@gmail.com> wrote:

Hello, here is a rough draft proposal, inspired by a conversation with Daniel Dunbar last week on the Slack channel. I realize it is light on details, but hopefully this will be a starting point for figuring out how to better support building dedicated test executable targets. Let me know what you think, and if there is anything else I can do to move this forward. Thanks, -George

Test Executables

Proposal: SE-NNNN <x-msg://24/NNNN-filename.md>
Author: George King <https://github.com/gwk>
Status: Awaiting review
Review manager: TBD
Introduction

The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables directly in the package manager would make command line testing of Swift packages easier and more robust.

Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.

XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing,and GUI testing, but not for testing the basic IO behavior of a process. Since the swift package manager supports building command-line executables, it should support testing them as well.

Consider the existing features of swiftpm as two orthogonal axes:

Build, Test
Library, Executable
Of the four combinations, only "Test Executable" is missing.

One preliminary step towards process-based testing would be to add conventions for building executables that are meant strictly for testing, rather than for public usage. These would be exercised by some external testing harness; adding such capabilities to XCTest is beyond the scope of this proposal.

Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:

Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
For simple projects that produce a single library module, adding a test executable no longer necessitates moving the library code from Sources/ to Sources/[Module] to allow for the second target.
Test executables would only be built by swift test, not swift build.
Proposed solution

swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.

The test command should, upon encountering a test executable directory, build the executable.

If the test executable build fails, test should report a test failure and return a nonzero exit code.

If the build succeeds, test should report a test success and clearly output the path to the resulting test executable, for consumption by external test harness scripts.

Future improvements could include invoking a test command specified in Package.swift upon successful build.

Detailed design

I am unclear on the exact semantics of @testable import (I see no difference when I remove the@testable modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.

swift test should output a list of successfully built executable test paths, for consumption by an external test harness. This should be a text file listing one path per line, so that simple shell scripts can run the tests without having to parse formats like json or yaml. The list should contain only executables that built successfully (or where deemed unchanged by dependency calculation), so that nonexistant and/or stale tests do not get executed. If a use case emerges that requires a list of all tests, then perhaps a complete listing could be produced separately.

Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.

Alternatives considered

No action

See the list of problems with treating test executables as regular targets in the "Motivation" section above.

Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.


(Daniel Dunbar) #4

This change makes sense to me, and the proposal looks good.

This change makes sense on its own, but it should also mitigate the current pain point where we don't have test-only dependencies, so people can't easily have "example" main executables which don't get build by downstream clients.

A couple minor comments inline...

Hello, here is a rough draft proposal, inspired by a conversation with Daniel Dunbar last week on the Slack channel. I realize it is light on details, but hopefully this will be a starting point for figuring out how to better support building dedicated test executable targets. Let me know what you think, and if there is anything else I can do to move this forward. Thanks, -George

Test Executables

Proposal: SE-NNNN <x-msg://152/NNNN-filename.md>
Author: George King <https://github.com/gwk>
Status: Awaiting review
Review manager: TBD
Introduction

The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables directly in the package manager would make command line testing of Swift packages easier and more robust.

Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.

XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing,and GUI testing, but not for testing the basic IO behavior of a process. Since the swift package manager supports building command-line executables, it should support testing them as well.

Consider the existing features of swiftpm as two orthogonal axes:

Build, Test
Library, Executable
Of the four combinations, only "Test Executable" is missing.

One preliminary step towards process-based testing would be to add conventions for building executables that are meant strictly for testing, rather than for public usage. These would be exercised by some external testing harness; adding such capabilities to XCTest is beyond the scope of this proposal.

Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:

Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
For simple projects that produce a single library module, adding a test executable no longer necessitates moving the library code from Sources/ to Sources/[Module] to allow for the second target.
Test executables would only be built by swift test, not swift build.
Proposed solution

swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.

The test command should, upon encountering a test executable directory, build the executable.

If the test executable build fails, test should report a test failure and return a nonzero exit code.

If the build succeeds, test should report a test success and clearly output the path to the resulting test executable, for consumption by external test harness scripts.

Future improvements could include invoking a test command specified in Package.swift upon successful build.

Detailed design

I am unclear on the exact semantics of @testable import (I see no difference when I remove the@testable modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.

@testable just gives you access to code which is `internal`, *if* the module was built with testability enabled. I don't think this proposal needs any change to manage it.

swift test should output a list of successfully built executable test paths, for consumption by an external test harness. This should be a text file listing one path per line, so that simple shell scripts can run the tests without having to parse formats like json or yaml. The list should contain only executables that built successfully (or where deemed unchanged by dependency calculation), so that nonexistant and/or stale tests do not get executed. If a use case emerges that requires a list of all tests, then perhaps a complete listing could be produced separately.

I don't think we should prescribe `swift test` output for these executables. Is what you want a mechanism so the tests know where to find them? If so, I would probably something else where SwiftPM just writes a compact representation of what it built into the build directory -- those kinds of small tool behaviors (versus package convention / manifest) don't need to be at the level of a Swift evolution proposal though.

As for doing it in a way that tests won't need to rerun if it has changed, I think that is out of scope for this proposal. There are too many other complicated factors (like what if the executable picks up some data file which *has* changed) for it to seem like something to role into what is an otherwise straightforward proposal.

- Daniel

···

On Jul 7, 2016, at 4:07 PM, George King via swift-build-dev <swift-build-dev@swift.org> wrote:
Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.

Alternatives considered

No action

See the list of problems with treating test executables as regular targets in the "Motivation" section above.

Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(George King) #5

Daniel, thanks for the feedback.

First, a process question: in general, do package manager proposals end up in the swift-evolution repo, or is that only for the language proper? If so, then I'll put it in a github fork of swift-evolution so that we can track changes. If not, then what is the final destination?

More comments inline...

Hi George!

Generally speaking, I like the direction for SwiftPM of making unit tests consistent with the other kinds of targets, and so the "is a test" becomes just a flag for each target rather than a somewhat different kind of target.

Your proposal calls out a couple things that there aren't concrete details on: why exactly do we need the output file with a list of test executables? Where and when would it be written, and how would other targets find it? I suspect this is something we should consider independently, and I would recommend exploring whether it is something we should expose as an API targets could use instead of a build artifact.

My basic assumption in all of this is that XCTest is not currently equipped to do process IO testing, and that as a first step, executable tests would be run by a third party test harness. The nature of the harness is left unspecified, apart from being a command line tool.

The output list is, I admit, not so well thought out. I was thinking that it was a simple way for swiftpm to communicate to a harness script the executables that had been produced.

Take this example harness.sh (totally made up):

set -e
swift test # if the XCTest unit tests fail, or any of the test executables do not build, this exits nonzero and the harness aborts.
built_test_executables_path='.build/built_test_executables.txt' # the output list - this name would be a swiftpm convention.
for test in $(cat "$built_test_executables_path"); do
  echo "running executable test: $test"
  $test # if the test fails, harness aborts.
  echo
done

A real testing framework would specify test expectations for each of these executables, so the list might not be so useful as I had initially thought.

The real problem to address is that swiftpm must communicate the availability of up-to-date test executables to the third party harness. If it doesn't, then when a build fails, the harness is liable to test a stale executable. Perhaps the simple solution is for swiftpm to delete the previous product when a build starts, so that consumers cannot possibly run tests on stale products.

My other concern is that we don't want harnesses trying to parse the output messages of swiftpm, and the output list concept was an attempt to mitigate that. At the moment, I think that as long as stale products are removed and their locations are sufficiently stable and obvious, a 3rd party harness should not need any further information from swiftpm because it ought to have test expectations written down anyway.

The big part of this proposal is a change to the conventions for test targets, and I'm not sure exactly what convention you are suggesting -- that `main.swift` inside a module within Tests just become an executable target?

Yes. I'm proposing that it parallel the existing convention for package targets; the presence of main.swift distinguishes a library from an executable in `Sources`, and it could do the exact same thing in `Tests`.

I am tempted towards a slight different direction:

- Rename test targets to `<NAME>Tests`, which matches the Xcode convention and solves an existing problem where the module name of the test module isn't apparent in the file system (despite being required to be named in the source code sometimes).

I'm not sure what you mean; at first I interpreted this as suggesting that for package `Foo` there would be a top level directory called `FooTests` instead of `Tests`. But from your example below, now I don't think so. Do you mean that a `Tests/Foo` compiles to `FooTests`?

Regardless, I think we should strive to enhance the parallels between how `Sources` works, and how `Tests` works.
* Swiftpm looks for sources in just two canonical places: `Sources` or `src`. This is a nice, easy convention. Looking for sources in `Tests` similar.
* Swiftpm can either interprets the contents of Sources as "single target" (sources contains source files), or "multiple target" (sources contains subdirectories of source files). `Tests` should be the same.
* In the "single target" interpretation of the `Tests` directory contents, swiftpm could infer a name for the test module, just like it does for a single target in sources. This could be `<PACKAGE-NAME>Tests`.
* In the "multiple target" interpretation of `Tests`, the test modules are named according to the subdirectories (I think this is how it works now). Maybe you just mean appending the "Tests" suffix to these, which seems fine.

- Accept all target forms under Tests/, so `Tests/foo/main.swift` would declare a new executable `foo`, just like it would in Sources. The only special behavior of these targets would be that they wouldn't automatically be built by downstream package dependencies.

Yes, this is what I had in mind. I assume that downstream packages don't build the unit tests either; again, I advocate for parallel behavior.

Are there other reasons why we need a separate "test executable" concept other than just putting another executable into `Sources/`, if SwiftPM itself isn't going to tie custom behaviors to them? Currently I can think of two reasons: organizational clarity and the inability to specify dependencies of test modules.

I agree about organizational clarity.

I do not know what you mean about inability to specify dependencies of test modules. Is this because you cannot write them down in Package.swift, like you would for a regular target? Is there any reason why we cannot just support the same sorts of specifications of dependencies (and anything else) for tests as we do for regular targets?

More important than not building the executables downstream, I don't want to expose the test executables as public, whatever that means. As far as I can tell, swiftpm does not yet do anything to make package executables available to the end user, but it might in the future. For example, Pypi can generate "entry points" that end up in the python bin directory. If swiftpm started making packages executables available to the user on the PATH or any way, keeping tests out of that process would be important.

Put another way, it does not seem far fetched that swiftpm could gain an `swift package install` command or similar, and it would be unfortunate if published packages had in the meantime been publishing test executables as public targets.

Bear in mind that the perspective I am coming from is that even if you want to do integration tests of test executables, the "driver" of those executables would still be an XCTest module adapting them, at least until we had third-party testing support.

This is the basic source of confusion I think; you mentioned in our initial slack chat that 3rd party testing frameworks was a concern, so I leapt to the conclusion that XCTest would not be the driver.

My perspective is that I already have my own harness written (github.com/gwk/iotest <http://github.com/gwk/iotest> - it's for personal use and depends on a fast-changing utility library so it might appear broken on a given day). I use it to test executable targets that I'm producing via swiftpm now.

I'm curious to hear a bit more about what you consider to be "3rd party testing support", because apart from this proposal, I can't think of anything else that iotest needs from the package manager!

Finally, I should add a few motivating examples for why test executables would be useful. The obvious case is for libraries that offer IO functionality; imagine a command line argument parser library, or an error logging framework. I have found that it is easiest to test these kinds of things with a harness that can specify environment, process arguments, stdin, and current working directory structure, and then capture exit status, stdout and stderr from a test process. iotest does this, and has a few nice tricks like printing diffs of the expected and actual output. Offering these sorts of features through XCTest would be a much larger proposal, so I think it is worth trying to carve out the test executable part as a first step.

···

On Jul 8, 2016, at 6:43 PM, Daniel Dunbar <daniel@zuster.org> wrote:

- Daniel

On Thursday, July 7, 2016, George King via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:
Hello, here is a rough draft proposal, inspired by a conversation with Daniel Dunbar last week on the Slack channel. I realize it is light on details, but hopefully this will be a starting point for figuring out how to better support building dedicated test executable targets. Let me know what you think, and if there is anything else I can do to move this forward. Thanks, -George

Test Executables

Proposal: SE-NNNN <http://nnnn-filename.md/>
Author: George King <https://github.com/gwk>
Status: Awaiting review
Review manager: TBD
Introduction

The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables directly in the package manager would make command line testing of Swift packages easier and more robust.

Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.

XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing,and GUI testing, but not for testing the basic IO behavior of a process. Since the swift package manager supports building command-line executables, it should support testing them as well.

Consider the existing features of swiftpm as two orthogonal axes:

Build, Test
Library, Executable
Of the four combinations, only "Test Executable" is missing.

One preliminary step towards process-based testing would be to add conventions for building executables that are meant strictly for testing, rather than for public usage. These would be exercised by some external testing harness; adding such capabilities to XCTest is beyond the scope of this proposal.

Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:

Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
For simple projects that produce a single library module, adding a test executable no longer necessitates moving the library code from Sources/ to Sources/[Module] to allow for the second target.
Test executables would only be built by swift test, not swift build.
Proposed solution

swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.

The test command should, upon encountering a test executable directory, build the executable.

If the test executable build fails, test should report a test failure and return a nonzero exit code.

If the build succeeds, test should report a test success and clearly output the path to the resulting test executable, for consumption by external test harness scripts.

Future improvements could include invoking a test command specified in Package.swift upon successful build.

Detailed design

I am unclear on the exact semantics of @testable import (I see no difference when I remove the@testable modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.

swift test should output a list of successfully built executable test paths, for consumption by an external test harness. This should be a text file listing one path per line, so that simple shell scripts can run the tests without having to parse formats like json or yaml. The list should contain only executables that built successfully (or where deemed unchanged by dependency calculation), so that nonexistant and/or stale tests do not get executed. If a use case emerges that requires a list of all tests, then perhaps a complete listing could be produced separately.

Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.

Alternatives considered

No action

See the list of problems with treating test executables as regular targets in the "Motivation" section above.

Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.


(Ankit Agarwal) #6

This proposal sounds similar to allowing regular modules (lib+exec) inside
Tests/ and IMO that is better than just allowing executables and is more
general. Right now we error out when some directory doesn't have Tests
prefix in Tests/ folder, We can just allow that and use same rule of
main.swift to distinguish between lib and exec.

George, would doing that solve your problem?

···

On Sat, Nov 12, 2016 at 6:39 AM, Daniel Dunbar via swift-build-dev < swift-build-dev@swift.org> wrote:

This change makes sense to me, and the proposal looks good.

This change makes sense on its own, but it should also mitigate the
current pain point where we don't have test-only dependencies, so people
can't easily have "example" main executables which don't get build by
downstream clients.

A couple minor comments inline...

On Jul 7, 2016, at 4:07 PM, George King via swift-build-dev < > swift-build-dev@swift.org> wrote:

Hello, here is a rough draft proposal, inspired by a conversation with
Daniel Dunbar last week on the Slack channel. I realize it is light on
details, but hopefully this will be a starting point for figuring out how
to better support building dedicated test executable targets. Let me know
what you think, and if there is anything else I can do to move this
forward. Thanks, -GeorgeTest Executables

   - Proposal: SE-NNNN
   - Author: George King <https://github.com/gwk>
   - Status: *Awaiting review*
   - Review manager: TBD

Introduction

The package manager currently supports testing via XCTest, but does not
provide support for other testing methodologies. In particular, facilities
for building test executables directly in the package manager would make
command line testing of Swift packages easier and more robust.
Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is,
broadly speaking, the practice of testing multiple software components such
as functions and classes (which may be "unit tested" individually) in
aggregate. Integration tests can reveal bugs arising from interactions
between components, can validate the intended usage patterns for individual
components, and serve as references for that intended usage.

XCTest offers facilities
<https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for
unit testing, performance testing,and GUI testing, but not for testing the
basic IO behavior of a process. Since the swift package manager supports
building command-line executables, it should support testing them as well.

Consider the existing features of swiftpm as two orthogonal axes:

   - Build, Test
   - Library, Executable

Of the four combinations, only "Test Executable" is missing.

One preliminary step towards process-based testing would be to add
conventions for building executables that are meant strictly for testing,
rather than for public usage. These would be exercised by some external
testing harness; adding such capabilities to XCTest is beyond the scope of
this proposal.

Currently, test executables can simply be placed in the Sources, where
they are compiled as regular executable targets. Distinguishing them as
test executables would be helpful for several reasons:

   - Putting all test code in the Tests directory will prevent test
   executables from being exposed to library consumers, and clarifies
   developer intent.
   - For simple projects that produce a single library module, adding a
   test executable no longer necessitates moving the library code from
   Sources/ to Sources/[Module] to allow for the second target.
   - Test executables would only be built by swift test, not swift build.

Proposed solution

swift test should distinguish between unit test directories and
executable directories, just as swift build distinguishes between libary
and executable directories.

The test command should, upon encountering a test executable directory,
build the executable.

If the test executable build fails, test should report a test failure and
return a nonzero exit code.

If the build succeeds, test should report a test success and clearly
output the path to the resulting test executable, for consumption by
external test harness scripts.

Future improvements could include invoking a test command specified in
Package.swift upon successful build.
Detailed design

I am unclear on the exact semantics of @testable import (I see no
difference when I remove the@testable modifier from a test case), but I
assume it has to do with linking a unit test against the library under
test. Superficially, it makes sense that a test executable would have the
same requirements and use the same syntax.

@testable just gives you access to code which is `internal`, *if* the
module was built with testability enabled. I don't think this proposal
needs any change to manage it.

swift test should output a list of successfully built executable test
paths, for consumption by an external test harness. This should be a text
file listing one path per line, so that simple shell scripts can run the
tests without having to parse formats like json or yaml. The list should
contain only executables that built successfully (or where deemed unchanged
by dependency calculation), so that nonexistant and/or stale tests do not
get executed. If a use case emerges that requires a list of all tests, then
perhaps a complete listing could be produced separately.

I don't think we should prescribe `swift test` output for these
executables. Is what you want a mechanism so the tests know where to find
them? If so, I would probably something else where SwiftPM just writes a
compact representation of what it built into the build directory -- those
kinds of small tool behaviors (versus package convention / manifest) don't
need to be at the level of a Swift evolution proposal though.

As for doing it in a way that tests won't need to rerun if it has changed,
I think that is out of scope for this proposal. There are too many other
complicated factors (like what if the executable picks up some data file
which *has* changed) for it to seem like something to role into what is an
otherwise straightforward proposal.

- Daniel

Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that
do not contain XCTest-based unit tests.
Alternatives consideredNo action

See the list of problems with treating test executables as regular targets
in the "Motivation" section above.
Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a
solution would necessitate the essentials of this proposal anyway.
_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

--
Ankit


(Daniel Dunbar) #7

Hi George,

Sorry for the delayed reply, I lost track of this while on vacation.

Daniel, thanks for the feedback.

First, a process question: in general, do package manager proposals end up in the swift-evolution repo, or is that only for the language proper? If so, then I'll put it in a github fork of swift-evolution so that we can track changes. If not, then what is the final destination?

They do end up in the swift-evolution repo, usually once the proposal is ready for official review and accept/reject stamp then a PR is created for the swift-evolution repo. Once review starts it is assigned a number and a feedback period.

More comments inline...

Hi George!

Generally speaking, I like the direction for SwiftPM of making unit tests consistent with the other kinds of targets, and so the "is a test" becomes just a flag for each target rather than a somewhat different kind of target.

Your proposal calls out a couple things that there aren't concrete details on: why exactly do we need the output file with a list of test executables? Where and when would it be written, and how would other targets find it? I suspect this is something we should consider independently, and I would recommend exploring whether it is something we should expose as an API targets could use instead of a build artifact.

My basic assumption in all of this is that XCTest is not currently equipped to do process IO testing, and that as a first step, executable tests would be run by a third party test harness.

It is possible to use XCTest to drive subprocess for IO testing, and you can even use dynamic test reporting to expose the results as distinct tests (although I am not sure if this can be done on Linux). So it should be possible, it just might not be convenient (e.g., if your "XCTest" ends up being call this subprocess and expect it to report 0).

In the interest of ensuring a focused & tractable proposal I would recommend decoupling this proposal itself from the 3rd-party test harness problem, which also deserves a proposal. That way each is easy to design and discuss.

The nature of the harness is left unspecified, apart from being a command line tool.

The output list is, I admit, not so well thought out. I was thinking that it was a simple way for swiftpm to communicate to a harness script the executables that had been produced.

I propose just dropping this requirement to start with... individual projects will already know the names of the executables they expected to be produced. If we do want this to be automated, then it probably makes the most sense to be a general purpose feature for SwiftPM to report all of its artifacts, which could be used for a wide variety of purposes.

Take this example harness.sh (totally made up):

set -e
swift test # if the XCTest unit tests fail, or any of the test executables do not build, this exits nonzero and the harness aborts.
built_test_executables_path='.build/built_test_executables.txt' # the output list - this name would be a swiftpm convention.
for test in $(cat "$built_test_executables_path"); do
  echo "running executable test: $test"
  $test # if the test fails, harness aborts.
  echo
done

A real testing framework would specify test expectations for each of these executables, so the list might not be so useful as I had initially thought.

The real problem to address is that swiftpm must communicate the availability of up-to-date test executables to the third party harness. If it doesn't, then when a build fails, the harness is liable to test a stale executable. Perhaps the simple solution is for swiftpm to delete the previous product when a build starts, so that consumers cannot possibly run tests on stale products.

My other concern is that we don't want harnesses trying to parse the output messages of swiftpm, and the output list concept was an attempt to mitigate that. At the moment, I think that as long as stale products are removed and their locations are sufficiently stable and obvious, a 3rd party harness should not need any further information from swiftpm because it ought to have test expectations written down anyway.

We don't currently ensure stale products are always removed, but I think we should treat that as a bug not something we need a feature for.

The big part of this proposal is a change to the conventions for test targets, and I'm not sure exactly what convention you are suggesting -- that `main.swift` inside a module within Tests just become an executable target?

Yes. I'm proposing that it parallel the existing convention for package targets; the presence of main.swift distinguishes a library from an executable in `Sources`, and it could do the exact same thing in `Tests`.

I am tempted towards a slight different direction:

- Rename test targets to `<NAME>Tests`, which matches the Xcode convention and solves an existing problem where the module name of the test module isn't apparent in the file system (despite being required to be named in the source code sometimes).

I'm not sure what you mean; at first I interpreted this as suggesting that for package `Foo` there would be a top level directory called `FooTests` instead of `Tests`. But from your example below, now I don't think so. Do you mean that a `Tests/Foo` compiles to `FooTests`?

This ended up being codified in Anders' proposal here:
  https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160718/000555.html

Regardless, I think we should strive to enhance the parallels between how `Sources` works, and how `Tests` works.

I agree.

* Swiftpm looks for sources in just two canonical places: `Sources` or `src`. This is a nice, easy convention. Looking for sources in `Tests` similar.
* Swiftpm can either interprets the contents of Sources as "single target" (sources contains source files), or "multiple target" (sources contains subdirectories of source files). `Tests` should be the same.

This seems reasonable to me.

* In the "single target" interpretation of the `Tests` directory contents, swiftpm could infer a name for the test module, just like it does for a single target in sources. This could be `<PACKAGE-NAME>Tests`.

Agreed.

* In the "multiple target" interpretation of `Tests`, the test modules are named according to the subdirectories (I think this is how it works now). Maybe you just mean appending the "Tests" suffix to these, which seems fine.

I meant making Tests explicit in the file organization.

- Accept all target forms under Tests/, so `Tests/foo/main.swift` would declare a new executable `foo`, just like it would in Sources. The only special behavior of these targets would be that they wouldn't automatically be built by downstream package dependencies.

Yes, this is what I had in mind. I assume that downstream packages don't build the unit tests either; again, I advocate for parallel behavior.

Are there other reasons why we need a separate "test executable" concept other than just putting another executable into `Sources/`, if SwiftPM itself isn't going to tie custom behaviors to them? Currently I can think of two reasons: organizational clarity and the inability to specify dependencies of test modules.

I agree about organizational clarity.

I do not know what you mean about inability to specify dependencies of test modules. Is this because you cannot write them down in Package.swift, like you would for a regular target?

Yup. Anders' proposal will fix this.

Is there any reason why we cannot just support the same sorts of specifications of dependencies (and anything else) for tests as we do for regular targets?

In the current system, the reason is that users have no idea how to "spell" the name of the test module.

More important than not building the executables downstream, I don't want to expose the test executables as public, whatever that means. As far as I can tell, swiftpm does not yet do anything to make package executables available to the end user, but it might in the future. For example, Pypi can generate "entry points" that end up in the python bin directory. If swiftpm started making packages executables available to the user on the PATH or any way, keeping tests out of that process would be important.

Put another way, it does not seem far fetched that swiftpm could gain an `swift package install` command or similar, and it would be unfortunate if published packages had in the meantime been publishing test executables as public targets.

This makes sense to me.

The only problem I see with this line of reasoning is that it only extends to one category ("Tests" in this case). If we were to support multiple categories it would become unwieldy if each new category was always mirrored in the file organization, and we would probably want to move to using attributes in the manifest. At that point, one might question why we didn't use an attribute to say "this executable is private" in the first place. So I think it comes down to motivating whether or not "test executables" are common/special enough to motivate having a special case for. Obviously we already decided Test modules are...

For what it is worth, my personal opinion is that the convention does make sense, mostly because it seems very intuitive. I don't expect test executables to be *that* common, but test support libraries (code shared among test modules, but not needed by clients) are, and having a nice convention for those seems valuable.

Bear in mind that the perspective I am coming from is that even if you want to do integration tests of test executables, the "driver" of those executables would still be an XCTest module adapting them, at least until we had third-party testing support.

This is the basic source of confusion I think; you mentioned in our initial slack chat that 3rd party testing frameworks was a concern, so I leapt to the conclusion that XCTest would not be the driver.

My perspective is that I already have my own harness written (github.com/gwk/iotest <http://github.com/gwk/iotest> - it's for personal use and depends on a fast-changing utility library so it might appear broken on a given day). I use it to test executable targets that I'm producing via swiftpm now.

I'm curious to hear a bit more about what you consider to be "3rd party testing support", because apart from this proposal, I can't think of anything else that iotest needs from the package manager!

My opinion is that what we want is for `swift test` to behave as a relatively framework neutral driver for each packages tests, while still being able to offer sophisticated features (parallel testing, performance tests, test filtering). I want test framework offers to be able to integrate with SwiftPM in a way that the "UI" for accessing the tests remains the same. I want "3rd-party" testing frameworks to feel just as native as XCTest (given sufficiently motivated framework providers) while also offering a uniform UI to SwiftPM users independent of the specific framework.

What this means is that we need a couple things:
1. We need some "protocol" (either an actual Swift protocol, or something else) by which we get all the information we need about tests. This will need to evolve over time, but the initial proposal would need to define where the protocol is, how it is implemented, as well as the actual APIs etc. I have ideas about how this can work... :slight_smile:
2. We need some way to find the right protocol for a given package (i.e., what framework is in use).

I don't think something as simple as assuming "each test executable is a test harness and exits with 0 or 1", because that limits what you can do with test executables (not all of them may be harnesses) but also just because the protocol isn't very rich. I would like for features like "list all tests" or "run this one test" to always work and behave uniformly -- that will require more work for us and more work for 3rd party framework providers (although hopefully not that much), but in the grand scheme of things will be much better for users IMHO.

HTH,
- Daniel

···

On Jul 9, 2016, at 2:31 PM, George King via swift-build-dev <swift-build-dev@swift.org> wrote:

On Jul 8, 2016, at 6:43 PM, Daniel Dunbar <daniel@zuster.org <mailto:daniel@zuster.org>> wrote:

Finally, I should add a few motivating examples for why test executables would be useful. The obvious case is for libraries that offer IO functionality; imagine a command line argument parser library, or an error logging framework. I have found that it is easiest to test these kinds of things with a harness that can specify environment, process arguments, stdin, and current working directory structure, and then capture exit status, stdout and stderr from a test process. iotest does this, and has a few nice tricks like printing diffs of the expected and actual output. Offering these sorts of features through XCTest would be a much larger proposal, so I think it is worth trying to carve out the test executable part as a first step.

- Daniel

On Thursday, July 7, 2016, George King via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:
Hello, here is a rough draft proposal, inspired by a conversation with Daniel Dunbar last week on the Slack channel. I realize it is light on details, but hopefully this will be a starting point for figuring out how to better support building dedicated test executable targets. Let me know what you think, and if there is anything else I can do to move this forward. Thanks, -George

Test Executables

Proposal: SE-NNNN <http://nnnn-filename.md/>
Author: George King <https://github.com/gwk>
Status: Awaiting review
Review manager: TBD
Introduction

The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables directly in the package manager would make command line testing of Swift packages easier and more robust.

Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.

XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing,and GUI testing, but not for testing the basic IO behavior of a process. Since the swift package manager supports building command-line executables, it should support testing them as well.

Consider the existing features of swiftpm as two orthogonal axes:

Build, Test
Library, Executable
Of the four combinations, only "Test Executable" is missing.

One preliminary step towards process-based testing would be to add conventions for building executables that are meant strictly for testing, rather than for public usage. These would be exercised by some external testing harness; adding such capabilities to XCTest is beyond the scope of this proposal.

Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:

Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
For simple projects that produce a single library module, adding a test executable no longer necessitates moving the library code from Sources/ to Sources/[Module] to allow for the second target.
Test executables would only be built by swift test, not swift build.
Proposed solution

swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.

The test command should, upon encountering a test executable directory, build the executable.

If the test executable build fails, test should report a test failure and return a nonzero exit code.

If the build succeeds, test should report a test success and clearly output the path to the resulting test executable, for consumption by external test harness scripts.

Future improvements could include invoking a test command specified in Package.swift upon successful build.

Detailed design

I am unclear on the exact semantics of @testable import (I see no difference when I remove the@testable modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.

swift test should output a list of successfully built executable test paths, for consumption by an external test harness. This should be a text file listing one path per line, so that simple shell scripts can run the tests without having to parse formats like json or yaml. The list should contain only executables that built successfully (or where deemed unchanged by dependency calculation), so that nonexistant and/or stale tests do not get executed. If a use case emerges that requires a list of all tests, then perhaps a complete listing could be produced separately.

Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.

Alternatives considered

No action

See the list of problems with treating test executables as regular targets in the "Motivation" section above.

Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(George King) #8

Daniel:

It looks like your last reply was referencing the original draft I sent in July, not the 2nd draft I sent 11/10. Pasted below is a 3rd draft, removing the bit about @testable.

Regarding your second comment, I had previously removed the stuff about outputting a manifest. I agree with both of your responses.

Ankit:

This proposal sounds similar to allowing regular modules (lib+exec) inside Tests/ and IMO that is better than just allowing executables and is more general. Right now we error out when some directory doesn't have Tests prefix in Tests/ folder, We can just allow that and use same rule of main.swift to distinguish between lib and exec.

Expanding the change to lib+exec seems reasonable. However, my understanding is that the motivation for supporting lib is a bit different. At the moment my proposal is almost entirely focused on motivation. If the change gets made, I'm happy to drop the proposal, but I just want to be clear about the use case.

Thanks,
George

Test Executables

Proposal: SE-NNNN <applewebdata://E6A571B4-6571-4D9B-B3EB-624D50AAC773/NNNN-filename.md>
Author: George King <https://github.com/gwk>
Status: Awaiting review / DRAFT 3
Review manager: TBD
Introduction

The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables (that is, executables meant strictly for testing and not as an end product) in the Swift Package Manager would make command line testing of Swift packages easier and more robust. We propose to add to the package manager the ability to specify and build test-only executable targets that are distinct from regular executables.

Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.

XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing, and GUI testing, but not for testing the basic IO behavior of a process. To be clear, this proposal does not address XCTest at all; while adding whole process, IO oriented APIs to XCTest sounds desirable, it is a larger project and out of scope for this proposal.

Instead, this proposal simply asks for the ability to build executables specifically for testing, rather than for public usage. The intended use case is execution via an external test harness, or from within an XCTest case via NSTask/Process.

Consider the existing features of swiftpm as two orthogonal axes:

Build, Test
Library, Executable
Of the four combinations, only "Test Executable" is missing.

Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:

Test executables would only be built by swift test, not swift build.
Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
For simple projects that produce a single library module, adding a test executable will no longer necessitate moving the library code from Sources/ to Sources/[Module] to allow for the second target.
Proposed solution

swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.

The swift test command should, upon encountering a test executable directory, build the executable.

If the test executable build fails, swift test should report a test failure and return a nonzero exit code. An external test harness can then run the test executables as it sees fit. Note that for an XCTest case to invoke the test executable, test executables would need to be built prior to running test cases.

Future improvements could include invoking an external test command specified in Package.swift upon successful build.

Detailed design

TODO.

Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.

Alternatives considered

No action

See the list of problems with treating test executables as regular targets in the "Motivation" section above.

Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.


(Daniel Dunbar) #9

Daniel:

It looks like your last reply was referencing the original draft I sent in July, not the 2nd draft I sent 11/10. Pasted below is a 3rd draft, removing the bit about @testable.

Regarding your second comment, I had previously removed the stuff about outputting a manifest. I agree with both of your responses.

Oops, dunno what happened there. :slight_smile:

Ankit:

This proposal sounds similar to allowing regular modules (lib+exec) inside Tests/ and IMO that is better than just allowing executables and is more general. Right now we error out when some directory doesn't have Tests prefix in Tests/ folder, We can just allow that and use same rule of main.swift to distinguish between lib and exec.

Expanding the change to lib+exec seems reasonable. However, my understanding is that the motivation for supporting lib is a bit different. At the moment my proposal is almost entirely focused on motivation. If the change gets made, I'm happy to drop the proposal, but I just want to be clear about the use case.

I agree with Ankit that it probably makes sense to add them both in one go. Your motivation partially applies here w.r.t. consistency in the matrix of supported options, the only additional bit of motivation for libraries is that test targets may want to share code in a place clearly distinguished from the package's product libraries -- thats similar to the motivation for not putting the test libraries in with Sources.

Aside from that, the proposal LGTM. I don't really think anything is needed for detailed design, this is just extending the existing convention.

If no one else has feedback before, say, Thursday, and you are willing to add the bit above, I can serve as review manager for this over next week if you make a PR against swift-evolution and cc me on it.

- Daniel

···

On Nov 15, 2016, at 10:16 AM, George King <george.w.king@gmail.com> wrote:

Thanks,
George

Test Executables

Proposal: SE-NNNN <x-msg://268/NNNN-filename.md>
Author: George King <https://github.com/gwk>
Status: Awaiting review / DRAFT 3
Review manager: TBD
Introduction

The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables (that is, executables meant strictly for testing and not as an end product) in the Swift Package Manager would make command line testing of Swift packages easier and more robust. We propose to add to the package manager the ability to specify and build test-only executable targets that are distinct from regular executables.

Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.

XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing, and GUI testing, but not for testing the basic IO behavior of a process. To be clear, this proposal does not address XCTest at all; while adding whole process, IO oriented APIs to XCTest sounds desirable, it is a larger project and out of scope for this proposal.

Instead, this proposal simply asks for the ability to build executables specifically for testing, rather than for public usage. The intended use case is execution via an external test harness, or from within an XCTest case via NSTask/Process.

Consider the existing features of swiftpm as two orthogonal axes:

Build, Test
Library, Executable
Of the four combinations, only "Test Executable" is missing.

Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:

Test executables would only be built by swift test, not swift build.
Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
For simple projects that produce a single library module, adding a test executable will no longer necessitate moving the library code from Sources/ to Sources/[Module] to allow for the second target.
Proposed solution

swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.

The swift test command should, upon encountering a test executable directory, build the executable.

If the test executable build fails, swift test should report a test failure and return a nonzero exit code. An external test harness can then run the test executables as it sees fit. Note that for an XCTest case to invoke the test executable, test executables would need to be built prior to running test cases.

Future improvements could include invoking an external test command specified in Package.swift upon successful build.

Detailed design

TODO.

Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.

Alternatives considered

No action

See the list of problems with treating test executables as regular targets in the "Motivation" section above.

Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.


(George King) #10

I agree with Ankit that it probably makes sense to add them both in one go. Your motivation partially applies here w.r.t. consistency in the matrix of supported options, the only additional bit of motivation for libraries is that test targets may want to share code in a place clearly distinguished from the package's product libraries -- thats similar to the motivation for not putting the test libraries in with Sources.

Aside from that, the proposal LGTM. I don't really think anything is needed for detailed design, this is just extending the existing convention.

If no one else has feedback before, say, Thursday, and you are willing to add the bit above, I can serve as review manager for this over next week if you make a PR against swift-evolution and cc me on it.

That's great. I'll fix it up to read as 'Test Libraries and Executables'.