[Draft] Test-Only Package Dependencies and Targets

[codafi (Robert Widmann)]

Hello Swift Community,

Harlan Haskins and I have been working on libraries <https://github.com/trill-lang&gt; to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces. While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies. To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager. This proposal can also be read in gist form <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38&gt;\.

Cheers,

~Robert Widmann

Test-Only Package Dependencies and Targets

Proposal: SE-NNNN <https://gist.github.com/CodaFi/NNNN-filename.md&gt;
Authors: Harlan Haskins <https://github.com/harlanhaskins&gt;, Robert Widmann <https://github.com/codafi&gt;
Review Manager: TBD
Status: Awaiting review
<Proposal.md · GitHub

This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/&gt;
<Proposal.md · GitHub

Soon after SE-0019 <swift-evolution/0019-package-manager-testing.md at master · apple/swift-evolution · GitHub; identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1] <https://github.com/ReactiveCocoa/ReactiveSwift/blob/master/.travis.yml#L85&gt;, [2] <RxSwift/Package.swift at main · ReactiveX/RxSwift · GitHub, [3] <https://github.com/Quick/Quick/blob/master/.Package.test.swift&gt;\) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.

<Proposal.md · GitHub solution

We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.

<Proposal.md · GitHub design

The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.

import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)
Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.

import PackageDescription

let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries

        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)
Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.

We propose the following syntax to execute all tests of all known test targets.

$ swift test
To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase
To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest
<Proposal.md · GitHub on Existing Code

As this change is purely additive there will be no impact on existing code and no impact on existing packages.

<Proposal.md · GitHub considered

Other names for the parameters to be added to the package manifest are possible.

The new support for executing specific tests could be exposed behind a flag.

[Russ_Bishop1 (Russ Bishop)]

I never agreed with removing it so this gets a bit fat +1 from me.

Russ

···

On Jan 24, 2017, at 2:32 PM, Robert Widmann via swift-evolution <swift-evolution@swift.org> wrote:

Hello Swift Community,

Harlan Haskins and I have been working on libraries <https://github.com/trill-lang&gt; to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces. While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies. To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager. This proposal can also be read in gist form <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38&gt;\.

Cheers,

~Robert Widmann

Test-Only Package Dependencies and Targets

Proposal: SE-NNNN <https://gist.github.com/CodaFi/NNNN-filename.md&gt;
Authors: Harlan Haskins <https://github.com/harlanhaskins&gt;, Robert Widmann <https://github.com/codafi&gt;
Review Manager: TBD
Status: Awaiting review
<Proposal.md · GitHub

This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/&gt;
<Proposal.md · GitHub

Soon after SE-0019 <https://github.com/apple/swift-evolution/blob/master/proposals/0019-package-manager-testing.md#test-only-dependencies&gt; identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1] <https://github.com/ReactiveCocoa/ReactiveSwift/blob/master/.travis.yml#L85&gt;, [2] <https://github.com/ReactiveX/RxSwift/blob/master/Package.swift#L3&gt;, [3] <https://github.com/Quick/Quick/blob/master/.Package.test.swift&gt;\) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.

<Proposal.md · GitHub solution

We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.

<Proposal.md · GitHub design

The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.

import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)
Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.

import PackageDescription

let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries

        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)
Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.

We propose the following syntax to execute all tests of all known test targets.

$ swift test
To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase
To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest
<Proposal.md · GitHub on Existing Code

As this change is purely additive there will be no impact on existing code and no impact on existing packages.

<Proposal.md · GitHub considered

Other names for the parameters to be added to the package manifest are possible.

The new support for executing specific tests could be exposed behind a flag.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[thislooksfun]

As the author of the Add support for test-only dependencies <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161226/029720.html&gt; thread, and the accompanying draft <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20170102/000811.html&gt;, this gets a big +1 from me.

-thislooksfun (tlf)

···

On Jan 24, 2017, at 4:32 PM, Robert Widmann via swift-evolution <swift-evolution@swift.org> wrote:

Hello Swift Community,

Harlan Haskins and I have been working on libraries <https://github.com/trill-lang&gt; to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces. While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies. To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager. This proposal can also be read in gist form <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38&gt;\.

Cheers,

~Robert Widmann

Test-Only Package Dependencies and Targets

Proposal: SE-NNNN <https://gist.github.com/CodaFi/NNNN-filename.md&gt;
Authors: Harlan Haskins <https://github.com/harlanhaskins&gt;, Robert Widmann <https://github.com/codafi&gt;
Review Manager: TBD
Status: Awaiting review
<Proposal.md · GitHub

This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/&gt;
<Proposal.md · GitHub

Soon after SE-0019 <https://github.com/apple/swift-evolution/blob/master/proposals/0019-package-manager-testing.md#test-only-dependencies&gt; identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1] <https://github.com/ReactiveCocoa/ReactiveSwift/blob/master/.travis.yml#L85&gt;, [2] <https://github.com/ReactiveX/RxSwift/blob/master/Package.swift#L3&gt;, [3] <https://github.com/Quick/Quick/blob/master/.Package.test.swift&gt;\) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.

<Proposal.md · GitHub solution

We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.

<Proposal.md · GitHub design

The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.

import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)
Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.

import PackageDescription

let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries

        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)
Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.

We propose the following syntax to execute all tests of all known test targets.

$ swift test
To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase
To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest
<Proposal.md · GitHub on Existing Code

As this change is purely additive there will be no impact on existing code and no impact on existing packages.

<Proposal.md · GitHub considered

Other names for the parameters to be added to the package manifest are possible.

The new support for executing specific tests could be exposed behind a flag.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Aciid (Ankit Aggarwal)]

Hello Swift Community,

Harlan Haskins and I have been working on libraries <https://github.com/trill-lang&gt; to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces. While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies. To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager. This proposal can also be read in gist form <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38&gt;\.

Cheers,

~Robert Widmann

Thanks for driving this! It is a very desirable feature which needs proposal work. Comments inline.

Test-Only Package Dependencies and Targets

Proposal: SE-NNNN <https://gist.github.com/CodaFi/NNNN-filename.md&gt;
Authors: Harlan Haskins <https://github.com/harlanhaskins&gt;, Robert Widmann <https://github.com/codafi&gt;
Review Manager: TBD
Status: Awaiting review
<Proposal.md · GitHub

This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/&gt;
<Proposal.md · GitHub

Soon after SE-0019 <https://github.com/apple/swift-evolution/blob/master/proposals/0019-package-manager-testing.md#test-only-dependencies&gt; identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1] <https://github.com/ReactiveCocoa/ReactiveSwift/blob/master/.travis.yml#L85&gt;, [2] <https://github.com/ReactiveX/RxSwift/blob/master/Package.swift#L3&gt;, [3] <https://github.com/Quick/Quick/blob/master/.Package.test.swift&gt;\) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.

<Proposal.md · GitHub solution

We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.

<Proposal.md · GitHub design

The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.

import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)

I think this feature should be called local dependencies (or maybe dev dependencies) because it can be used for tests as well as regular targets.
As an example say you have a networking library package and you want to create an example CLI target which uses a JSON mapper package. You wouldn't want to vend the CLI tool when you act as a dependency to other packages, or include the JSON mapper in your dependencies. Test dependency doesn't sound right in that context.

After the product proposal <https://github.com/apple/swift-evolution/blob/master/proposals/0146-package-manager-product-definitions.md&gt; is implemented, you will be able to control what you vend to your clients, and we thought about adding ability to define dependencies in-line with target dependencies but left it out of the proposal to keep it simpler. Maybe this proposal can add that instead of a separate `testDependencies` property. Consider this manifest:

let package = Package(
    name: "FooNetworking",
    targets: [
        Target(
            name: "FooNetworking"),
        Target(
            name: "FooNetworkingExample",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/BarJSON.git", version: "1.0.0", product: "BarJSON"), // Note: This doesn't actually exists right now.
            ]),
        Target(
            name: "FooNetworkingTests",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/Quick.git", version: "1.0.0", product: "Quick"), // Note: This doesn't actually exists right now.
            ]),
   ],
   products: [
       .Library(name: "FooNetworking", targets: ["FooNetworking"]),
   ]
)

This manifest defines three targets and one product.

FooNetworking: The base target and the actual library.
FooNetworkingExample: The example cli tool. It depends on FooNetworking target and BarJSON product from an external package.
FooNetworkingTests: The test target depends on an external package Quick.

Both BarJSON and Quick are local dependencies to this package. If FooNetworkingExample was also vended as a product, BarJSON would automatically become a regular external dependency.

Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.

import PackageDescription

let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries

        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)
Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.

We propose the following syntax to execute all tests of all known test targets.

$ swift test
To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase
To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest

We already have all these features. A target is a test target if it has a suffix Tests and is placed under Tests/ directory — we might loosen these restrictions with custom target conventions feature. This proposal <https://github.com/apple/swift-evolution/blob/master/proposals/0129-package-manager-test-naming-conventions.md&gt; could clarify further.

Here is a summary of what `swift test` currently supports:
* Test targets are only built (and run) when you run `swift test`.
* `swift test --skip-build` skips building of test targets and runs whatever was last built.
* `swift test -s <test-module>.<test-case>` will run a test case class.
* `swift test -s <test-module>.<test-case>/<test>` will run an individual test.
* `swift test --list-tests` will list all tests in above format.
* `swift test --parallel` will run tests in parallel.

You can view these options and their help text using `swift test --help`.

···

On 25-Jan-2017, at 4:02 AM, Robert Widmann via swift-evolution <swift-evolution@swift.org> wrote:

[codafi (Robert Widmann)]

Oh, I'm sorry. I didn't know there was existing work in this space. Considering this proposal looks like it expands on your earlier work, would you like to sign on to this and merge the two efforts?

~Robert Widmann

2017/01/25 0:47、thislooksfun <thislooksfun@repbot.org> のメッセージ:

···

As the author of the Add support for test-only dependencies thread, and the accompanying draft, this gets a big +1 from me.

-thislooksfun (tlf)

On Jan 24, 2017, at 4:32 PM, Robert Widmann via swift-evolution <swift-evolution@swift.org> wrote:

Hello Swift Community,

Harlan Haskins and I have been working on libraries to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces. While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies. To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager. This proposal can also be read in gist form.

Cheers,

~Robert Widmann

Test-Only Package Dependencies and Targets
Proposal: SE-NNNN
Authors: Harlan Haskins, Robert Widmann
Review Manager: TBD
Status: Awaiting review
Introduction

This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation

Soon after SE-0019 identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1], [2], [3]) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.

Proposed solution

We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.

Detailed design

The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.

import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)
Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.

import PackageDescription

let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries

        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)
Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.

We propose the following syntax to execute all tests of all known test targets.

$ swift test
To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase
To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest
Impact on Existing Code

As this change is purely additive there will be no impact on existing code and no impact on existing packages.

Alternatives considered

Other names for the parameters to be added to the package manifest are possible.

The new support for executing specific tests could be exposed behind a flag.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[thislooksfun]

I still much prefer 'testDependencies/Targets'. You seem to be confusing this proposal with the (already accepted) Package Manager Product Definitions <https://github.com/apple/swift-evolution/blob/master/proposals/0146-package-manager-product-definitions.md&gt; proposal.
This proposal is strictly for running under `swift test`. The issue currently is that running any swift command (specifically anything that ends up calling `swift build`), will attempt to build testing libraries / targets, which then fail because they rely on XCTest, which is only available for import when running `swift test`. The idea here is that the libraries / targets will only be compiled when running in a testing environment, without having to manually switch out Package.swift files.

-thislooksfun (tlf)

···

On Jan 25, 2017, at 2:07 AM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org> wrote:

On 25-Jan-2017, at 4:02 AM, Robert Widmann via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hello Swift Community,

Harlan Haskins and I have been working on libraries <https://github.com/trill-lang&gt; to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces. While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies. To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager. This proposal can also be read in gist form <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38&gt;\.

Cheers,

~Robert Widmann

Thanks for driving this! It is a very desirable feature which needs proposal work. Comments inline.

Test-Only Package Dependencies and Targets

Proposal: SE-NNNN <https://gist.github.com/CodaFi/NNNN-filename.md&gt;
Authors: Harlan Haskins <https://github.com/harlanhaskins&gt;, Robert Widmann <https://github.com/codafi&gt;
Review Manager: TBD
Status: Awaiting review
<Proposal.md · GitHub

This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/&gt;
<Proposal.md · GitHub

Soon after SE-0019 <https://github.com/apple/swift-evolution/blob/master/proposals/0019-package-manager-testing.md#test-only-dependencies&gt; identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1] <https://github.com/ReactiveCocoa/ReactiveSwift/blob/master/.travis.yml#L85&gt;, [2] <https://github.com/ReactiveX/RxSwift/blob/master/Package.swift#L3&gt;, [3] <https://github.com/Quick/Quick/blob/master/.Package.test.swift&gt;\) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.

<Proposal.md · GitHub solution

We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.

<Proposal.md · GitHub design

The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.

import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)

I think this feature should be called local dependencies (or maybe dev dependencies) because it can be used for tests as well as regular targets.
As an example say you have a networking library package and you want to create an example CLI target which uses a JSON mapper package. You wouldn't want to vend the CLI tool when you act as a dependency to other packages, or include the JSON mapper in your dependencies. Test dependency doesn't sound right in that context.

After the product proposal <https://github.com/apple/swift-evolution/blob/master/proposals/0146-package-manager-product-definitions.md&gt; is implemented, you will be able to control what you vend to your clients, and we thought about adding ability to define dependencies in-line with target dependencies but left it out of the proposal to keep it simpler. Maybe this proposal can add that instead of a separate `testDependencies` property. Consider this manifest:

let package = Package(
    name: "FooNetworking",
    targets: [
        Target(
            name: "FooNetworking"),
        Target(
            name: "FooNetworkingExample",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/BarJSON.git", version: "1.0.0", product: "BarJSON"), // Note: This doesn't actually exists right now.
            ]),
        Target(
            name: "FooNetworkingTests",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/Quick.git", version: "1.0.0", product: "Quick"), // Note: This doesn't actually exists right now.
            ]),
   ],
   products: [
       .Library(name: "FooNetworking", targets: ["FooNetworking"]),
   ]
)

This manifest defines three targets and one product.

FooNetworking: The base target and the actual library.
FooNetworkingExample: The example cli tool. It depends on FooNetworking target and BarJSON product from an external package.
FooNetworkingTests: The test target depends on an external package Quick.

Both BarJSON and Quick are local dependencies to this package. If FooNetworkingExample was also vended as a product, BarJSON would automatically become a regular external dependency.

Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.

import PackageDescription

let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries

        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)
Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.

We propose the following syntax to execute all tests of all known test targets.

$ swift test
To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase
To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest

We already have all these features. A target is a test target if it has a suffix Tests and is placed under Tests/ directory — we might loosen these restrictions with custom target conventions feature. This proposal <https://github.com/apple/swift-evolution/blob/master/proposals/0129-package-manager-test-naming-conventions.md&gt; could clarify further.

Here is a summary of what `swift test` currently supports:
* Test targets are only built (and run) when you run `swift test`.
* `swift test --skip-build` skips building of test targets and runs whatever was last built.
* `swift test -s <test-module>.<test-case>` will run a test case class.
* `swift test -s <test-module>.<test-case>/<test>` will run an individual test.
* `swift test --list-tests` will list all tests in above format.
* `swift test --parallel` will run tests in parallel.

You can view these options and their help text using `swift test --help`.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[codafi (Robert Widmann)]

Three parts for three points:

1) Thanks!

2) I'm not sure generalizing this is particularly useful post-products. The larger point is not to have non-exported/local target dependencies but to make clear the divide between a package and its test suite in the manifest as well as on disk. Local dependencies seem like a separate proposal entirely.

3) Wow, I honestly should have noticed this: I took the time to peruse the source for the help text and still managed to miss it. It has been struck from the gist of this proposal.

~Robert Widmann

2017/01/25 3:07、Ankit Aggarwal <ankit_aggarwal@apple.com> のメッセージ:

···

On 25-Jan-2017, at 4:02 AM, Robert Widmann via swift-evolution <swift-evolution@swift.org> wrote:

Hello Swift Community,

Harlan Haskins and I have been working on libraries to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces. While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies. To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager. This proposal can also be read in gist form.

Cheers,

~Robert Widmann

Thanks for driving this! It is a very desirable feature which needs proposal work. Comments inline.

Test-Only Package Dependencies and Targets
Proposal: SE-NNNN
Authors: Harlan Haskins, Robert Widmann
Review Manager: TBD
Status: Awaiting review
Introduction

This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation

Soon after SE-0019 identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1], [2], [3]) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.

Proposed solution

We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.

Detailed design

The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.

import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)

I think this feature should be called local dependencies (or maybe dev dependencies) because it can be used for tests as well as regular targets.
As an example say you have a networking library package and you want to create an example CLI target which uses a JSON mapper package. You wouldn't want to vend the CLI tool when you act as a dependency to other packages, or include the JSON mapper in your dependencies. Test dependency doesn't sound right in that context.

After the product proposal is implemented, you will be able to control what you vend to your clients, and we thought about adding ability to define dependencies in-line with target dependencies but left it out of the proposal to keep it simpler. Maybe this proposal can add that instead of a separate `testDependencies` property. Consider this manifest:

let package = Package(
    name: "FooNetworking",
    targets: [
        Target(
            name: "FooNetworking"),
        Target(
            name: "FooNetworkingExample",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/BarJSON.git", version: "1.0.0", product: "BarJSON"), // Note: This doesn't actually exists right now.
            ]),
        Target(
            name: "FooNetworkingTests",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/Quick.git", version: "1.0.0", product: "Quick"), // Note: This doesn't actually exists right now.
            ]),
   ],
   products: [
       .Library(name: "FooNetworking", targets: ["FooNetworking"]),
   ]
)

This manifest defines three targets and one product.

FooNetworking: The base target and the actual library.
FooNetworkingExample: The example cli tool. It depends on FooNetworking target and BarJSON product from an external package.
FooNetworkingTests: The test target depends on an external package Quick.

Both BarJSON and Quick are local dependencies to this package. If FooNetworkingExample was also vended as a product, BarJSON would automatically become a regular external dependency.

Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.

import PackageDescription

let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries

        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)
Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.

We propose the following syntax to execute all tests of all known test targets.

$ swift test
To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase
To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest

We already have all these features. A target is a test target if it has a suffix Tests and is placed under Tests/ directory — we might loosen these restrictions with custom target conventions feature. This proposal could clarify further.

Here is a summary of what `swift test` currently supports:
* Test targets are only built (and run) when you run `swift test`.
* `swift test --skip-build` skips building of test targets and runs whatever was last built.
* `swift test -s <test-module>.<test-case>` will run a test case class.
* `swift test -s <test-module>.<test-case>/<test>` will run an individual test.
* `swift test --list-tests` will list all tests in above format.
* `swift test --parallel` will run tests in parallel.

You can view these options and their help text using `swift test --help`.

[thislooksfun]

No problem, it's been inactive for a couple weeks since I've been really busy. And yeah, yours is more thorough. I didn't even consider adding a `testTargets` section. That alone is worth merging.

-thislooksfun (tlf)

···

On Jan 24, 2017, at 11:55 PM, Robert Widmann <devteam.codafi@gmail.com> wrote:

Oh, I'm sorry. I didn't know there was existing work in this space. Considering this proposal looks like it expands on your earlier work, would you like to sign on to this and merge the two efforts?

~Robert Widmann

2017/01/25 0:47、thislooksfun <thislooksfun@repbot.org <mailto:thislooksfun@repbot.org>> のメッセージ:

As the author of the Add support for test-only dependencies <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161226/029720.html&gt; thread, and the accompanying draft <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20170102/000811.html&gt;, this gets a big +1 from me.

-thislooksfun (tlf)

On Jan 24, 2017, at 4:32 PM, Robert Widmann via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hello Swift Community,

Harlan Haskins and I have been working on libraries <https://github.com/trill-lang&gt; to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces. While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies. To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager. This proposal can also be read in gist form <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38&gt;\.

Cheers,

~Robert Widmann

Test-Only Package Dependencies and Targets

Proposal: SE-NNNN <https://gist.github.com/CodaFi/NNNN-filename.md&gt;
Authors: Harlan Haskins <https://github.com/harlanhaskins&gt;, Robert Widmann <https://github.com/codafi&gt;
Review Manager: TBD
Status: Awaiting review
<Proposal.md · GitHub

This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/&gt;
<Proposal.md · GitHub

Soon after SE-0019 <https://github.com/apple/swift-evolution/blob/master/proposals/0019-package-manager-testing.md#test-only-dependencies&gt; identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1] <https://github.com/ReactiveCocoa/ReactiveSwift/blob/master/.travis.yml#L85&gt;, [2] <https://github.com/ReactiveX/RxSwift/blob/master/Package.swift#L3&gt;, [3] <https://github.com/Quick/Quick/blob/master/.Package.test.swift&gt;\) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.

<Proposal.md · GitHub solution

We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.

<Proposal.md · GitHub design

The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.

import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)
Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.

import PackageDescription

let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries

        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)
Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.

We propose the following syntax to execute all tests of all known test targets.

$ swift test
To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase
To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest
<Proposal.md · GitHub on Existing Code

As this change is purely additive there will be no impact on existing code and no impact on existing packages.

<Proposal.md · GitHub considered

Other names for the parameters to be added to the package manifest are possible.

The new support for executing specific tests could be exposed behind a flag.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[codafi (Robert Widmann)]

Excellent. I've added your name to the authors list on the gist. That is the version I'll be submitting to evolution soon.

~Robert Widmann

2017/01/25 0:58、thislooksfun <thislooksfun@repbot.org> のメッセージ:

···

No problem, it's been inactive for a couple weeks since I've been really busy. And yeah, yours is more thorough. I didn't even consider adding a `testTargets` section. That alone is worth merging.

-thislooksfun (tlf)

On Jan 24, 2017, at 11:55 PM, Robert Widmann <devteam.codafi@gmail.com> wrote:

Oh, I'm sorry. I didn't know there was existing work in this space. Considering this proposal looks like it expands on your earlier work, would you like to sign on to this and merge the two efforts?

~Robert Widmann

2017/01/25 0:47、thislooksfun <thislooksfun@repbot.org> のメッセージ:

As the author of the Add support for test-only dependencies thread, and the accompanying draft, this gets a big +1 from me.

-thislooksfun (tlf)

On Jan 24, 2017, at 4:32 PM, Robert Widmann via swift-evolution <swift-evolution@swift.org> wrote:

Hello Swift Community,

Harlan Haskins and I have been working on libraries to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces. While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies. To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager. This proposal can also be read in gist form.

Cheers,

~Robert Widmann

Test-Only Package Dependencies and Targets
Proposal: SE-NNNN
Authors: Harlan Haskins, Robert Widmann
Review Manager: TBD
Status: Awaiting review
Introduction

This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation

Soon after SE-0019 identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1], [2], [3]) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.

Proposed solution

We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.

Detailed design

The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.

import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)
Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.

import PackageDescription

let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries

        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)
Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.

We propose the following syntax to execute all tests of all known test targets.

$ swift test
To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase
To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest
Impact on Existing Code

As this change is purely additive there will be no impact on existing code and no impact on existing packages.

Alternatives considered

Other names for the parameters to be added to the package manifest are possible.

The new support for executing specific tests could be exposed behind a flag.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Ankit_Agarwal1 (Ankit Agarwal)]

I still much prefer 'testDependencies/Targets'. You seem to be confusing
this proposal with the (already accepted) Package Manager Product
Definitions
<https://github.com/apple/swift-evolution/blob/master/proposals/0146-package-manager-product-definitions.md&gt;
proposal.
This proposal is strictly for running under `swift test`. The issue
currently is that running any swift command (specifically anything that
ends up calling `swift build`), will attempt to build testing libraries /
targets, which then fail because they rely on XCTest, which is only
available for import when running `swift test`. The idea here is that the
libraries / targets will *only* be compiled when running in a testing
environment, without having to manually switch out Package.swift files.

The test targets are *not* compiled when you run swift build. They're only
complied on invocation of `swift test`. If a test target depends on a
testing library (as in my example), they will *not* be built on invocation
of `swift build`.

Also, you can import XCTest from your regular targets but that code will
only run from a test target. (see TestSupport module in SwiftPM).

···

On Thu, Jan 26, 2017 at 6:37 AM, thislooksfun via swift-evolution < swift-evolution@swift.org> wrote:

-thislooksfun (tlf)

On Jan 25, 2017, at 2:07 AM, Ankit Aggarwal via swift-evolution < > swift-evolution@swift.org> wrote:

On 25-Jan-2017, at 4:02 AM, Robert Widmann via swift-evolution < > swift-evolution@swift.org> wrote:

Hello Swift Community,

Harlan Haskins and I have been working on libraries
<https://github.com/trill-lang&gt; to make interacting with LLVM and Clang’s
APIs more elegant with native Swift interfaces. While writing up the
packages we realized the package manager wouldn’t allow us to specify
testing targets and test-only dependencies. To rectify that, I have
attached a draft proposal for adding test-only targets and dependency
fields to the Swift Package manager. This proposal can also be read in
gist form
<https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38&gt;\.

Cheers,

~Robert Widmann

Thanks for driving this! It is a very desirable feature which needs
proposal work. Comments inline.

Test-Only Package Dependencies and Targets

   - Proposal: SE-NNNN <https://gist.github.com/CodaFi/NNNN-filename.md&gt;
   - Authors: Harlan Haskins <https://github.com/harlanhaskins&gt;, Robert
   Widmann <https://github.com/codafi&gt;
   - Review Manager: TBD
   - Status: Awaiting review

<Proposal.md · GitHub;
Introduction

This proposal reinstates Swift package manager’s ability to fetch
dependencies and build targets scoped exclusively to the testing module(s)
of a given package.

Swift-evolution thread: Discussion thread topic for that proposal
<https://lists.swift.org/pipermail/swift-evolution/&gt;

<Proposal.md · GitHub;
Motivation

Soon after SE-0019
<https://github.com/apple/swift-evolution/blob/master/proposals/0019-package-manager-testing.md#test-only-dependencies&gt; identified
the need for richer test-only dependencies and targets, a decision was made
to remove the package manager’s fledgling ability to treat certain
dependencies as test-only. This has led to a myriad of
clever-but-needlessly-complex workarounds ([1]
<https://github.com/ReactiveCocoa/ReactiveSwift/blob/master/.travis.yml#L85&gt;
, [2] <https://github.com/ReactiveX/RxSwift/blob/master/Package.swift#L3&gt;
, [3] <https://github.com/Quick/Quick/blob/master/.Package.test.swift&gt;\)
on the part of 3rd parties to recover the feature themselves. In addition,
the Swift community has come up with a number of their own frameworks to
augment functionality in XCTest but depending on these external testing
frameworks is brittle and difficult to get right.

<Proposal.md · GitHub
solution

We propose the re-introduction of the testDependencies parameter in
Package Manifests to support external test-only dependencies. To support
local test-only targets we also propose the introduction of the
testTargets parameter and an extension of the existing swift test command
to support individual invocation of these targets.

<Proposal.md · GitHub
design

The behavior of the new testDependencies parameter mirrors that of the
existing dependencies parameter with one important difference: fetched
dependencies are only built to support package-defined test targets as part
of an invocation of swift test.

import PackageDescription
let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)

I think this feature should be called local dependencies (or maybe dev
dependencies) because it can be used for tests as well as regular targets.
As an example say you have a networking library package and you want to
create an example CLI target which uses a JSON mapper package. You wouldn't
want to vend the CLI tool when you act as a dependency to other packages,
or include the JSON mapper in your dependencies. Test dependency doesn't
sound right in that context.

After the product proposal
<https://github.com/apple/swift-evolution/blob/master/proposals/0146-package-manager-product-definitions.md&gt; is
implemented, you will be able to control what you vend to your clients, and
we thought about adding ability to define dependencies in-line with target
dependencies but left it out of the proposal to keep it simpler. Maybe this
proposal can add that instead of a separate `testDependencies` property.
Consider this manifest:

let package = Package(
    name: "FooNetworking",
    targets: [
        Target(
            name: "FooNetworking"),
        Target(
            name: "FooNetworkingExample",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/BarJSON.git", version: "1.0.0", product: "BarJSON"), // Note: This doesn't actually exists right now.
            ]),
        Target(
            name: "FooNetworkingTests",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/Quick.git", version: "1.0.0", product: "Quick"), // Note: This doesn't actually exists right now.
            ]),
   ],
   products: [
       .Library(name: "FooNetworking", targets: ["FooNetworking"]),
   ])

This manifest defines three targets and one product.

*FooNetworking*: The base target and the actual library.
*FooNetworkingExample*: The example cli tool. It depends on
*FooNetworking* target and *BarJSON* product from an external package.
*FooNetworkingTests*: The test target depends on an external package
*Quick*.

Both *BarJSON* and *Quick* are local dependencies to this package. If
*FooNetworkingExample* was also vended as a product, *BarJSON* would
automatically become a regular external dependency.

Similarly, the behavior of the testTargets field mirrors that of the
existing targets field but defines a set of targets that are only built
during an invocation of swift test. Importantly, a target defined in
testTargets may reference a target defined in targets but not vice-versa.
Should that behavior be needed, the test target should be promoted to a
“full” target.

import PackageDescription
let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries
        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)

Finally, with well-defined test targets in hand, we propose swift test be
amended to support individual test execution.

We propose the following syntax to execute all tests of all known test
targets.

$ swift test

To run a set of specific test cases, reference the module-defining test
target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase

To run an individual test case, reference the module-defining test target,
the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest

We already have all these features. A target is a test target if it has a
suffix *Tests *and is placed under *Tests/ *directory — we might loosen
these restrictions with custom target conventions feature. This proposal
<https://github.com/apple/swift-evolution/blob/master/proposals/0129-package-manager-test-naming-conventions.md&gt; could
clarify further.

Here is a summary of what `swift test` currently supports:
* Test targets are only built (and run) when you run `swift test`.
* `swift test --skip-build` skips building of test targets and runs
whatever was last built.
* `swift test -s <test-module>.<test-case>` will run a test case class.
* `swift test -s <test-module>.<test-case>/<test>` will run an individual
test.
* `swift test --list-tests` will list all tests in above format.
* `swift test --parallel` will run tests in parallel.

You can view these options and their help text using `swift test --help`.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Ankit

[Ankit_Agarwal1 (Ankit Agarwal)]

Three parts for three points:

1) Thanks!

2) I'm not sure generalizing this is particularly useful post-products.
The larger point is not to have non-exported/local target dependencies but
to make clear the divide between a package and its test suite in the
manifest as well as on disk. Local dependencies seem like a separate
proposal entirely.

We do have a clear divide between normal modules and test modules.
I am not sure adding a special case test dependencies makes sense right
now. We might end up having conditional dependencies later w.r.t platform
and other parameters.

Daniel, do you want to pitch in?

3) Wow, I honestly should have noticed this: I took the time to peruse the

···

On Thu, Jan 26, 2017 at 1:55 PM, Robert Widmann via swift-evolution < swift-evolution@swift.org> wrote:

source for the help text and still managed to miss it. It has been struck
from the gist of this proposal.

~Robert Widmann

2017/01/25 3:07、Ankit Aggarwal <ankit_aggarwal@apple.com> のメッセージ:

On 25-Jan-2017, at 4:02 AM, Robert Widmann via swift-evolution < > swift-evolution@swift.org> wrote:

Hello Swift Community,

Harlan Haskins and I have been working on libraries
<https://github.com/trill-lang&gt; to make interacting with LLVM and Clang’s
APIs more elegant with native Swift interfaces. While writing up the
packages we realized the package manager wouldn’t allow us to specify
testing targets and test-only dependencies. To rectify that, I have
attached a draft proposal for adding test-only targets and dependency
fields to the Swift Package manager. This proposal can also be read in
gist form
<https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38&gt;\.

Cheers,

~Robert Widmann

Thanks for driving this! It is a very desirable feature which needs
proposal work. Comments inline.

Test-Only Package Dependencies and Targets

   - Proposal: SE-NNNN <https://gist.github.com/CodaFi/NNNN-filename.md&gt;
   - Authors: Harlan Haskins <https://github.com/harlanhaskins&gt;, Robert
   Widmann <https://github.com/codafi&gt;
   - Review Manager: TBD
   - Status: Awaiting review

<Proposal.md · GitHub;
Introduction

This proposal reinstates Swift package manager’s ability to fetch
dependencies and build targets scoped exclusively to the testing module(s)
of a given package.

Swift-evolution thread: Discussion thread topic for that proposal
<https://lists.swift.org/pipermail/swift-evolution/&gt;

<Proposal.md · GitHub;
Motivation

Soon after SE-0019
<https://github.com/apple/swift-evolution/blob/master/proposals/0019-package-manager-testing.md#test-only-dependencies&gt; identified
the need for richer test-only dependencies and targets, a decision was made
to remove the package manager’s fledgling ability to treat certain
dependencies as test-only. This has led to a myriad of
clever-but-needlessly-complex workarounds ([1]
<https://github.com/ReactiveCocoa/ReactiveSwift/blob/master/.travis.yml#L85&gt;
, [2] <https://github.com/ReactiveX/RxSwift/blob/master/Package.swift#L3&gt;
, [3] <https://github.com/Quick/Quick/blob/master/.Package.test.swift&gt;\)
on the part of 3rd parties to recover the feature themselves. In addition,
the Swift community has come up with a number of their own frameworks to
augment functionality in XCTest but depending on these external testing
frameworks is brittle and difficult to get right.

<Proposal.md · GitHub
solution

We propose the re-introduction of the testDependencies parameter in
Package Manifests to support external test-only dependencies. To support
local test-only targets we also propose the introduction of the
testTargets parameter and an extension of the existing swift test command
to support individual invocation of these targets.

<Proposal.md · GitHub
design

The behavior of the new testDependencies parameter mirrors that of the
existing dependencies parameter with one important difference: fetched
dependencies are only built to support package-defined test targets as part
of an invocation of swift test.

import PackageDescription
let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)

I think this feature should be called local dependencies (or maybe dev
dependencies) because it can be used for tests as well as regular targets.
As an example say you have a networking library package and you want to
create an example CLI target which uses a JSON mapper package. You wouldn't
want to vend the CLI tool when you act as a dependency to other packages,
or include the JSON mapper in your dependencies. Test dependency doesn't
sound right in that context.

After the product proposal
<https://github.com/apple/swift-evolution/blob/master/proposals/0146-package-manager-product-definitions.md&gt; is
implemented, you will be able to control what you vend to your clients, and
we thought about adding ability to define dependencies in-line with target
dependencies but left it out of the proposal to keep it simpler. Maybe this
proposal can add that instead of a separate `testDependencies` property.
Consider this manifest:

let package = Package(
    name: "FooNetworking",
    targets: [
        Target(
            name: "FooNetworking"),
        Target(
            name: "FooNetworkingExample",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/BarJSON.git", version: "1.0.0", product: "BarJSON"), // Note: This doesn't actually exists right now.
            ]),
        Target(
            name: "FooNetworkingTests",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/Quick.git", version: "1.0.0", product: "Quick"), // Note: This doesn't actually exists right now.
            ]),
   ],
   products: [
       .Library(name: "FooNetworking", targets: ["FooNetworking"]),
   ])

This manifest defines three targets and one product.

*FooNetworking*: The base target and the actual library.
*FooNetworkingExample*: The example cli tool. It depends on
*FooNetworking* target and *BarJSON* product from an external package.
*FooNetworkingTests*: The test target depends on an external package
*Quick*.

Both *BarJSON* and *Quick* are local dependencies to this package. If
*FooNetworkingExample* was also vended as a product, *BarJSON* would
automatically become a regular external dependency.

Similarly, the behavior of the testTargets field mirrors that of the
existing targets field but defines a set of targets that are only built
during an invocation of swift test. Importantly, a target defined in
testTargets may reference a target defined in targets but not vice-versa.
Should that behavior be needed, the test target should be promoted to a
“full” target.

import PackageDescription
let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries
        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)

Finally, with well-defined test targets in hand, we propose swift test be
amended to support individual test execution.

We propose the following syntax to execute all tests of all known test
targets.

$ swift test

To run a set of specific test cases, reference the module-defining test
target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase

To run an individual test case, reference the module-defining test target,
the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest

We already have all these features. A target is a test target if it has a
suffix *Tests *and is placed under *Tests/ *directory — we might loosen
these restrictions with custom target conventions feature. This proposal
<https://github.com/apple/swift-evolution/blob/master/proposals/0129-package-manager-test-naming-conventions.md&gt; could
clarify further.

Here is a summary of what `swift test` currently supports:
* Test targets are only built (and run) when you run `swift test`.
* `swift test --skip-build` skips building of test targets and runs
whatever was last built.
* `swift test -s <test-module>.<test-case>` will run a test case class.
* `swift test -s <test-module>.<test-case>/<test>` will run an individual
test.
* `swift test --list-tests` will list all tests in above format.
* `swift test --parallel` will run tests in parallel.

You can view these options and their help text using `swift test --help`.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Ankit

[Ankit_Agarwal1 (Ankit Agarwal)]

We don't have support for inlining package dependencies in targets (see the
comment in above manifest example). I suggested that we turn this proposal
into adding that feature.

···

On Thu, 26 Jan 2017 at 1:04 PM, David Hart <david@hartbit.com> wrote:

I'm confused. I've followed this from afar as I don't have much SwiftPM
experience, but if that is true, what's the point of the proposal?

> On 26 Jan 2017, at 08:12, Ankit Agarwal via swift-evolution < > swift-evolution@swift.org> wrote:

>

> The test targets are *not* compiled when you run swift build. They're
only complied on invocation of `swift test`. If a test target depends on a
testing library (as in my example), they will *not* be built on invocation
of `swift build`.

[hartbit (David Hart)]

I'm confused. I've followed this from afar as I don't have much SwiftPM experience, but if that is true, what's the point of the proposal?

···

On 26 Jan 2017, at 08:12, Ankit Agarwal via swift-evolution <swift-evolution@swift.org> wrote:

The test targets are *not* compiled when you run swift build. They're only complied on invocation of `swift test`. If a test target depends on a testing library (as in my example), they will *not* be built on invocation of `swift build`.

[codafi (Robert Widmann)]

~Robert Widmann

2017/01/26 4:11、Ankit Agarwal <ankit@ankit.im> のメッセージ:

Three parts for three points:

1) Thanks!

2) I'm not sure generalizing this is particularly useful post-products. The larger point is not to have non-exported/local target dependencies but to make clear the divide between a package and its test suite in the manifest as well as on disk. Local dependencies seem like a separate proposal entirely.

We do have a clear divide between normal modules and test modules.
I am not sure adding a special case test dependencies makes sense right now. We might end up having conditional dependencies later w.r.t platform and other parameters.

Not in practice (with respect to package manifests). In fact, it seems that, given there are separate commands (swift build and swift test), separate directories (Sources and Tests), and separate products, that there's a hole to be filled here by separate handling for test suites in package manifests.

This is all not to discount the features you've brought up as well, but I'm having trouble seeing why a distinction here is such a problem.

···

On Thu, Jan 26, 2017 at 1:55 PM, Robert Widmann via swift-evolution <swift-evolution@swift.org> wrote:

Daniel, do you want to pitch in?

3) Wow, I honestly should have noticed this: I took the time to peruse the source for the help text and still managed to miss it. It has been struck from the gist of this proposal.

~Robert Widmann

2017/01/25 3:07、Ankit Aggarwal <ankit_aggarwal@apple.com> のメッセージ:

On 25-Jan-2017, at 4:02 AM, Robert Widmann via swift-evolution <swift-evolution@swift.org> wrote:

Hello Swift Community,

Harlan Haskins and I have been working on libraries to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces. While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies. To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager. This proposal can also be read in gist form.

Cheers,

~Robert Widmann

Thanks for driving this! It is a very desirable feature which needs proposal work. Comments inline.

Test-Only Package Dependencies and Targets
Proposal: SE-NNNN
Authors: Harlan Haskins, Robert Widmann
Review Manager: TBD
Status: Awaiting review
Introduction

This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation

Soon after SE-0019 identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1], [2], [3]) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.

Proposed solution

We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.

Detailed design

The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.

import PackageDescription

let package = Package(
    name: "Foo",
    targets: [
        Target(name: "Foo")
    ],
    dependencies: [
        .Package(url: "https://github.com/org/ana.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ],
    testDependencies: [
        .Package(url: "https://github.com/org/anism.git&quot;, versions: Version(1,0,0)...Version(1,9,9)),
    ]
)

I think this feature should be called local dependencies (or maybe dev dependencies) because it can be used for tests as well as regular targets.
As an example say you have a networking library package and you want to create an example CLI target which uses a JSON mapper package. You wouldn't want to vend the CLI tool when you act as a dependency to other packages, or include the JSON mapper in your dependencies. Test dependency doesn't sound right in that context.

After the product proposal is implemented, you will be able to control what you vend to your clients, and we thought about adding ability to define dependencies in-line with target dependencies but left it out of the proposal to keep it simpler. Maybe this proposal can add that instead of a separate `testDependencies` property. Consider this manifest:

let package = Package(
    name: "FooNetworking",
    targets: [
        Target(
            name: "FooNetworking"),
        Target(
            name: "FooNetworkingExample",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/BarJSON.git", version: "1.0.0", product: "BarJSON"), // Note: This doesn't actually exists right now.
            ]),
        Target(
            name: "FooNetworkingTests",
            dependencies: [
                "FooNetworking",
                .package(url: "/path/to/Quick.git", version: "1.0.0", product: "Quick"), // Note: This doesn't actually exists right now.
            ]),
   ],
   products: [
       .Library(name: "FooNetworking", targets: ["FooNetworking"]),
   ]
)

This manifest defines three targets and one product.

FooNetworking: The base target and the actual library.
FooNetworkingExample: The example cli tool. It depends on FooNetworking target and BarJSON product from an external package.
FooNetworkingTests: The test target depends on an external package Quick.

Both BarJSON and Quick are local dependencies to this package. If FooNetworkingExample was also vended as a product, BarJSON would automatically become a regular external dependency.

Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.

import PackageDescription

let package = Package(
    name: "SwiftPM",
    targets: [
        Target(
            name: "PackageDescription",
            dependencies: ),

        // MARK: Support libraries

        Target(
            /** Cross-platform access to bare `libc` functionality. */
            name: "libc",
            dependencies: ),
        Target(
            /** “Swifty” POSIX functions from libc */
            name: "POSIX",
            dependencies: ["libc"]),
        Target(
            /** Basic support library */
            name: "Basic",
            dependencies: ["libc", "POSIX"]),

        /* Omitted for Brevity */
     ],
     testTargets: [
        Target(
            name: "BasicPerformanceTests",
            dependencies: ["Basic"]),
       /* Omitted for Brevity */
     ]
)
Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.

We propose the following syntax to execute all tests of all known test targets.

$ swift test
To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:

$ swift test TestModule.TestCase
To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:

$ swift test TestModule.TestCase.exampleTest

We already have all these features. A target is a test target if it has a suffix Tests and is placed under Tests/ directory — we might loosen these restrictions with custom target conventions feature. This proposal could clarify further.

Here is a summary of what `swift test` currently supports:
* Test targets are only built (and run) when you run `swift test`.
* `swift test --skip-build` skips building of test targets and runs whatever was last built.
* `swift test -s <test-module>.<test-case>` will run a test case class.
* `swift test -s <test-module>.<test-case>/<test>` will run an individual test.
* `swift test --list-tests` will list all tests in above format.
* `swift test --parallel` will run tests in parallel.

You can view these options and their help text using `swift test --help`.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Ankit

[Ankit_Agarwal1 (Ankit Agarwal)]

Not in practice (with respect to package manifests). In fact, it seems
that, given there are separate commands (swift build and swift test),
separate directories (Sources and Tests), and separate products, that
there's a hole to be filled here by separate handling for test suites in
package manifests.

FWIW, overriding conventions will potentially allow you control the layout
of the package (when we have that feature).

This is all not to discount the features you've brought up as well, but I'm

having trouble seeing why a distinction here is such a problem.

Could you elaborate on what kind of distinction are you proposing? Is it
separating targets and testTargets?

···

--
Ankit

[codafi (Robert Widmann)]

Not in practice (with respect to package manifests). In fact, it seems that, given there are separate commands (swift build and swift test), separate directories (Sources and Tests), and separate products, that there's a hole to be filled here by separate handling for test suites in package manifests.

FWIW, overriding conventions will potentially allow you control the layout of the package (when we have that feature).

In that case, as long as there are plans for this kind of feature, I can hold off on submitting this proposal until it materializes.

This is all not to discount the features you've brought up as well, but I'm having trouble seeing why a distinction here is such a problem.

Could you elaborate on what kind of distinction are you proposing? Is it separating targets and testTargets?

Yes.

···

On Jan 30, 2017, at 2:38 PM, Ankit Agarwal <ankit@ankit.im> wrote:

--
Ankit

[rballard (Rick Ballard)]

+ swift-build-dev

To take a step back here, there are two problems I see in this space:

1) Building unnecessary test-only stuff.

Say you have a package A which depends on a package B. B might declare a library, a test module, and some auxiliary target that's only needed by the tests. Today, if you check out A and run `swift build`, it will build B's auxiliary target (among other things built). That's unnecessary, and it would be nice to avoid that.

The product definitions proposal will fix this. Once that's implemented, B would publish its library as its only product, and A would declare that it depends on that product. Then, when building A, we would know that we don't need the auxiliary target from B, and wouldn't build it. So I think the products proposal will solve this problem.

2) Cloning unnecessary test-only stuff.

Say you have a package A which depends on a package B, and B has a test module which depends on a package C. Once the products proposal has been implemented and you adopt it, running `swift build` on A will be smart enough not to build the test module from B or any of the targets from C; however, SwiftPM will still clone the C repository when doing dependency resolution, even though it never winds up building its products.

I don't personally see that as a major problem, but it is one that would be nice to address at some point. Ways we could address this include:

– We could add a "testDependencies" property on the package. These dependencies would only be resolved and cloned in situations where one or more test targets from that package need to be built. Some problems with this would be:

  * Since these dependencies would only conditionally resolve, but otherwise would affect dependency resolution as normal, this could lead to some really unexpected emergent behavior. For example, if a package that's only included in the graph via testDependencies happens to depend on another package that's otherwise already in the graph, but with a more restrictive version specification, then whether or not you're building tests would force that other package to re-resolve with a different version.

  * This is a non-generalized solution to a general problem. That is, once we have the products proposal implemented, you could wind up in this situation with no tests involved, but this mechanism wouldn't solve the problem in that case. To give a concrete example, if A depends on B, B vends two products, and only one of them depends on C, it's unnecessary to clone C unless the product that needs it is used by A. A properly general mechanism would let you make that dependency conditional on whether you need it for whatever reason, and not tie it to testing in particular. Another reason you might have this problem is if you have a dependency that only applies on one platform (e.g. linux but not macOS). (Conditional compilation blocks can be used for the platform case, but – once we have a better mechanism – I don't think that they should be used, for reasons that would be best discussed in its own thread).

  * Adding additional properties to the Package should be done carefully. Every new property is new API surface area, and makes our documentation, initializer, etc more complicated. In my opinion, this new property doesn't justify that cost if it's just saving you from initial cloning of the extra packages.

– We could add in-line dependencies for targets, as Ankit suggested. This is a more generalizable mechanism than "testDependencies". That said, it could potentially require you to duplicate your dependency declaration, which is bad. (E.g. if two test targets in your package depend on the same auxiliary package, that dependency would need to be stated twice). It also adds unfortunate complexity.

– We could add a general mechanism for declaring dependencies as conditional, supporting a variety of possible conditions. Some of the SwiftPM developers are in the early discussion stages for a proposal for how to to make various parts of a manifest conditional, as part of a larger conversation on build settings; we'll bring our thoughts to this list once we have something coherent to propose. But if we come up with a general syntax for making things like settings conditional, the same consistent syntax could potentially be applied to dependencies as well.

- - -

Personally, I think that the products proposal solves the first problem nicely, and that we should revisit the second problem as part of an upcoming conversation about conditionals in the manifest.

  - Rick

···

On Jan 30, 2017, at 11:55 AM, Robert Widmann via swift-evolution <swift-evolution@swift.org> wrote:

On Jan 30, 2017, at 2:38 PM, Ankit Agarwal <ankit@ankit.im <mailto:ankit@ankit.im>> wrote:

Not in practice (with respect to package manifests). In fact, it seems that, given there are separate commands (swift build and swift test), separate directories (Sources and Tests), and separate products, that there's a hole to be filled here by separate handling for test suites in package manifests.

FWIW, overriding conventions will potentially allow you control the layout of the package (when we have that feature).

In that case, as long as there are plans for this kind of feature, I can hold off on submitting this proposal until it materializes.