Canvasing Opinions About Testing


(Joshua Scott Emmons) #1

The proposal process for swiftpm is the same as that for Swift; we will submit a proper document later today (if possible) and everyone will be able to submit feedback. It is important to note that our proposal is just as valid as yours, so if you dislike what you read you should feel free to submit your own via the same process.

I haven't seen this go up on github.com/apple/swift-package-manager yet — maybe I'm looking in the wrong place? But to kick off some conversation, my own naive proposal would be something like:

* By convention, tests live in "./Tests" relative to "Package.swift" (though this location could be explicitly set in the manifest, just like ./Sources)

* Running `swift test` from the "Package.swift" directory would first build the package libraries, then build everything in "./Tests" linked with those libs.

* In the fullness of time, the build system would find all subclasses of `XCTestCase`, running each method of said subclass beginning with `test…`

* Until we have the necessary introspection capabilities, though, we can create a "main.swift" file that creates `XCTMain` with the list of tests to run. Once introspection is up to par, we should be able to delete "main.swift" and everything will just work.

* Subdirectories under "/Tests" could specify different "test suites". Each would have its own "main.swift", to aide in organization of tests for large packages. Test suites could be run individually via argument to the command line.

I'm sure there's tons that's being overlooked here. Comments?

···

--
Cheers,
-jemmons


(Daniel Dunbar) #2

As Marc noted, can you take a look at:
  https://github.com/apple/swift-evolution/pull/51/files
and see if there are specific changes you would propose? It matches most of your proposals, I think.

The proposal process for swiftpm is the same as that for Swift; we will submit a proper document later today (if possible) and everyone will be able to submit feedback. It is important to note that our proposal is just as valid as yours, so if you dislike what you read you should feel free to submit your own via the same process.

I haven't seen this go up on github.com/apple/swift-package-manager yet — maybe I'm looking in the wrong place? But to kick off some conversation, my own naive proposal would be something like:

* By convention, tests live in "./Tests" relative to "Package.swift" (though this location could be explicitly set in the manifest, just like ./Sources)

* Running `swift test` from the "Package.swift" directory would first build the package libraries, then build everything in "./Tests" linked with those libs.

Our intention is to build everything together, and not impose any artificial serialization point.

* In the fullness of time, the build system would find all subclasses of `XCTestCase`, running each method of said subclass beginning with `test…`

* Until we have the necessary introspection capabilities, though, we can create a "main.swift" file that creates `XCTMain` with the list of tests to run. Once introspection is up to par, we should be able to delete "main.swift" and everything will just work.

As far as running the tests goes, for now the plan is to just use XCTest's facilities for running all the tests in a "test target" (bundle on OS X, executable on Linux). Evolving those facilities should be done under the auspices of XCTest.

- Daniel

···

On Dec 13, 2015, at 12:26 PM, Joshua Scott Emmons via swift-build-dev <swift-build-dev@swift.org> wrote:

* Subdirectories under "/Tests" could specify different "test suites". Each would have its own "main.swift", to aide in organization of tests for large packages. Test suites could be run individually via argument to the command line.

I'm sure there's tons that's being overlooked here. Comments?

--
Cheers,
-jemmons

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Marc Knaup) #3

Do you refer to this proposal?
https://github.com/apple/swift-evolution/pull/51/files

···

On Sun, Dec 13, 2015 at 9:26 PM, Joshua Scott Emmons via swift-build-dev < swift-build-dev@swift.org> wrote:

> The proposal process for swiftpm is the same as that for Swift; we will
submit a proper document later today (if possible) and everyone will be
able to submit feedback. It is important to note that our proposal is just
as valid as yours, so if you dislike what you read you should feel free to
submit your own via the same process.

I haven't seen this go up on github.com/apple/swift-package-manager yet —
maybe I'm looking in the wrong place? But to kick off some conversation, my
own naive proposal would be something like:

* By convention, tests live in "./Tests" relative to "Package.swift"
(though this location could be explicitly set in the manifest, just like
./Sources)

* Running `swift test` from the "Package.swift" directory would first
build the package libraries, then build everything in "./Tests" linked with
those libs.

* In the fullness of time, the build system would find all subclasses of
`XCTestCase`, running each method of said subclass beginning with `test…`

* Until we have the necessary introspection capabilities, though, we can
create a "main.swift" file that creates `XCTMain` with the list of tests to
run. Once introspection is up to par, we should be able to delete
"main.swift" and everything will just work.

* Subdirectories under "/Tests" could specify different "test suites".
Each would have its own "main.swift", to aide in organization of tests for
large packages. Test suites could be run individually via argument to the
command line.

I'm sure there's tons that's being overlooked here. Comments?

--
Cheers,
-jemmons

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev