protocol for third-party testing frameworks


(Drew Crawford) #1

Rick Ballard has asked for a proposal for third-party testing frameworks:

Would you mind starting a thread on swift-build-dev@swift.org about your proposal here? I'd love for us to put together a concrete proposal for how we'll support other test frameworks sooner rather than later – it sounds like you and others would like to get that support added soon!

FWIW, the main concern I have with the lightweight "just use a main.swift that runs whatever test code you want" approach is that it doesn't lead to unified test behavior for swift packages. I'd really like package users to be able to rely on being able to run the tests for all their dependencies without needing to manually install other test support first (re: @bppr 's concern), and to get parseable test output in a uniform format, so that tools like a CI dashboard, or a package index that checks package health, can easily be built on top.

Let's discuss on-list what sort of protocol we'd need to define to let you easily get up and running with a different test framework while a) making it possible for swiftPM to get all the components it needs to run your tests automatically and b) providing the test output in a uniform format. Thanks!

It doesn't look like there is a thread yet.

I have some interest in this problem since I have problems such that it isn't clear whether they can be solved within the mandate of XCTest. So I am potentially a test framework author, if only as an insurance policy against fixing XCTest to not suck.

I see some places for test framework to innovate, notably:

Parallelization of tests. Swift language tests use all cores, which is AWESOME. XCTest does not, which is sad. But a general-purpose threading model for all software is not clear. This is a good problem for third-party frameworks to study and try out different things.
Configuring what tests are to be performed. Group them by suites? Burn-in for 10 minutes, regardless of the number of tests? Turn off and on tests in the sourcecode?
Complex pass/fail conditions. XCTest is notoriously bad at failing an entire run because of one performance outlier.
Data reporting. Performance tests might have various unusual metrics etc. that should be reported. For example, I have tests that have metrics measured in "bytes copied".

Based on those requirements, I propose the following API as a starting point of discussion.

protocol TestFramework {
    /**This function is called by SwiftPM to begin all tests
    A TestFramework discovers tests and reports them to the delegate.
    It performs the tests, keeping the delegate updated in realtime.
    Finally, it calls delegate.finish() */
    func begin()
    
    ///A test framework reports back data to SwiftPM (or other data gatherer) here
    var delegate: TestFrameworkDelegate { get set }
}

protocol Test {
    /**A unique name. Reverse-domain notation is encouraged, and XCTest should use these to represent suites. */
    var name : String { get }
    
    /**The current status of the test */
    var status: TestStatus { get }
    
    /**Metrics that are standardized by this specification. These metrics are well-known and interoperable between tools */
    var metrics: [StandardMetric] { get }
    
    /**A test may have other metrics, defined by third parties. */
    var userMetrics: [UserMetric] { get }
}

protocol TestFrameworkDelegate {
    /**Call this function to report an update to the test.
     
The implementation of this fuction should scan the ivars of the Test for a new name / status / metrics and update UX or reporting appropriately.
- warning: The implementation of this function must be threadsafe.
    */
    func updateTest(test: Test)
    
    /**Create a log event associated with the test, which is appended to the event log.
     
SwiftPM uses this information to collate logs in the case that multiple tests are executing simultaneously.
- warning: The implementation of this function must be threadsafe. In the case that the same test logs multiple events simultaneously, the order in which they are appended to the log is undefined.
*/
    func log(test: Test, event: String)
    
    /**Call this function to indicate that a test framework has discovered all tests and they have been updated to Planned status. Novel tests will no longer be discovered.

The use of this method is optional, since some test frameworks may not plan a fixed number of tests.
This function can be used by UIs that want a complete list of tests for some reason (e.g. to put them in a UITableView). */
    func allTestsPlanned()
    
    /**Indicates that testing is complete and no more delegate callbacks will be made.
- parameter status: The status of the overall testing framework. In general this is .Failed if any of the underlying tests failed, but a test framework may choose to apply custom logic to rule on its underlying tests.
     */
    func finish(status: TestStatus)
}

enum TestStatus {
    case Planned ///The test has been discovered
    case Running ///The test is currently executing
    case Passed ///The test finished successfully
    case Failed(ErrorType) ///The test failed for the specified reason
    case Skipped(String) ///The test will not execute. The string contains more information (e.g. wrong suite, ran out of time, etc.)
    case Indeterminate ///The test executed, and ended in a state that is neither a pass nor a failure
    case Orange ///The test ended in a state that may warrant further investigation but is not automatically ruled a failure. https://wiki.mozilla.org/Auto-tools/Projects/OrangeFactor
}

enum StandardMetric {
    /// a list of test executions, with the number of seconds of each execution. In XCTest's situation, this is 10 doubles, one for each run.
    case Runtime([Double])
    
    ///A unique string that identifies the "performance class" of the machine, such as "iPhone 6s". Equal strings mean test results are from a machine in the same performance class and so are directly comparable. An unequal string means they are not comparable.
    case PerformanceClass(String)

}

protocol UserMetric {}

There is an extremely straightforward implementation of this API for XCTest. Simply loop over the tests and update their status.

However this API is much more powerful than XCTest and would allow considerable experimentation for third-party frameworks on matters of parallelization, test discovery, pass/fail logic, etc., and so I think it is more agnostic and free of XCTest assumptions, which Kyle Fuller expressed concerns about, and I share.

Meanwhile the reporting APIs here makes sure we have parseable tests in a uniform format that UX tools can work with as Rick Ballard was concerned about.

I certainly think there are more ideas to consider here, but this is a reasonable first draft to have a conversation about.


(Max Howell) #2

I have some interest in this problem since I have problems such that it isn't clear whether they can be solved within the mandate of XCTest. So I am potentially a test framework author, if only as an insurance policy against fixing XCTest to not suck.

I see some places for test framework to innovate, notably:

Parallelization of tests. Swift language tests use all cores, which is AWESOME. XCTest does not, which is sad. But a general-purpose threading model for all software is not clear. This is a good problem for third-party frameworks to study and try out different things.

Sounds great.

Configuring what tests are to be performed. Group them by suites? Burn-in for 10 minutes, regardless of the number of tests? Turn off and on tests in the source code?

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

Complex pass/fail conditions. XCTest is notoriously bad at failing an entire run because of one performance outlier.

Sounds good. Would fit into a “protocol” design easily enough.

Data reporting. Performance tests might have various unusual metrics etc. that should be reported. For example, I have tests that have metrics measured in "bytes copied”.

Sounds good.

Based on those requirements, I propose the following API as a starting point of discussion.

protocol TestFramework {
    /**This function is called by SwiftPM to begin all tests
    A TestFramework discovers tests and reports them to the delegate.
    It performs the tests, keeping the delegate updated in realtime.
    Finally, it calls delegate.finish() */
    func begin()
    
    ///A test framework reports back data to SwiftPM (or other data gatherer) here
    var delegate: TestFrameworkDelegate { get set }
}

protocol Test {
    /**A unique name. Reverse-domain notation is encouraged, and XCTest should use these to represent suites. */
    var name : String { get }
    
    /**The current status of the test */
    var status: TestStatus { get }
    
    /**Metrics that are standardized by this specification. These metrics are well-known and interoperable between tools */
    var metrics: [StandardMetric] { get }
    
    /**A test may have other metrics, defined by third parties. */
    var userMetrics: [UserMetric] { get }
}

protocol TestFrameworkDelegate {
    /**Call this function to report an update to the test.
     
The implementation of this fuction should scan the ivars of the Test for a new name / status / metrics and update UX or reporting appropriately.
- warning: The implementation of this function must be threadsafe.
    */
    func updateTest(test: Test)
    
    /**Create a log event associated with the test, which is appended to the event log.
     
SwiftPM uses this information to collate logs in the case that multiple tests are executing simultaneously.
- warning: The implementation of this function must be threadsafe. In the case that the same test logs multiple events simultaneously, the order in which they are appended to the log is undefined.
*/
    func log(test: Test, event: String)
    
    /**Call this function to indicate that a test framework has discovered all tests and they have been updated to Planned status. Novel tests will no longer be discovered.

The use of this method is optional, since some test frameworks may not plan a fixed number of tests.
This function can be used by UIs that want a complete list of tests for some reason (e.g. to put them in a UITableView). */
    func allTestsPlanned()
    
    /**Indicates that testing is complete and no more delegate callbacks will be made.
- parameter status: The status of the overall testing framework. In general this is .Failed if any of the underlying tests failed, but a test framework may choose to apply custom logic to rule on its underlying tests.
     */
    func finish(status: TestStatus)
}

enum TestStatus {
    case Planned ///The test has been discovered
    case Running ///The test is currently executing
    case Passed ///The test finished successfully
    case Failed(ErrorType) ///The test failed for the specified reason
    case Skipped(String) ///The test will not execute. The string contains more information (e.g. wrong suite, ran out of time, etc.)
    case Indeterminate ///The test executed, and ended in a state that is neither a pass nor a failure
    case Orange ///The test ended in a state that may warrant further investigation but is not automatically ruled a failure. https://wiki.mozilla.org/Auto-tools/Projects/OrangeFactor
}

enum StandardMetric {
    /// a list of test executions, with the number of seconds of each execution. In XCTest's situation, this is 10 doubles, one for each run.
    case Runtime([Double])
    
    ///A unique string that identifies the "performance class" of the machine, such as "iPhone 6s". Equal strings mean test results are from a machine in the same performance class and so are directly comparable. An unequal string means they are not comparable.
    case PerformanceClass(String)

}

protocol UserMetric {}

There is an extremely straightforward implementation of this API for XCTest. Simply loop over the tests and update their status.

My initial glance thinks this could be simpler, but maybe not. I’d have to give more time to this and I don’t right now have it. Certainly I think it is a great start.

However this API is much more powerful than XCTest and would allow considerable experimentation for third-party frameworks on matters of parallelization, test discovery, pass/fail logic, etc., and so I think it is more agnostic and free of XCTest assumptions, which Kyle Fuller expressed concerns about, and I share.

Meanwhile the reporting APIs here makes sure we have parseable tests in a uniform format that UX tools can work with as Rick Ballard was concerned about.

I certainly think there are more ideas to consider here, but this is a reasonable first draft to have a conversation about.

I’d like us to try and map existing popular testing styles onto this protocol before we right up the full proposal.

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.


(Drew Crawford) #3

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

Well I think some kind of CLI support (like perhaps passing the whole CLI to the testing framework?) makes sense.

Specifically one problem that motivates flexibility here is the per-commit continuous integration. If my test suite is 15 minutes, and I'm going to run that in a few different configurations (32 vs 64-bit, or say we support iOS someday and I'm cross-compiling for N simulators or hardware devices, etc.) I'm in for a bad time trying to do all that every commit.

There are a lot of "solutions" here–parallelization, random sampling, running an abbreviated test suite ordinarily and the full test suite every 10th commit–a test suite that tries to maximize code coverage per unit time–there are all kinds of ways to look at this problem that may make sense to somebody.

I'm simply pushing that decision out to individual test frameworks to study. XCTest thinks you should have test suites that are human curated. That's one way to skin the cat. Let's give another test framework the flexibility to think differently on this problem.

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.

Thanks for picking this up. Getting basic testing support in place is really important, and IMO the current proposal is really strong. I am focusing on some of these other areas not as a distraction but because I think the basic testing ideas are strong enough already that they don't need my help :slight_smile:

···

On Jan 4, 2016, at 4:00 PM, Max Howell <max.howell@apple.com> wrote:


(Max Howell) #4

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

Well I think some kind of CLI support (like perhaps passing the whole CLI to the testing framework?) makes sense.

That’s our current thinking.

Specifically one problem that motivates flexibility here is the per-commit continuous integration. If my test suite is 15 minutes, and I'm going to run that in a few different configurations (32 vs 64-bit, or say we support iOS someday and I'm cross-compiling for N simulators or hardware devices, etc.) I'm in for a bad time trying to do all that every commit.

There are a lot of "solutions" here–parallelization, random sampling, running an abbreviated test suite ordinarily and the full test suite every 10th commit–a test suite that tries to maximize code coverage per unit time–there are all kinds of ways to look at this problem that may make sense to somebody.

I'm simply pushing that decision out to individual test frameworks to study. XCTest thinks you should have test suites that are human curated. That's one way to skin the cat. Let's give another test framework the flexibility to think differently on this problem.

Sounds super interesting. We should make sure the protocol we design can do these things.

···

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.

Thanks for picking this up. Getting basic testing support in place is really important, and IMO the current proposal is really strong. I am focusing on some of these other areas not as a distraction but because I think the basic testing ideas are strong enough already that they don't need my help :slight_smile:


(Max Howell) #5

To follow up here, I’d like to put this particular proposal on the back burner for a couple weeks until we have the initial testing infrastructure in place.

We should keep this proposal in mind as we build the testing infrastructure, as this protocol is in my mind at least, the more important of the two.

···

On Jan 6, 2016, at 11:58 AM, Max Howell via swift-build-dev <swift-build-dev@swift.org> wrote:

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

Well I think some kind of CLI support (like perhaps passing the whole CLI to the testing framework?) makes sense.

That’s our current thinking.

Specifically one problem that motivates flexibility here is the per-commit continuous integration. If my test suite is 15 minutes, and I'm going to run that in a few different configurations (32 vs 64-bit, or say we support iOS someday and I'm cross-compiling for N simulators or hardware devices, etc.) I'm in for a bad time trying to do all that every commit.

There are a lot of "solutions" here–parallelization, random sampling, running an abbreviated test suite ordinarily and the full test suite every 10th commit–a test suite that tries to maximize code coverage per unit time–there are all kinds of ways to look at this problem that may make sense to somebody.

I'm simply pushing that decision out to individual test frameworks to study. XCTest thinks you should have test suites that are human curated. That's one way to skin the cat. Let's give another test framework the flexibility to think differently on this problem.

Sounds super interesting. We should make sure the protocol we design can do these things.

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.

Thanks for picking this up. Getting basic testing support in place is really important, and IMO the current proposal is really strong. I am focusing on some of these other areas not as a distraction but because I think the basic testing ideas are strong enough already that they don't need my help :slight_smile:

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Max Howell) #6

It’s time to resurrect this proposal.

By all means let’s talk a bit more here, but I hope Drew will submit a proposal, or if he doesn’t have the time I can write it up, co-authored and we’ll submit to evolution.

···

On Jan 12, 2016, at 5:02 PM, Max Howell <max.howell@apple.com> wrote:

To follow up here, I’d like to put this particular proposal on the back burner for a couple weeks until we have the initial testing infrastructure in place.

We should keep this proposal in mind as we build the testing infrastructure, as this protocol is in my mind at least, the more important of the two.

On Jan 6, 2016, at 11:58 AM, Max Howell via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

Well I think some kind of CLI support (like perhaps passing the whole CLI to the testing framework?) makes sense.

That’s our current thinking.

Specifically one problem that motivates flexibility here is the per-commit continuous integration. If my test suite is 15 minutes, and I'm going to run that in a few different configurations (32 vs 64-bit, or say we support iOS someday and I'm cross-compiling for N simulators or hardware devices, etc.) I'm in for a bad time trying to do all that every commit.

There are a lot of "solutions" here–parallelization, random sampling, running an abbreviated test suite ordinarily and the full test suite every 10th commit–a test suite that tries to maximize code coverage per unit time–there are all kinds of ways to look at this problem that may make sense to somebody.

I'm simply pushing that decision out to individual test frameworks to study. XCTest thinks you should have test suites that are human curated. That's one way to skin the cat. Let's give another test framework the flexibility to think differently on this problem.

Sounds super interesting. We should make sure the protocol we design can do these things.

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.

Thanks for picking this up. Getting basic testing support in place is really important, and IMO the current proposal is really strong. I am focusing on some of these other areas not as a distraction but because I think the basic testing ideas are strong enough already that they don't need my help :slight_smile:

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Drew Crawford) #7

Please take the lead on this.

For background. We required a solution around the time I was working on this originally, and given that upstream wasn't ready (which is understandable) we decided to go alone, and we're now committed to our solution. So from our POV the problem has been solved.

···

On Mar 16, 2016, at 8:12 PM, Max Howell <max.howell@apple.com> wrote:

It’s time to resurrect this proposal.

By all means let’s talk a bit more here, but I hope Drew will submit a proposal, or if he doesn’t have the time I can write it up, co-authored and we’ll submit to evolution.

On Jan 12, 2016, at 5:02 PM, Max Howell <max.howell@apple.com <mailto:max.howell@apple.com>> wrote:

To follow up here, I’d like to put this particular proposal on the back burner for a couple weeks until we have the initial testing infrastructure in place.

We should keep this proposal in mind as we build the testing infrastructure, as this protocol is in my mind at least, the more important of the two.

On Jan 6, 2016, at 11:58 AM, Max Howell via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

Well I think some kind of CLI support (like perhaps passing the whole CLI to the testing framework?) makes sense.

That’s our current thinking.

Specifically one problem that motivates flexibility here is the per-commit continuous integration. If my test suite is 15 minutes, and I'm going to run that in a few different configurations (32 vs 64-bit, or say we support iOS someday and I'm cross-compiling for N simulators or hardware devices, etc.) I'm in for a bad time trying to do all that every commit.

There are a lot of "solutions" here–parallelization, random sampling, running an abbreviated test suite ordinarily and the full test suite every 10th commit–a test suite that tries to maximize code coverage per unit time–there are all kinds of ways to look at this problem that may make sense to somebody.

I'm simply pushing that decision out to individual test frameworks to study. XCTest thinks you should have test suites that are human curated. That's one way to skin the cat. Let's give another test framework the flexibility to think differently on this problem.

Sounds super interesting. We should make sure the protocol we design can do these things.

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.

Thanks for picking this up. Getting basic testing support in place is really important, and IMO the current proposal is really strong. I am focusing on some of these other areas not as a distraction but because I think the basic testing ideas are strong enough already that they don't need my help :slight_smile:

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev


#8

Hey folks,

I think from my perspective SwiftPM could solve this problem by simply allowing me to define and build targets that aren't going to be included in the distributed package. Maybe a field like `testTargets` or something that allows the author downstream to link `testDependencies`?

SwiftPM is great as a package manager, but for a lot of my most common development workflows, it falls short because of this limitation -- I can't use it to build things that I don't mean to distribute as part of the package.

···

--
Brian Pratt

On March 16, 2016 at 8:39:02 PM, Drew Crawford via swift-build-dev (swift-build-dev@swift.org) wrote:

Please take the lead on this.

For background. We required a solution around the time I was working on this originally, and given that upstream wasn't ready (which is understandable) we decided to go alone, and we're now committed to our solution. So from our POV the problem has been solved.

On Mar 16, 2016, at 8:12 PM, Max Howell <max.howell@apple.com> wrote:

It’s time to resurrect this proposal.

By all means let’s talk a bit more here, but I hope Drew will submit a proposal, or if he doesn’t have the time I can write it up, co-authored and we’ll submit to evolution.

On Jan 12, 2016, at 5:02 PM, Max Howell <max.howell@apple.com> wrote:

To follow up here, I’d like to put this particular proposal on the back burner for a couple weeks until we have the initial testing infrastructure in place.

We should keep this proposal in mind as we build the testing infrastructure, as this protocol is in my mind at least, the more important of the two.

On Jan 6, 2016, at 11:58 AM, Max Howell via swift-build-dev <swift-build-dev@swift.org> wrote:

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

Well I think some kind of CLI support (like perhaps passing the whole CLI to the testing framework?) makes sense.

That’s our current thinking.

Specifically one problem that motivates flexibility here is the per-commit continuous integration. If my test suite is 15 minutes, and I'm going to run that in a few different configurations (32 vs 64-bit, or say we support iOS someday and I'm cross-compiling for N simulators or hardware devices, etc.) I'm in for a bad time trying to do all that every commit.

There are a lot of "solutions" here–parallelization, random sampling, running an abbreviated test suite ordinarily and the full test suite every 10th commit–a test suite that tries to maximize code coverage per unit time–there are all kinds of ways to look at this problem that may make sense to somebody.

I'm simply pushing that decision out to individual test frameworks to study. XCTest thinks you should have test suites that are human curated. That's one way to skin the cat. Let's give another test framework the flexibility to think differently on this problem.

Sounds super interesting. We should make sure the protocol we design can do these things.

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.

Thanks for picking this up. Getting basic testing support in place is really important, and IMO the current proposal is really strong. I am focusing on some of these other areas not as a distraction but because I think the basic testing ideas are strong enough already that they don't need my help :slight_smile:

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Max Howell) #9

Hey folks,

I think from my perspective SwiftPM could solve this problem by simply allowing me to define and build targets that aren't going to be included in the distributed package. Maybe a field like `testTargets` or something that allows the author downstream to link `testDependencies`?

SwiftPM is great as a package manager, but for a lot of my most common development workflows, it falls short because of this limitation -- I can't use it to build things that I don't mean to distribute as part of the package.

Can you refine your request a little?

I don’t quite understand because: everything is distributed in the package, because the package is sources.


#10

Ah! Sorry, that word choice ("distributed") was probably misleading. I'm definitely not concerned about the sources being distributed -- moreso about being able to have finer-grained control over what gets built and what doesn't.

Right now I have two choices:
- define a target (and have it be built when someone lists my package as a dependency)
- exclude the target, and never have it be built at all by SwiftPM

I'd like a middle-ground where I can define targets that are *only* built in the development context, link them with test dependencies, and then build only targets that I choose to make available when someone depends on the package.

···

--
Brian Pratt

On March 17, 2016 at 11:53:01 AM, Max Howell (max.howell@apple.com) wrote:

Hey folks,

I think from my perspective SwiftPM could solve this problem by simply allowing me to define and build targets that aren't going to be included in the distributed package. Maybe a field like `testTargets` or something that allows the author downstream to link `testDependencies`?

SwiftPM is great as a package manager, but for a lot of my most common development workflows, it falls short because of this limitation -- I can't use it to build things that I don't mean to distribute as part of the package.

Can you refine your request a little?

I don’t quite understand because: everything is distributed in the package, because the package is sources.


#11

To get a bit more concrete:

let package = Package(
name: "MyCoolPackage",
// etc
targets: [
Target(name: "MyCoolPackageTarget")
],
testTargets: [
Target(name: "MyCoolTestingTarget", dependencies: [.Dependency("one-of-my-test-dependencies")])
])

When someone else uses my package, MyCoolTestingTarget would not be built (although its sources would be present), but MyCoolPackageTarget would be -- if that makes sense. In this way, we'd get ad-hoc support for third-party testing frameworks and also improve the Package Manager to be able to be more useful as a build tool for a whole dev environment as opposed to just a build tool for creating and consuming packages.

The only drawback I can think of that falls out of this approach is that 3rd-party testing frameworks wouldn't support the `swift test` command-line invocation, but I'd personally rank that as a lower priority for me than just being able to get more granular control over what gets built in a local context vs what gets built when I'm a dependency of something else.

Hopefully that clears my idea up a bit more.

···

--
Brian Pratt

On March 17, 2016 at 10:05:23 AM, Brian Pratt (brian@pratt.io) wrote:

Hey folks,

I think from my perspective SwiftPM could solve this problem by simply allowing me to define and build targets that aren't going to be included in the distributed package. Maybe a field like `testTargets` or something that allows the author downstream to link `testDependencies`?

SwiftPM is great as a package manager, but for a lot of my most common development workflows, it falls short because of this limitation -- I can't use it to build things that I don't mean to distribute as part of the package.

--
Brian Pratt

On March 16, 2016 at 8:39:02 PM, Drew Crawford via swift-build-dev (swift-build-dev@swift.org) wrote:

Please take the lead on this.

For background. We required a solution around the time I was working on this originally, and given that upstream wasn't ready (which is understandable) we decided to go alone, and we're now committed to our solution. So from our POV the problem has been solved.

On Mar 16, 2016, at 8:12 PM, Max Howell <max.howell@apple.com> wrote:

It’s time to resurrect this proposal.

By all means let’s talk a bit more here, but I hope Drew will submit a proposal, or if he doesn’t have the time I can write it up, co-authored and we’ll submit to evolution.

On Jan 12, 2016, at 5:02 PM, Max Howell <max.howell@apple.com> wrote:

To follow up here, I’d like to put this particular proposal on the back burner for a couple weeks until we have the initial testing infrastructure in place.

We should keep this proposal in mind as we build the testing infrastructure, as this protocol is in my mind at least, the more important of the two.

On Jan 6, 2016, at 11:58 AM, Max Howell via swift-build-dev <swift-build-dev@swift.org> wrote:

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

Well I think some kind of CLI support (like perhaps passing the whole CLI to the testing framework?) makes sense.

That’s our current thinking.

Specifically one problem that motivates flexibility here is the per-commit continuous integration. If my test suite is 15 minutes, and I'm going to run that in a few different configurations (32 vs 64-bit, or say we support iOS someday and I'm cross-compiling for N simulators or hardware devices, etc.) I'm in for a bad time trying to do all that every commit.

There are a lot of "solutions" here–parallelization, random sampling, running an abbreviated test suite ordinarily and the full test suite every 10th commit–a test suite that tries to maximize code coverage per unit time–there are all kinds of ways to look at this problem that may make sense to somebody.

I'm simply pushing that decision out to individual test frameworks to study. XCTest thinks you should have test suites that are human curated. That's one way to skin the cat. Let's give another test framework the flexibility to think differently on this problem.

Sounds super interesting. We should make sure the protocol we design can do these things.

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.

Thanks for picking this up. Getting basic testing support in place is really important, and IMO the current proposal is really strong. I am focusing on some of these other areas not as a distraction but because I think the basic testing ideas are strong enough already that they don't need my help :slight_smile:

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Max Howell) #12

Ah! Sorry, that word choice ("distributed") was probably misleading. I'm definitely not concerned about the sources being distributed -- moreso about being able to have finer-grained control over what gets built and what doesn't.

Right now I have two choices:
- define a target (and have it be built when someone lists my package as a dependency)
- exclude the target, and never have it be built at all by SwiftPM

I'd like a middle-ground where I can define targets that are *only* built in the development context, link them with test dependencies, and then build only targets that I choose to make available when someone depends on the package.

Yes please open a ticket for this, we’ve discussed the need for this in the past and should schedule its development.


#13

Here's the ticket: https://bugs.swift.org/browse/SR-968
Feel free to add comments!

···

--
Brian Pratt

On March 17, 2016 at 12:45:08 PM, Max Howell (max.howell@apple.com) wrote:

Ah! Sorry, that word choice ("distributed") was probably misleading. I'm definitely not concerned about the sources being distributed -- moreso about being able to have finer-grained control over what gets built and what doesn't.

Right now I have two choices:
- define a target (and have it be built when someone lists my package as a dependency)
- exclude the target, and never have it be built at all by SwiftPM

I'd like a middle-ground where I can define targets that are *only* built in the development context, link them with test dependencies, and then build only targets that I choose to make available when someone depends on the package.

Yes please open a ticket for this, we’ve discussed the need for this in the past and should schedule its development.


(David Owens II) #14

Resending as Outlook apparently has trouble including the list instead of just the recipients…

···

---

Honestly, I’m still confused why this is the approach that is desired to be taken. Why not just provide a way to invoke different runners from within SwiftPM? Like Drew mentioned, this approach still isn’t going to solve the problems we had. We simply provided a mechanism to run various things. This made it extremely trivial to integrate the running of any type of test framework we wanted with no additional work required by anyone.

With this approach, it is trivial for us to support XCTest runners, shell script tests, tests that required files to be generated first, etc…

I’ve also been doing some experiments with different ways to write tests, how would I go about doing that with SwiftPM?

class Sum {
    func add(x: Int, _ y: Int) -> Int { return x + y }
}

func __test_sum_add_checkin() throws {
    let sum = Sum()
    assert(sum.add(4, 5) == 10, "Math is hard!")
    assert(sum.add(-3, 3) == 0)
}

Then with our build files:

(package

  :name "IntegratedUnitTests"

  :tasks {

    :build {

      :tool "atllbuild"

      :sources ["Sum.swift"]

      :name "math"

      :output-type "static-library"

      :publish-product true

      :compile-options ["-enable-testing"]

    }

    :test {

      :dependencies ["generate-test-file"]

      :tool "atllbuild"

      :sources ["sum_test.swift"]

      :name "sum_test"

      :output-type "executable"

      :publish-product true

      :link-with ["math.a"]

    }

    :generate-test-file {

      :dependencies ["build"]

      :tool "shell"

      :script "echo '@testable import math' > sum_test.swift && xcrun -sdk macosx swiftc -print-ast Sum.swift | grep __test | sed 's/internal func/try/g' | sed 's/throws//g' >> sum_test.swift"

    }

    :run {

      :dependencies ["test"]

      :tool "shell"

      :script "./bin/sum_test"

    }

  }

)

More details: http://owensd.io/blog/tooling-around---testing-in-swift/

It would seem the plan is that I’d then have to go write some additional wrapper if I simply wanted it to work in SwiftPM. That’s not a problem that scales very well.

-David

From: swift-build-dev-bounces@swift.org [mailto:swift-build-dev-bounces@swift.org] On Behalf Of Brian Pratt via swift-build-dev
Sent: Thursday, March 17, 2016 10:32 AM
To: Drew Crawford <drew@sealedabstract.com>; Max Howell <max.howell@apple.com>; Drew Crawford via swift-build-dev <swift-build-dev@swift.org>
Subject: Re: [swift-build-dev] protocol for third-party testing frameworks

To get a bit more concrete:

let package = Package(

  name: "MyCoolPackage",

  // etc

  targets: [

    Target(name: "MyCoolPackageTarget")

],

testTargets: [

    Target(name: "MyCoolTestingTarget", dependencies: [.Dependency("one-of-my-test-dependencies")])

  ])

When someone else uses my package, MyCoolTestingTarget would not be built (although its sources would be present), but MyCoolPackageTarget would be -- if that makes sense. In this way, we'd get ad-hoc support for third-party testing frameworks and also improve the Package Manager to be able to be more useful as a build tool for a whole dev environment as opposed to just a build tool for creating and consuming packages.

The only drawback I can think of that falls out of this approach is that 3rd-party testing frameworks wouldn't support the `swift test` command-line invocation, but I'd personally rank that as a lower priority for me than just being able to get more granular control over what gets built in a local context vs what gets built when I'm a dependency of something else.

Hopefully that clears my idea up a bit more.

--

Brian Pratt

On March 17, 2016 at 10:05:23 AM, Brian Pratt (brian@pratt.io <mailto:brian@pratt.io> ) wrote:

Hey folks,

I think from my perspective SwiftPM could solve this problem by simply allowing me to define and build targets that aren't going to be included in the distributed package. Maybe a field like `testTargets` or something that allows the author downstream to link `testDependencies`?

SwiftPM is great as a package manager, but for a lot of my most common development workflows, it falls short because of this limitation -- I can't use it to build things that I don't mean to distribute as part of the package.

--
Brian Pratt

On March 16, 2016 at 8:39:02 PM, Drew Crawford via swift-build-dev (swift-build-dev@swift.org <mailto:swift-build-dev@swift.org> ) wrote:

Please take the lead on this.

For background. We required a solution around the time I was working on this originally, and given that upstream wasn't ready (which is understandable) we decided to go alone, and we're now committed to our solution. So from our POV the problem has been solved.

On Mar 16, 2016, at 8:12 PM, Max Howell <max.howell@apple.com <mailto:max.howell@apple.com> > wrote:

It’s time to resurrect this proposal.

By all means let’s talk a bit more here, but I hope Drew will submit a proposal, or if he doesn’t have the time I can write it up, co-authored and we’ll submit to evolution.

On Jan 12, 2016, at 5:02 PM, Max Howell <max.howell@apple.com <mailto:max.howell@apple.com> > wrote:

To follow up here, I’d like to put this particular proposal on the back burner for a couple weeks until we have the initial testing infrastructure in place.

We should keep this proposal in mind as we build the testing infrastructure, as this protocol is in my mind at least, the more important of the two.

On Jan 6, 2016, at 11:58 AM, Max Howell via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org> > wrote:

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

Well I think some kind of CLI support (like perhaps passing the whole CLI to the testing framework?) makes sense.

That’s our current thinking.

Specifically one problem that motivates flexibility here is the per-commit continuous integration. If my test suite is 15 minutes, and I'm going to run that in a few different configurations (32 vs 64-bit, or say we support iOS someday and I'm cross-compiling for N simulators or hardware devices, etc.) I'm in for a bad time trying to do all that every commit.

There are a lot of "solutions" here–parallelization, random sampling, running an abbreviated test suite ordinarily and the full test suite every 10th commit–a test suite that tries to maximize code coverage per unit time–there are all kinds of ways to look at this problem that may make sense to somebody.

I'm simply pushing that decision out to individual test frameworks to study. XCTest thinks you should have test suites that are human curated. That's one way to skin the cat. Let's give another test framework the flexibility to think differently on this problem.

Sounds super interesting. We should make sure the protocol we design can do these things.

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.

Thanks for picking this up. Getting basic testing support in place is really important, and IMO the current proposal is really strong. I am focusing on some of these other areas not as a distraction but because I think the basic testing ideas are strong enough already that they don't need my help :slight_smile:

  <https://u2002410.ct.sendgrid.net/wf/open?upn=cbMbdH1LnH6O78Q-2BHw3jtU8ikibH470Fh9meAJKpwSqpJeTKnMxtBKMcwgO6eNd7RiBAIn9BZYPOs2eHgzutG3iFkECUBpSh2YJRbpFs5gYIFdbbC8ItMNA7gp-2BqGK4cWwBss4mY6Uv8sMAwKkzb1-2B1rswus8P2lPVMLer7xFPfSEQS6c96lmiGJyiZhqf-2F-2BJOnpV-2BhXXERxFHP94bzeH6uMWF-2B662RYrOYdPl-2FmMCE-3D> _______________________________________________
swift-build-dev mailing list
<mailto:swift-build-dev@swift.org> swift-build-dev@swift.org
<https://lists.swift.org/mailman/listinfo/swift-build-dev> https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev