[xctest] Who tests the tests?

Development Core Libraries
1

Hello! This is in reference to
https://github.com/apple/swift-corelibs-xctest/pull/3\. That pull request
contains a commit that attempts to refactor XCTest such that it is more
"unit-testable".

To do so, it gives XCTMain an additional parameter: a list of objects
conforming to the Reporter protocol. I think of this as a minimal, corelibs
equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because
Reporter only defines Reporter.log(), whereas XCTestObserver has one method
for each kind of test event (started, failed, finished, etc.).

These reporters are, for now, storied in a global array. In the future, I'd
like to discuss moving XCTest to a model in which all tests are
(optionally) run in sub-processes, each of which may (optionally) run in
parallel. This global array most certainly won't work for such a change,
but for now, I simply want to have regression tests on the project. It's
hard to send pull requests without knowing everything still works!

Besides this approach, which modifies XCTest in order to test it, it may be
more prudent to add tests *without* changing XCTest at all. To do so, I
could add tests that run programs that call XCTMain(), then verify what's
printed to stdout. This could be done using a Python script (which would go
well with the build script, also in Python).

I'd love input on which of these approaches sounds more viable. Other ideas
are also, of course, welcome!

- Brian Gesiak

2

Hi Brian,

···

On Dec 3, 2015, at 3:45 PM, Brian Gesiak <modocache@gmail.com> wrote:

Hello! This is in reference to https://github.com/apple/swift-corelibs-xctest/pull/3\. That pull request contains a commit that attempts to refactor XCTest such that it is more "unit-testable".

To do so, it gives XCTMain an additional parameter: a list of objects conforming to the Reporter protocol. I think of this as a minimal, corelibs equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because Reporter only defines Reporter.log(), whereas XCTestObserver has one method for each kind of test event (started, failed, finished, etc.).

These reporters are, for now, storied in a global array. In the future, I'd like to discuss moving XCTest to a model in which all tests are (optionally) run in sub-processes, each of which may (optionally) run in parallel. This global array most certainly won't work for such a change, but for now, I simply want to have regression tests on the project. It's hard to send pull requests without knowing everything still works!

Besides this approach, which modifies XCTest in order to test it, it may be more prudent to add tests *without* changing XCTest at all. To do so, I could add tests that run programs that call XCTMain(), then verify what's printed to stdout. This could be done using a Python script (which would go well with the build script, also in Python).

It should be possible to use an out-of-process model that still uses XCTest itself to run the tests. For example, in the package manager we have some tests which spawn the package manager in order to test the end-to-end behavior. Ideally we would only do this for a small number of tests that really need this level of testing, and use unit testing for the rest.

To be more concrete, what I am proposing here is that we have test cases which amount to:
--
class SomeTestCase : XCSelfTestCase {
  func testFailure() {
     withTestCase("Inputs/failing-test.swift") { results in
       ... check the results
     }
  }
}
--
where "Inputs/failing-test.swift" would be a checked in input file which has some expected behavior, and withTestCase() would be infrastructure (in XCSelfTestCase) which spawns a process to build and run that test, and then capture the output into a results object which could have further assertions run against it.

- Daniel

I'd love input on which of these approaches sounds more viable. Other ideas are also, of course, welcome!

- Brian Gesiak
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

3

Hi Brian,

Hello! This is in reference to https://github.com/apple/swift-corelibs-xctest/pull/3\. That pull request contains a commit that attempts to refactor XCTest such that it is more "unit-testable”.

Cool, thanks for looking into this area.

To do so, it gives XCTMain an additional parameter: a list of objects conforming to the Reporter protocol. I think of this as a minimal, corelibs equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because Reporter only defines Reporter.log(), whereas XCTestObserver has one method for each kind of test event (started, failed, finished, etc.).

Do you think it’d be possible to split out the idea of adding this new API to XCTest from getting some tests for XCTest itself?

The reason I’m asking is that (like Foundation and dispatch), we’re trying to keep the API surface of this XCTest very similar to the one that ships today with Xcode. This will help developers who need to integrate their cross-platform tests into suites that include features that Obj-C XCTest has that we will probably not add to the Swift one (e.g., UI testing).

We made a concession to language limitations with the XCTMain function, because there is no way to dynamically discover all of the test cases. I’d really like to get rid of it in the long term in favor of something else; maybe a decoration like @testable that we could find automatically.

- Tony

···

On Dec 3, 2015, at 3:45 PM, Brian Gesiak <modocache@gmail.com> wrote:

These reporters are, for now, storied in a global array. In the future, I'd like to discuss moving XCTest to a model in which all tests are (optionally) run in sub-processes, each of which may (optionally) run in parallel. This global array most certainly won't work for such a change, but for now, I simply want to have regression tests on the project. It's hard to send pull requests without knowing everything still works!

Besides this approach, which modifies XCTest in order to test it, it may be more prudent to add tests *without* changing XCTest at all. To do so, I could add tests that run programs that call XCTMain(), then verify what's printed to stdout. This could be done using a Python script (which would go well with the build script, also in Python).

I'd love input on which of these approaches sounds more viable. Other ideas are also, of course, welcome!

- Brian Gesiak

4

One possible direction to consider which would be more consistent with our goals this year for API-compatibility would be to look into implementing XCTestObserver.

Mike

···

On Dec 3, 2015, at 8:03 PM, Tony Parker <anthony.parker@apple.com> wrote:

Hi Brian,

On Dec 3, 2015, at 3:45 PM, Brian Gesiak <modocache@gmail.com <mailto:modocache@gmail.com>> wrote:

Hello! This is in reference to https://github.com/apple/swift-corelibs-xctest/pull/3\. That pull request contains a commit that attempts to refactor XCTest such that it is more "unit-testable”.

Cool, thanks for looking into this area.

To do so, it gives XCTMain an additional parameter: a list of objects conforming to the Reporter protocol. I think of this as a minimal, corelibs equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because Reporter only defines Reporter.log(), whereas XCTestObserver has one method for each kind of test event (started, failed, finished, etc.).

Do you think it’d be possible to split out the idea of adding this new API to XCTest from getting some tests for XCTest itself?

The reason I’m asking is that (like Foundation and dispatch), we’re trying to keep the API surface of this XCTest very similar to the one that ships today with Xcode. This will help developers who need to integrate their cross-platform tests into suites that include features that Obj-C XCTest has that we will probably not add to the Swift one (e.g., UI testing).

We made a concession to language limitations with the XCTMain function, because there is no way to dynamically discover all of the test cases. I’d really like to get rid of it in the long term in favor of something else; maybe a decoration like @testable that we could find automatically.

- Tony

These reporters are, for now, storied in a global array. In the future, I'd like to discuss moving XCTest to a model in which all tests are (optionally) run in sub-processes, each of which may (optionally) run in parallel. This global array most certainly won't work for such a change, but for now, I simply want to have regression tests on the project. It's hard to send pull requests without knowing everything still works!

Besides this approach, which modifies XCTest in order to test it, it may be more prudent to add tests *without* changing XCTest at all. To do so, I could add tests that run programs that call XCTMain(), then verify what's printed to stdout. This could be done using a Python script (which would go well with the build script, also in Python).

I'd love input on which of these approaches sounds more viable. Other ideas are also, of course, welcome!

- Brian Gesiak

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

5

Excellent, thanks for the feedback everyone!

It should be possible to use an out-of-process model that still uses XCTest itself to run the tests. For example, in the package manager we have some tests which spawn the package manager in order to test the end-to-end behavior. Ideally we would only do this for a small number of tests that really need this level of testing, and use unit testing for the rest.

I'll send a pull request with this approach, since I believe it's the
least invasive to the current API while still getting us regression
tests for the current implementation. Of course I'd be thrilled if
someone beats me to it! :)

The reason I’m asking is that (like Foundation and dispatch), we’re trying to keep the API surface of this XCTest very similar to the one that ships today with Xcode. This will help developers who need to integrate their cross-platform tests into suites that include features that Obj-C XCTest has that we will probably not add to the Swift one (e.g., UI testing).

Absolutely agree. Still, once we have regression tests in place, I'd
love to start a conversation about whether we could provide a Obj-C
XCTest-compatible API layer on top of some more flexible architecture.

One possible direction to consider which would be more consistent with our goals this year for API-compatibility would be to look into implementing XCTestObserver.

Yes! Very exciting. We'll need to consider how methods like
`-[XCTestObservation testBundleWillStart:]` map to SwiftXCTest,
especially considering we don't use NSBundle at all (which I think is
a great thing). I'll be looking forward to participating in more
discussions on this mailing list.

- Brian Gesiak

···

On Thu, Dec 3, 2015 at 10:39 PM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

On Dec 3, 2015, at 8:03 PM, Tony Parker <anthony.parker@apple.com> wrote:
On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris <mferris@apple.com> wrote:

On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris <mferris@apple.com> wrote:

One possible direction to consider which would be more consistent with our goals this year for API-compatibility would be to look into implementing XCTestObserver.

Mike

On Dec 3, 2015, at 8:03 PM, Tony Parker <anthony.parker@apple.com> wrote:

Hi Brian,

On Dec 3, 2015, at 3:45 PM, Brian Gesiak <modocache@gmail.com> wrote:

Hello! This is in reference to https://github.com/apple/swift-corelibs-xctest/pull/3\. That pull request contains a commit that attempts to refactor XCTest such that it is more "unit-testable”.

Cool, thanks for looking into this area.

To do so, it gives XCTMain an additional parameter: a list of objects conforming to the Reporter protocol. I think of this as a minimal, corelibs equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because Reporter only defines Reporter.log(), whereas XCTestObserver has one method for each kind of test event (started, failed, finished, etc.).

Do you think it’d be possible to split out the idea of adding this new API to XCTest from getting some tests for XCTest itself?

The reason I’m asking is that (like Foundation and dispatch), we’re trying to keep the API surface of this XCTest very similar to the one that ships today with Xcode. This will help developers who need to integrate their cross-platform tests into suites that include features that Obj-C XCTest has that we will probably not add to the Swift one (e.g., UI testing).

We made a concession to language limitations with the XCTMain function, because there is no way to dynamically discover all of the test cases. I’d really like to get rid of it in the long term in favor of something else; maybe a decoration like @testable that we could find automatically.

- Tony

These reporters are, for now, storied in a global array. In the future, I'd like to discuss moving XCTest to a model in which all tests are (optionally) run in sub-processes, each of which may (optionally) run in parallel. This global array most certainly won't work for such a change, but for now, I simply want to have regression tests on the project. It's hard to send pull requests without knowing everything still works!

Besides this approach, which modifies XCTest in order to test it, it may be more prudent to add tests *without* changing XCTest at all. To do so, I could add tests that run programs that call XCTMain(), then verify what's printed to stdout. This could be done using a Python script (which would go well with the build script, also in Python).

I'd love input on which of these approaches sounds more viable. Other ideas are also, of course, welcome!

- Brian Gesiak

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

6

Oops. It was pointed out to me that XCTestObserver is deprecated. I get it confused sometimes with its replacement.. What we’d presumably want to pursue adding the the corelibs XCTest is the API from XCTestObservation.h (and associated stuff).

Your same questions about the bundle stuff will still apply, though. On the other hand, perhaps a partial implementation that at least allowed observation starting at the test suites or even just at the test cases would suffice to start things off.

Mike

···

On Dec 4, 2015, at 9:08 AM, Brian Gesiak <modocache@gmail.com> wrote:

Excellent, thanks for the feedback everyone!

On Thu, Dec 3, 2015 at 10:39 PM, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>> wrote:

It should be possible to use an out-of-process model that still uses XCTest itself to run the tests. For example, in the package manager we have some tests which spawn the package manager in order to test the end-to-end behavior. Ideally we would only do this for a small number of tests that really need this level of testing, and use unit testing for the rest.

I'll send a pull request with this approach, since I believe it's the
least invasive to the current API while still getting us regression
tests for the current implementation. Of course I'd be thrilled if
someone beats me to it! :)

On Dec 3, 2015, at 8:03 PM, Tony Parker <anthony.parker@apple.com <mailto:anthony.parker@apple.com>> wrote:
The reason I’m asking is that (like Foundation and dispatch), we’re trying to keep the API surface of this XCTest very similar to the one that ships today with Xcode. This will help developers who need to integrate their cross-platform tests into suites that include features that Obj-C XCTest has that we will probably not add to the Swift one (e.g., UI testing).

Absolutely agree. Still, once we have regression tests in place, I'd
love to start a conversation about whether we could provide a Obj-C
XCTest-compatible API layer on top of some more flexible architecture.

On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris <mferris@apple.com <mailto:mferris@apple.com>> wrote:
One possible direction to consider which would be more consistent with our goals this year for API-compatibility would be to look into implementing XCTestObserver.

Yes! Very exciting. We'll need to consider how methods like
`-[XCTestObservation testBundleWillStart:]` map to SwiftXCTest,
especially considering we don't use NSBundle at all (which I think is
a great thing). I'll be looking forward to participating in more
discussions on this mailing list.

- Brian Gesiak

On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris <mferris@apple.com <mailto:mferris@apple.com>> wrote:

One possible direction to consider which would be more consistent with our goals this year for API-compatibility would be to look into implementing XCTestObserver.

Mike

On Dec 3, 2015, at 8:03 PM, Tony Parker <anthony.parker@apple.com> wrote:

Hi Brian,

On Dec 3, 2015, at 3:45 PM, Brian Gesiak <modocache@gmail.com> wrote:

Hello! This is in reference to https://github.com/apple/swift-corelibs-xctest/pull/3\. That pull request contains a commit that attempts to refactor XCTest such that it is more "unit-testable”.

Cool, thanks for looking into this area.

To do so, it gives XCTMain an additional parameter: a list of objects conforming to the Reporter protocol. I think of this as a minimal, corelibs equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because Reporter only defines Reporter.log(), whereas XCTestObserver has one method for each kind of test event (started, failed, finished, etc.).

Do you think it’d be possible to split out the idea of adding this new API to XCTest from getting some tests for XCTest itself?

The reason I’m asking is that (like Foundation and dispatch), we’re trying to keep the API surface of this XCTest very similar to the one that ships today with Xcode. This will help developers who need to integrate their cross-platform tests into suites that include features that Obj-C XCTest has that we will probably not add to the Swift one (e.g., UI testing).

We made a concession to language limitations with the XCTMain function, because there is no way to dynamically discover all of the test cases. I’d really like to get rid of it in the long term in favor of something else; maybe a decoration like @testable that we could find automatically.

- Tony

These reporters are, for now, storied in a global array. In the future, I'd like to discuss moving XCTest to a model in which all tests are (optionally) run in sub-processes, each of which may (optionally) run in parallel. This global array most certainly won't work for such a change, but for now, I simply want to have regression tests on the project. It's hard to send pull requests without knowing everything still works!

Besides this approach, which modifies XCTest in order to test it, it may be more prudent to add tests *without* changing XCTest at all. To do so, I could add tests that run programs that call XCTMain(), then verify what's printed to stdout. This could be done using a Python script (which would go well with the build script, also in Python).

I'd love input on which of these approaches sounds more viable. Other ideas are also, of course, welcome!

- Brian Gesiak

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

7

I sent up another attempt for this: [Tests] Add regression test suite for OS X and Linux by modocache · Pull Request #10 · apple/swift-corelibs-xctest · GitHub
Based on the discussion in that pull request, I'm now exploring using lit to test the XCTest output.
Rather than asking contributors to install lit via the Python Package Index, I was considering using the apple/llvm repository's copy. Contributors would need to have cloned llvm in a specific directory relative to the cloned swift-corelibs-xctest, like so:
rootdir/ llvm/ swift-corelibs-xctest/
This appears to be a convention in the Swift family of repositories, so I don't anticipate this to cause confusion among contributors.
If anyone has any input on this approach to lit, or on the direction of these changes in general, please let me know! All feedback is greatly appreciated.
- Brian Gesiak

···

_____________________________
From: Mike Ferris <mferris@apple.com>
Sent: Friday, December 4, 2015 3:42 PM
Subject: Re: [swift-corelibs-dev] [xctest] Who tests the tests?
To: Brian Gesiak <modocache@gmail.com>
Cc: Tony Parker <anthony.parker@apple.com>, <swift-corelibs-dev@swift.org>, Daniel Dunbar <daniel_dunbar@apple.com>

       Oops. It was pointed out to me that XCTestObserver is deprecated. I get it confused sometimes with its replacement.. What we’d presumably want to pursue adding the the corelibs XCTest is the API from XCTestObservation.h (and associated stuff).
       Your same questions about the bundle stuff will still apply, though. On the other hand, perhaps a partial implementation that at least allowed observation starting at the test suites or even just at the test cases would suffice to start things off.
       
       Mike
                  On Dec 4, 2015, at 9:08 AM, Brian Gesiak < modocache@gmail.com> wrote:
           Excellent, thanks for the feedback everyone!
      
      On Thu, Dec 3, 2015 at 10:39 PM, Daniel Dunbar < daniel_dunbar@apple.com > wrote:
             It should be possible to use an out-of-process model that still uses XCTest itself to run the tests. For example, in the package manager we have some tests which spawn the package manager in order to test the end-to-end behavior. Ideally we would only do this for a small number of tests that really need this level of testing, and use unit testing for the rest.
            
      I'll send a pull request with this approach, since I believe it's the
      least invasive to the current API while still getting us regression
      tests for the current implementation. Of course I'd be thrilled if
      someone beats me to it! :)
      
             On Dec 3, 2015, at 8:03 PM, Tony Parker < anthony.parker@apple.com> wrote:
The reason I’m asking is that (like Foundation and dispatch), we’re trying to keep the API surface of this XCTest very similar to the one that ships today with Xcode. This will help developers who need to integrate their cross-platform tests into suites that include features that Obj-C XCTest has that we will probably not add to the Swift one (e.g., UI testing).
            
      Absolutely agree. Still, once we have regression tests in place, I'd
      love to start a conversation about whether we could provide a Obj-C
      XCTest-compatible API layer on top of some more flexible architecture.
      
             On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris < mferris@apple.com> wrote:
One possible direction to consider which would be more consistent with our goals this year for API-compatibility would be to look into implementing XCTestObserver.
            
      Yes! Very exciting. We'll need to consider how methods like
      `-[XCTestObservation testBundleWillStart:]` map to SwiftXCTest,
      especially considering we don't use NSBundle at all (which I think is
      a great thing). I'll be looking forward to participating in more
      discussions on this mailing list.
      
      - Brian Gesiak
      
      On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris < mferris@apple.com > wrote:
             
One possible direction to consider which would be more consistent with our goals this year for API-compatibility would be to look into implementing XCTestObserver.
       
Mike
       
On Dec 3, 2015, at 8:03 PM, Tony Parker < anthony.parker@apple.com> wrote:
       
Hi Brian,
       
On Dec 3, 2015, at 3:45 PM, Brian Gesiak < modocache@gmail.com> wrote:
       
Hello! This is in reference to https://github.com/apple/swift-corelibs-xctest/pull/3\. That pull request contains a commit that attempts to refactor XCTest such that it is more "unit-testable”.
       
Cool, thanks for looking into this area.
       
To do so, it gives XCTMain an additional parameter: a list of objects conforming to the Reporter protocol. I think of this as a minimal, corelibs equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because Reporter only defines Reporter.log(), whereas XCTestObserver has one method for each kind of test event (started, failed, finished, etc.).
       
Do you think it’d be possible to split out the idea of adding this new API to XCTest from getting some tests for XCTest itself?
       
The reason I’m asking is that (like Foundation and dispatch), we’re trying to keep the API surface of this XCTest very similar to the one that ships today with Xcode. This will help developers who need to integrate their cross-platform tests into suites that include features that Obj-C XCTest has that we will probably not add to the Swift one (e.g., UI testing).
       
We made a concession to language limitations with the XCTMain function, because there is no way to dynamically discover all of the test cases. I’d really like to get rid of it in the long term in favor of something else; maybe a decoration like @testable that we could find automatically.
       
- Tony
       
These reporters are, for now, storied in a global array. In the future, I'd like to discuss moving XCTest to a model in which all tests are (optionally) run in sub-processes, each of which may (optionally) run in parallel. This global array most certainly won't work for such a change, but for now, I simply want to have regression tests on the project. It's hard to send pull requests without knowing everything still works!
       
Besides this approach, which modifies XCTest in order to test it, it may be more prudent to add tests *without* changing XCTest at all. To do so, I could add tests that run programs that call XCTMain(), then verify what's printed to stdout. This could be done using a Python script (which would go well with the build script, also in Python).
       
I'd love input on which of these approaches sounds more viable. Other ideas are also, of course, welcome!
       
- Brian Gesiak
       
_______________________________________________
swift-corelibs-dev mailing list
       swift-corelibs-dev@swift.org
       https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

8

Hi Brian,

I sent up another attempt for this: [Tests] Add regression test suite for OS X and Linux by modocache · Pull Request #10 · apple/swift-corelibs-xctest · GitHub

Based on the discussion in that pull request, I'm now exploring using lit to test the XCTest output.

Rather than asking contributors to install lit via the Python Package Index, I was considering using the apple/llvm repository's copy. Contributors would need to have cloned llvm in a specific directory relative to the cloned swift-corelibs-xctest, like so:

rootdir/
    llvm/
    swift-corelibs-xctest/

This appears to be a convention in the Swift family of repositories, so I don't anticipate this to cause confusion among contributors.

If anyone has any input on this approach to lit, or on the direction of these changes in general, please let me know! All feedback is greatly appreciated.

This is probably a fine approach - swift-corelibs-foundation, for example, assumes that swift-corelibs-xctest is a sibling directory to it when building.

Even if we change our minds later, there’s no doubt that having a starting point is valuable.

- Tony

···

On Dec 7, 2015, at 3:54 PM, Brian Gesiak <modocache@gmail.com> wrote:

- Brian Gesiak
_____________________________
From: Mike Ferris <mferris@apple.com <mailto:mferris@apple.com>>
Sent: Friday, December 4, 2015 3:42 PM
Subject: Re: [swift-corelibs-dev] [xctest] Who tests the tests?
To: Brian Gesiak <modocache@gmail.com <mailto:modocache@gmail.com>>
Cc: Tony Parker <anthony.parker@apple.com <mailto:anthony.parker@apple.com>>, <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>>, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>>

Oops. It was pointed out to me that XCTestObserver is deprecated. I get it confused sometimes with its replacement.. What we’d presumably want to pursue adding the the corelibs XCTest is the API from XCTestObservation.h (and associated stuff).

Your same questions about the bundle stuff will still apply, though. On the other hand, perhaps a partial implementation that at least allowed observation starting at the test suites or even just at the test cases would suffice to start things off.

Mike

On Dec 4, 2015, at 9:08 AM, Brian Gesiak < modocache@gmail.com <mailto:modocache@gmail.com>> wrote:

Excellent, thanks for the feedback everyone!

On Thu, Dec 3, 2015 at 10:39 PM, Daniel Dunbar < daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com> > wrote:
It should be possible to use an out-of-process model that still uses XCTest itself to run the tests. For example, in the package manager we have some tests which spawn the package manager in order to test the end-to-end behavior. Ideally we would only do this for a small number of tests that really need this level of testing, and use unit testing for the rest.

I'll send a pull request with this approach, since I believe it's the
least invasive to the current API while still getting us regression
tests for the current implementation. Of course I'd be thrilled if
someone beats me to it! :)

On Dec 3, 2015, at 8:03 PM, Tony Parker < anthony.parker@apple.com <mailto:anthony.parker@apple.com>> wrote:
The reason I’m asking is that (like Foundation and dispatch), we’re trying to keep the API surface of this XCTest very similar to the one that ships today with Xcode. This will help developers who need to integrate their cross-platform tests into suites that include features that Obj-C XCTest has that we will probably not add to the Swift one (e.g., UI testing).

Absolutely agree. Still, once we have regression tests in place, I'd
love to start a conversation about whether we could provide a Obj-C
XCTest-compatible API layer on top of some more flexible architecture.

On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris < mferris@apple.com <mailto:mferris@apple.com>> wrote:
One possible direction to consider which would be more consistent with our goals this year for API-compatibility would be to look into implementing XCTestObserver.

Yes! Very exciting. We'll need to consider how methods like
`-[XCTestObservation testBundleWillStart:]` map to SwiftXCTest,
especially considering we don't use NSBundle at all (which I think is
a great thing). I'll be looking forward to participating in more
discussions on this mailing list.

- Brian Gesiak

On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris < mferris@apple.com <mailto:mferris@apple.com> > wrote:

One possible direction to consider which would be more consistent with our goals this year for API-compatibility would be to look into implementing XCTestObserver.

Mike

On Dec 3, 2015, at 8:03 PM, Tony Parker < anthony.parker@apple.com <mailto:anthony.parker@apple.com>> wrote:

Hi Brian,

On Dec 3, 2015, at 3:45 PM, Brian Gesiak < modocache@gmail.com <mailto:modocache@gmail.com>> wrote:

Hello! This is in reference to https://github.com/apple/swift-corelibs-xctest/pull/3\. That pull request contains a commit that attempts to refactor XCTest such that it is more "unit-testable”.

Cool, thanks for looking into this area.

To do so, it gives XCTMain an additional parameter: a list of objects conforming to the Reporter protocol. I think of this as a minimal, corelibs equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because Reporter only defines Reporter.log(), whereas XCTestObserver has one method for each kind of test event (started, failed, finished, etc.).

Do you think it’d be possible to split out the idea of adding this new API to XCTest from getting some tests for XCTest itself?

The reason I’m asking is that (like Foundation and dispatch), we’re trying to keep the API surface of this XCTest very similar to the one that ships today with Xcode. This will help developers who need to integrate their cross-platform tests into suites that include features that Obj-C XCTest has that we will probably not add to the Swift one (e.g., UI testing).

We made a concession to language limitations with the XCTMain function, because there is no way to dynamically discover all of the test cases. I’d really like to get rid of it in the long term in favor of something else; maybe a decoration like @testable that we could find automatically.

- Tony

These reporters are, for now, storied in a global array. In the future, I'd like to discuss moving XCTest to a model in which all tests are (optionally) run in sub-processes, each of which may (optionally) run in parallel. This global array most certainly won't work for such a change, but for now, I simply want to have regression tests on the project. It's hard to send pull requests without knowing everything still works!

Besides this approach, which modifies XCTest in order to test it, it may be more prudent to add tests *without* changing XCTest at all. To do so, I could add tests that run programs that call XCTMain(), then verify what's printed to stdout. This could be done using a Python script (which would go well with the build script, also in Python).

I'd love input on which of these approaches sounds more viable. Other ideas are also, of course, welcome!

- Brian Gesiak

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

9

I've submitted Add lit tests by modocache · Pull Request #14 · apple/swift-corelibs-xctest · GitHub
as another attempt at this.

Take a look at the pull request message for details. Turns out I'd
overlooked another dependency: nearly all lit tests use FileCheck,
which means to run the tests developers will need to specify that
file's location on their PATH.

Feedback welcome as always! Sorry for the churn on the pull request.

- Brian Gesiak

···

On Mon, Dec 7, 2015 at 7:05 PM, Tony Parker <anthony.parker@apple.com> wrote:

Hi Brian,

On Dec 7, 2015, at 3:54 PM, Brian Gesiak <modocache@gmail.com> wrote:

I sent up another attempt for this:
[Tests] Add regression test suite for OS X and Linux by modocache · Pull Request #10 · apple/swift-corelibs-xctest · GitHub

Based on the discussion in that pull request, I'm now exploring using lit to
test the XCTest output.

Rather than asking contributors to install lit via the Python Package Index,
I was considering using the apple/llvm repository's copy. Contributors would
need to have cloned llvm in a specific directory relative to the cloned
swift-corelibs-xctest, like so:

rootdir/
    llvm/
    swift-corelibs-xctest/

This appears to be a convention in the Swift family of repositories, so I
don't anticipate this to cause confusion among contributors.

If anyone has any input on this approach to lit, or on the direction of
these changes in general, please let me know! All feedback is greatly
appreciated.

This is probably a fine approach - swift-corelibs-foundation, for example,
assumes that swift-corelibs-xctest is a sibling directory to it when
building.

Even if we change our minds later, there’s no doubt that having a starting
point is valuable.

- Tony

- Brian Gesiak
_____________________________
From: Mike Ferris <mferris@apple.com>
Sent: Friday, December 4, 2015 3:42 PM
Subject: Re: [swift-corelibs-dev] [xctest] Who tests the tests?
To: Brian Gesiak <modocache@gmail.com>
Cc: Tony Parker <anthony.parker@apple.com>, <swift-corelibs-dev@swift.org>,
Daniel Dunbar <daniel_dunbar@apple.com>

Oops. It was pointed out to me that XCTestObserver is deprecated. I get it
confused sometimes with its replacement.. What we’d presumably want to
pursue adding the the corelibs XCTest is the API from XCTestObservation.h
(and associated stuff).

Your same questions about the bundle stuff will still apply, though. On the
other hand, perhaps a partial implementation that at least allowed
observation starting at the test suites or even just at the test cases would
suffice to start things off.

Mike

On Dec 4, 2015, at 9:08 AM, Brian Gesiak < modocache@gmail.com> wrote:

Excellent, thanks for the feedback everyone!

On Thu, Dec 3, 2015 at 10:39 PM, Daniel Dunbar < daniel_dunbar@apple.com > > wrote:

It should be possible to use an out-of-process model that still uses XCTest
itself to run the tests. For example, in the package manager we have some
tests which spawn the package manager in order to test the end-to-end
behavior. Ideally we would only do this for a small number of tests that
really need this level of testing, and use unit testing for the rest.

I'll send a pull request with this approach, since I believe it's the
least invasive to the current API while still getting us regression
tests for the current implementation. Of course I'd be thrilled if
someone beats me to it! :)

On Dec 3, 2015, at 8:03 PM, Tony Parker < anthony.parker@apple.com> wrote:
The reason I’m asking is that (like Foundation and dispatch), we’re trying
to keep the API surface of this XCTest very similar to the one that ships
today with Xcode. This will help developers who need to integrate their
cross-platform tests into suites that include features that Obj-C XCTest has
that we will probably not add to the Swift one (e.g., UI testing).

Absolutely agree. Still, once we have regression tests in place, I'd
love to start a conversation about whether we could provide a Obj-C
XCTest-compatible API layer on top of some more flexible architecture.

On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris < mferris@apple.com> wrote:
One possible direction to consider which would be more consistent with our
goals this year for API-compatibility would be to look into implementing
XCTestObserver.

Yes! Very exciting. We'll need to consider how methods like
`-[XCTestObservation testBundleWillStart:]` map to SwiftXCTest,
especially considering we don't use NSBundle at all (which I think is
a great thing). I'll be looking forward to participating in more
discussions on this mailing list.

- Brian Gesiak

On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris < mferris@apple.com > wrote:

One possible direction to consider which would be more consistent with our
goals this year for API-compatibility would be to look into implementing
XCTestObserver.

Mike

On Dec 3, 2015, at 8:03 PM, Tony Parker < anthony.parker@apple.com> wrote:

Hi Brian,

On Dec 3, 2015, at 3:45 PM, Brian Gesiak < modocache@gmail.com> wrote:

Hello! This is in reference to
https://github.com/apple/swift-corelibs-xctest/pull/3\. That pull request
contains a commit that attempts to refactor XCTest such that it is more
"unit-testable”.

Cool, thanks for looking into this area.

To do so, it gives XCTMain an additional parameter: a list of objects
conforming to the Reporter protocol. I think of this as a minimal, corelibs
equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because
Reporter only defines Reporter.log(), whereas XCTestObserver has one method
for each kind of test event (started, failed, finished, etc.).

Do you think it’d be possible to split out the idea of adding this new API
to XCTest from getting some tests for XCTest itself?

The reason I’m asking is that (like Foundation and dispatch), we’re trying
to keep the API surface of this XCTest very similar to the one that ships
today with Xcode. This will help developers who need to integrate their
cross-platform tests into suites that include features that Obj-C XCTest has
that we will probably not add to the Swift one (e.g., UI testing).

We made a concession to language limitations with the XCTMain function,
because there is no way to dynamically discover all of the test cases. I’d
really like to get rid of it in the long term in favor of something else;
maybe a decoration like @testable that we could find automatically.

- Tony

These reporters are, for now, storied in a global array. In the future, I'd
like to discuss moving XCTest to a model in which all tests are (optionally)
run in sub-processes, each of which may (optionally) run in parallel. This
global array most certainly won't work for such a change, but for now, I
simply want to have regression tests on the project. It's hard to send pull
requests without knowing everything still works!

Besides this approach, which modifies XCTest in order to test it, it may be
more prudent to add tests *without* changing XCTest at all. To do so, I
could add tests that run programs that call XCTMain(), then verify what's
printed to stdout. This could be done using a Python script (which would go
well with the build script, also in Python).

I'd love input on which of these approaches sounds more viable. Other ideas
are also, of course, welcome!

- Brian Gesiak

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev