SwiftXCTest proposals to make it more useful as a framework and for other platforms

Ilya_Laryionau · December 2, 2016, 10:57am

Hello Community,

I’m working on adding iOS platform support for SwiftXCTest. So I’ve have the following proposals and would like to hear feedback from the community.
SwiftXCTest runs the tests in a way `XCTMain([ testCase(TestFoo.allTests) ])`. XCTMain supports only Linux / FreeBSD / macOS. So I suggest introducing an API that will provide a similar way of running tests as XCTMain, but in a case using SwiftXCTest as a framework e.g. on iOS. It could be the same logic that just returns exit code, when XCTMain just exits with this code.
The suggestion above (number 1) is good, but too simple for using SwiftXCTest as a framework, since we only have exit code and debug logs. So my second proposal is providing mechanism of initialising root test suite with array of testCases (array of XCTestCaseEntry objects) with the same structure that XCTMain does. This isn’t possible now since XCTestCaseSuite class that is used to initialise root test suite (see XCTMain.swift line 80) is internal.
This proposal is because Swift has very poor reflection at the moment. So I suggest creating a generator (pre-build script) for Swift test cases that for example is able analysing concrete folder with swift files and generate e.g. TestCases.swift file. This script is intended to be used manually.
Looking forward to any feedback.

Ilya

Brian_Gesiak · December 2, 2016, 6:00pm

Hey Ilya,

I'm glad you're thinking about how to use corelibs-xctest on mobile
devices. I'd like to prepare it for Android as well, and I think there's a
lot of overlap here.

This proposal is because Swift has very poor reflection at the moment. So

I suggest creating a generator (pre-build script) for Swift test cases that
for example is able analysing concrete folder with swift files and generate
e.g. TestCases.swift file. This script is intended to be used manually.

You should check out https://bugs.swift.org/browse/SR-710\. There's been a
great deal of discussion on this topic in the past. I think we left off
with the idea that we should use SourceKit to generate the list of tests
(+cc Daniel Dunbar). This work has largely been abandoned. If you'd like to
pick it up, you could have huge impact!

I'm not sure I understand your first two points. Are you suggesting that we
implement some method of kicking off tests on an iOS simulator, but only
when corelibs-xctest framework is linked with the application bundle...?
I'd appreciate it if you could explain what you mean in more detail.

- Brian Gesiak

···

On Fri, Dec 2, 2016 at 5:57 AM, Ilya Laryionau via swift-corelibs-dev < swift-corelibs-dev@swift.org> wrote:

Hello Community,

I’m working on adding iOS platform support for SwiftXCTest. So I’ve have
the following proposals and would like to hear feedback from the community.

   1. SwiftXCTest runs the tests in a way `XCTMain([
   testCase(TestFoo.allTests) ])`. XCTMain supports only Linux / FreeBSD /
   macOS. So I suggest introducing an API that will provide a similar way of
   running tests as XCTMain, but in a case using SwiftXCTest as a framework
   e.g. on iOS. It could be the same logic that just returns exit code, when
   XCTMain just exits with this code.
   2. The suggestion above (number 1) is good, but too simple for using
   SwiftXCTest as a framework, since we only have exit code and debug logs. So
   my second proposal is providing mechanism of initialising root test suite
   with array of testCases (array of XCTestCaseEntry objects) with the same
   structure that XCTMain does. This isn’t possible now since XCTestCaseSuite
   class that is used to initialise root test suite (see XCTMain.swift line
   80) is internal.
   3. This proposal is because Swift has very poor reflection at the
   moment. So I suggest creating a generator (pre-build script) for Swift test
   cases that for example is able analysing concrete folder with swift files
   and generate e.g. TestCases.swift file. This script is intended to be used
   manually.

Looking forward to any feedback.

Ilya

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Ilya_Laryionau · December 5, 2016, 9:30am

Brian,

I appreciate your comments.

You should check out https://bugs.swift.org/browse/SR-710\. There's been a great deal of discussion on this topic in the past. I think we left off with the idea that we should use SourceKit to generate the list of tests (+cc Daniel Dunbar). This work has largely been abandoned. If you'd like to pick it up, you could have huge impact!

This’s a great comment. I’ll check what I can do.

I'm not sure I understand your first two points. Are you suggesting that we implement some method of kicking off tests on an iOS simulator, but only when corelibs-xctest framework is linked with the application bundle...? I'd appreciate it if you could explain what you mean in more detail.

Let me clarify my first two points.

In general you’re correct. I suggest implementing some method that triggers the same logic that XCTestMain does. But instead of calling exit, when the work is done, returning the exit code.
I suggest implementing API that allows creating XCTestSuite from a list of XCTestCaseEntries.

Ilya

···

On 2 Dec 2016, at 21:00, Brian Gesiak <modocache@gmail.com> wrote:

Hey Ilya,

I'm glad you're thinking about how to use corelibs-xctest on mobile devices. I'd like to prepare it for Android as well, and I think there's a lot of overlap here.

> This proposal is because Swift has very poor reflection at the moment. So I suggest creating a generator (pre-build script) for Swift test cases that for example is able analysing concrete folder with swift files and generate e.g. TestCases.swift file. This script is intended to be used manually.

You should check out https://bugs.swift.org/browse/SR-710\. There's been a great deal of discussion on this topic in the past. I think we left off with the idea that we should use SourceKit to generate the list of tests (+cc Daniel Dunbar). This work has largely been abandoned. If you'd like to pick it up, you could have huge impact!

I'm not sure I understand your first two points. Are you suggesting that we implement some method of kicking off tests on an iOS simulator, but only when corelibs-xctest framework is linked with the application bundle...? I'd appreciate it if you could explain what you mean in more detail.

- Brian Gesiak

On Fri, Dec 2, 2016 at 5:57 AM, Ilya Laryionau via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:
Hello Community,

I’m working on adding iOS platform support for SwiftXCTest. So I’ve have the following proposals and would like to hear feedback from the community.
SwiftXCTest runs the tests in a way `XCTMain([ testCase(TestFoo.allTests) ])`. XCTMain supports only Linux / FreeBSD / macOS. So I suggest introducing an API that will provide a similar way of running tests as XCTMain, but in a case using SwiftXCTest as a framework e.g. on iOS. It could be the same logic that just returns exit code, when XCTMain just exits with this code.
The suggestion above (number 1) is good, but too simple for using SwiftXCTest as a framework, since we only have exit code and debug logs. So my second proposal is providing mechanism of initialising root test suite with array of testCases (array of XCTestCaseEntry objects) with the same structure that XCTMain does. This isn’t possible now since XCTestCaseSuite class that is used to initialise root test suite (see XCTMain.swift line 80) is internal.
This proposal is because Swift has very poor reflection at the moment. So I suggest creating a generator (pre-build script) for Swift test cases that for example is able analysing concrete folder with swift files and generate e.g. TestCases.swift file. This script is intended to be used manually.
Looking forward to any feedback.

Ilya

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Brian_Gesiak · December 5, 2016, 5:50pm

Ilya,

I would love to see both of your suggestions implemented, but you may need
to submit a swift-evolution proposal for them. I think Apple employees
maintaining swift-corelibs-xctest would have a better idea of whether your
proposals would be accepted by the core team. (+cc Brian Croom).

I think a good next step would be to reply to this email with the function
signatures for the additions you're proposing. That will allow us to
discuss them in more concrete terms.

Thanks for all the great suggestions!

- Brian Gesiak

···

On Mon, Dec 5, 2016 at 1:30 AM, Ilya Laryionau <larryonoff@gmail.com> wrote:

Brian,

I appreciate your comments.

You should check out https://bugs.swift.org/browse/SR-710\. There's been a
great deal of discussion on this topic in the past. I think we left off
with the idea that we should use SourceKit to generate the list of tests
(+cc Daniel Dunbar). This work has largely been abandoned. If you'd like to
pick it up, you could have huge impact!

This’s a great comment. I’ll check what I can do.

I'm not sure I understand your first two points. Are you suggesting that
we implement some method of kicking off tests on an iOS simulator, but only
when corelibs-xctest framework is linked with the application bundle...?
I'd appreciate it if you could explain what you mean in more detail.

Let me clarify my first two points.

   1. In general you’re correct. I suggest implementing some method that
   triggers the same logic that XCTestMain does. But instead of calling exit,
   when the work is done, returning the exit code.
   2. I suggest implementing API that allows creating XCTestSuite from a
   list of XCTestCaseEntries.

Ilya

On 2 Dec 2016, at 21:00, Brian Gesiak <modocache@gmail.com> wrote:

Hey Ilya,

I'm glad you're thinking about how to use corelibs-xctest on mobile
devices. I'd like to prepare it for Android as well, and I think there's a
lot of overlap here.

> This proposal is because Swift has very poor reflection at the moment.
So I suggest creating a generator (pre-build script) for Swift test cases
that for example is able analysing concrete folder with swift files and
generate e.g. TestCases.swift file. This script is intended to be used
manually.

You should check out https://bugs.swift.org/browse/SR-710\. There's been a
great deal of discussion on this topic in the past. I think we left off
with the idea that we should use SourceKit to generate the list of tests
(+cc Daniel Dunbar). This work has largely been abandoned. If you'd like to
pick it up, you could have huge impact!

I'm not sure I understand your first two points. Are you suggesting that
we implement some method of kicking off tests on an iOS simulator, but only
when corelibs-xctest framework is linked with the application bundle...?
I'd appreciate it if you could explain what you mean in more detail.

- Brian Gesiak

On Fri, Dec 2, 2016 at 5:57 AM, Ilya Laryionau via swift-corelibs-dev < > swift-corelibs-dev@swift.org> wrote:

Hello Community,

I’m working on adding iOS platform support for SwiftXCTest. So I’ve have
the following proposals and would like to hear feedback from the community.

   1. SwiftXCTest runs the tests in a way `XCTMain([
   testCase(TestFoo.allTests) ])`. XCTMain supports only Linux / FreeBSD /
   macOS. So I suggest introducing an API that will provide a similar way of
   running tests as XCTMain, but in a case using SwiftXCTest as a framework
   e.g. on iOS. It could be the same logic that just returns exit code, when
   XCTMain just exits with this code.
   2. The suggestion above (number 1) is good, but too simple for using
   SwiftXCTest as a framework, since we only have exit code and debug logs. So
   my second proposal is providing mechanism of initialising root test suite
   with array of testCases (array of XCTestCaseEntry objects) with the same
   structure that XCTMain does. This isn’t possible now since XCTestCaseSuite
   class that is used to initialise root test suite (see XCTMain.swift line
   80) is internal.
   3. This proposal is because Swift has very poor reflection at the
   moment. So I suggest creating a generator (pre-build script) for Swift test
   cases that for example is able analysing concrete folder with swift files
   and generate e.g. TestCases.swift file. This script is intended to be used
   manually.

Looking forward to any feedback.

Ilya

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Ilya_Laryionau · December 6, 2016, 9:50am

Brian,

Thank you for your feedback.

I need some time to think about the API for the first and the seconds points. I’m going to create it as a list of protocol in gist.

Ilya

···

On 5 Dec 2016, at 20:50, Brian Gesiak <modocache@gmail.com> wrote:

Ilya,

I would love to see both of your suggestions implemented, but you may need to submit a swift-evolution proposal for them. I think Apple employees maintaining swift-corelibs-xctest would have a better idea of whether your proposals would be accepted by the core team. (+cc Brian Croom).

I think a good next step would be to reply to this email with the function signatures for the additions you're proposing. That will allow us to discuss them in more concrete terms.

Thanks for all the great suggestions!

- Brian Gesiak

On Mon, Dec 5, 2016 at 1:30 AM, Ilya Laryionau <larryonoff@gmail.com <mailto:larryonoff@gmail.com>> wrote:
Brian,

I appreciate your comments.

You should check out https://bugs.swift.org/browse/SR-710\. There's been a great deal of discussion on this topic in the past. I think we left off with the idea that we should use SourceKit to generate the list of tests (+cc Daniel Dunbar). This work has largely been abandoned. If you'd like to pick it up, you could have huge impact!

This’s a great comment. I’ll check what I can do.

I'm not sure I understand your first two points. Are you suggesting that we implement some method of kicking off tests on an iOS simulator, but only when corelibs-xctest framework is linked with the application bundle...? I'd appreciate it if you could explain what you mean in more detail.

Let me clarify my first two points.

In general you’re correct. I suggest implementing some method that triggers the same logic that XCTestMain does. But instead of calling exit, when the work is done, returning the exit code.
I suggest implementing API that allows creating XCTestSuite from a list of XCTestCaseEntries.

Ilya

On 2 Dec 2016, at 21:00, Brian Gesiak <modocache@gmail.com <mailto:modocache@gmail.com>> wrote:

Hey Ilya,

I'm glad you're thinking about how to use corelibs-xctest on mobile devices. I'd like to prepare it for Android as well, and I think there's a lot of overlap here.

> This proposal is because Swift has very poor reflection at the moment. So I suggest creating a generator (pre-build script) for Swift test cases that for example is able analysing concrete folder with swift files and generate e.g. TestCases.swift file. This script is intended to be used manually.

You should check out https://bugs.swift.org/browse/SR-710\. There's been a great deal of discussion on this topic in the past. I think we left off with the idea that we should use SourceKit to generate the list of tests (+cc Daniel Dunbar). This work has largely been abandoned. If you'd like to pick it up, you could have huge impact!

I'm not sure I understand your first two points. Are you suggesting that we implement some method of kicking off tests on an iOS simulator, but only when corelibs-xctest framework is linked with the application bundle...? I'd appreciate it if you could explain what you mean in more detail.

- Brian Gesiak

On Fri, Dec 2, 2016 at 5:57 AM, Ilya Laryionau via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:
Hello Community,

I’m working on adding iOS platform support for SwiftXCTest. So I’ve have the following proposals and would like to hear feedback from the community.
SwiftXCTest runs the tests in a way `XCTMain([ testCase(TestFoo.allTests) ])`. XCTMain supports only Linux / FreeBSD / macOS. So I suggest introducing an API that will provide a similar way of running tests as XCTMain, but in a case using SwiftXCTest as a framework e.g. on iOS. It could be the same logic that just returns exit code, when XCTMain just exits with this code.
The suggestion above (number 1) is good, but too simple for using SwiftXCTest as a framework, since we only have exit code and debug logs. So my second proposal is providing mechanism of initialising root test suite with array of testCases (array of XCTestCaseEntry objects) with the same structure that XCTMain does. This isn’t possible now since XCTestCaseSuite class that is used to initialise root test suite (see XCTMain.swift line 80) is internal.
This proposal is because Swift has very poor reflection at the moment. So I suggest creating a generator (pre-build script) for Swift test cases that for example is able analysing concrete folder with swift files and generate e.g. TestCases.swift file. This script is intended to be used manually.
Looking forward to any feedback.

Ilya

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Ilya_Laryionau · December 6, 2016, 12:44pm

I suggest adding XCTestSuite.init method with the following signature: public init(name: String, testCaseEntries: [XCTestCaseEntry]) as for 1.

I suggest implementing API that allows creating XCTestSuite from a list of XCTestCaseEntries.

Ilya

···

On 6 Dec 2016, at 12:50, Ilya Laryionau <larryonoff@gmail.com> wrote:

Brian,

Thank you for your feedback.

I need some time to think about the API for the first and the seconds points. I’m going to create it as a list of protocol in gist.

Ilya

On 5 Dec 2016, at 20:50, Brian Gesiak <modocache@gmail.com <mailto:modocache@gmail.com>> wrote:

Ilya,

I would love to see both of your suggestions implemented, but you may need to submit a swift-evolution proposal for them. I think Apple employees maintaining swift-corelibs-xctest would have a better idea of whether your proposals would be accepted by the core team. (+cc Brian Croom).

I think a good next step would be to reply to this email with the function signatures for the additions you're proposing. That will allow us to discuss them in more concrete terms.

Thanks for all the great suggestions!

- Brian Gesiak

On Mon, Dec 5, 2016 at 1:30 AM, Ilya Laryionau <larryonoff@gmail.com <mailto:larryonoff@gmail.com>> wrote:
Brian,

I appreciate your comments.

You should check out https://bugs.swift.org/browse/SR-710\. There's been a great deal of discussion on this topic in the past. I think we left off with the idea that we should use SourceKit to generate the list of tests (+cc Daniel Dunbar). This work has largely been abandoned. If you'd like to pick it up, you could have huge impact!

This’s a great comment. I’ll check what I can do.

I'm not sure I understand your first two points. Are you suggesting that we implement some method of kicking off tests on an iOS simulator, but only when corelibs-xctest framework is linked with the application bundle...? I'd appreciate it if you could explain what you mean in more detail.

Let me clarify my first two points.

In general you’re correct. I suggest implementing some method that triggers the same logic that XCTestMain does. But instead of calling exit, when the work is done, returning the exit code.
I suggest implementing API that allows creating XCTestSuite from a list of XCTestCaseEntries.

Ilya

On 2 Dec 2016, at 21:00, Brian Gesiak <modocache@gmail.com <mailto:modocache@gmail.com>> wrote:

Hey Ilya,

I'm glad you're thinking about how to use corelibs-xctest on mobile devices. I'd like to prepare it for Android as well, and I think there's a lot of overlap here.

> This proposal is because Swift has very poor reflection at the moment. So I suggest creating a generator (pre-build script) for Swift test cases that for example is able analysing concrete folder with swift files and generate e.g. TestCases.swift file. This script is intended to be used manually.

You should check out https://bugs.swift.org/browse/SR-710\. There's been a great deal of discussion on this topic in the past. I think we left off with the idea that we should use SourceKit to generate the list of tests (+cc Daniel Dunbar). This work has largely been abandoned. If you'd like to pick it up, you could have huge impact!

I'm not sure I understand your first two points. Are you suggesting that we implement some method of kicking off tests on an iOS simulator, but only when corelibs-xctest framework is linked with the application bundle...? I'd appreciate it if you could explain what you mean in more detail.

- Brian Gesiak

On Fri, Dec 2, 2016 at 5:57 AM, Ilya Laryionau via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:
Hello Community,

I’m working on adding iOS platform support for SwiftXCTest. So I’ve have the following proposals and would like to hear feedback from the community.
SwiftXCTest runs the tests in a way `XCTMain([ testCase(TestFoo.allTests) ])`. XCTMain supports only Linux / FreeBSD / macOS. So I suggest introducing an API that will provide a similar way of running tests as XCTMain, but in a case using SwiftXCTest as a framework e.g. on iOS. It could be the same logic that just returns exit code, when XCTMain just exits with this code.
The suggestion above (number 1) is good, but too simple for using SwiftXCTest as a framework, since we only have exit code and debug logs. So my second proposal is providing mechanism of initialising root test suite with array of testCases (array of XCTestCaseEntry objects) with the same structure that XCTMain does. This isn’t possible now since XCTestCaseSuite class that is used to initialise root test suite (see XCTMain.swift line 80) is internal.
This proposal is because Swift has very poor reflection at the moment. So I suggest creating a generator (pre-build script) for Swift test cases that for example is able analysing concrete folder with swift files and generate e.g. TestCases.swift file. This script is intended to be used manually.
Looking forward to any feedback.

Ilya

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev