URLSession test fest!


(Tony Parker) #1

Hi everyone,

As we’re wrapping up Swift 3.1, I think it’d be a great opportunity to have a quality focus area on one of our most important APIs: URLSession. If you’ve been wondering how to make a meaningful contribution, this is the perfect time.

It’d be great to greatly expand both unit test and higher level test coverage of URLSession. For testing that can’t really be part of a unit test and CI like poorly performing networks or esoteric configurations, it would also be nice to build a different kind of test tool. That tool would provide a lot of knobs for controlling a URLSession. We could make this tool part of the swift-corelibs-foundation project.

Is anyone interested in chipping in here? We don’t have to start big. Even small contributions could get us moving in the right direction.

Thanks,
- Tony


(Pushkar N Kulkarni) #2

That sounds like a great plan, Tony! I am keen on contributing.

We all know that current set of unit tests for URLSession are extremely trivial and nowhere close to it’s real-world use cases! We’ve tried using external/public API endpoints in the tests, but intermittent connection failures (network delays) broke the CI. I believe a stronger testing framework for URLSession will be provide support in making the implementation robust.

Thanks!

Pushkar N Kulkarni,

IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

Hi everyone,

As we’re wrapping up Swift 3.1, I think it’d be a great opportunity to have a quality focus area on one of our most important APIs: URLSession. If you’ve been wondering how to make a meaningful contribution, this is the perfect time.

It’d be great to greatly expand both unit test and higher level test coverage of URLSession. For testing that can’t really be part of a unit test and CI like poorly performing networks or esoteric configurations, it would also be nice to build a different kind of test tool. That tool would provide a lot of knobs for controlling a URLSession. We could make this tool part of the swift-corelibs-foundation project.

Is anyone interested in chipping in here? We don’t have to start big. Even small contributions could get us moving in the right direction.

Thanks,

  • Tony
···

swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

To: swift-corelibs-dev swift-corelibs-dev@swift.org
From: Tony Parker via swift-corelibs-dev
Sent by: swift-corelibs-dev-bounces@swift.org
Date: 01/25/2017 06:43AM
Subject: [swift-corelibs-dev] URLSession test fest!

-----swift-corelibs-dev-bounces@swift.org wrote: -----


(Pushkar N Kulkarni) #3

Hi Tony,

PRs 911 and 913 reminded me of this conversation you started.

The current unit testing infrastructure for URLSession may prove insufficient for future contributions. I’ve been looking at these tests written by Mike Ferenduros and they appear quite exhaustive! The only issue is that they connect to an external HTTP test service - http://httpbin.org.

A quick look at httpbin.org tells me it has sufficient functionality to aid in URLSession unit testing. Do you think it would make sense to emulate this functionality within TestFoundation?

Pushkar N Kulkarni,

IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

Hi everyone,

As we’re wrapping up Swift 3.1, I think it’d be a great opportunity to have a quality focus area on one of our most important APIs: URLSession. If you’ve been wondering how to make a meaningful contribution, this is the perfect time.

It’d be great to greatly expand both unit test and higher level test coverage of URLSession. For testing that can’t really be part of a unit test and CI like poorly performing networks or esoteric configurations, it would also be nice to build a different kind of test tool. That tool would provide a lot of knobs for controlling a URLSession. We could make this tool part of the swift-corelibs-foundation project.

Is anyone interested in chipping in here? We don’t have to start big. Even small contributions could get us moving in the right direction.

Thanks,

  • Tony
···

swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

To: swift-corelibs-dev swift-corelibs-dev@swift.org
From: Tony Parker via swift-corelibs-dev
Sent by: swift-corelibs-dev-bounces@swift.org
Date: 01/25/2017 01:13AM
Subject: [swift-corelibs-dev] URLSession test fest!

-----swift-corelibs-dev-bounces@swift.org wrote: -----


(Tony Parker) #4

Hi Pushkar,

Another option is to continue to use httpbin.org <http://httpbin.org/>, but make the tests that use that network service not part of the normal unit tests but runnable on demand by us as we develop. That means we don’t have to reimplement the whole thing but we at least have some tools at our disposal to help debug and diagnose issues. As we make fixes we can improve our local unit test server’s infrastructure to prevent regressions.

- Tony

···

On Mar 9, 2017, at 7:02 AM, Pushkar N Kulkarni <pushkar.nk@in.ibm.com> wrote:

Hi Tony,

PRs 911 and 913 reminded me of this conversation you started.

The current unit testing infrastructure for URLSession may prove insufficient for future contributions. I've been looking at these tests <https://github.com/mike-ferenduros/swift-corelibs-foundation/blob/more-urlsession-tests/TestFoundation/TestNSURLSession.swift> written by Mike Ferenduros and they appear quite exhaustive! The only issue is that they connect to an external HTTP test service - http://httpbin.org. <http://httpbin.org./>

A quick look at httpbin.org tells me it has sufficient functionality to aid in URLSession unit testing. Do you think it would make sense to emulate this functionality within TestFoundation?

Pushkar N Kulkarni,
IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

-----swift-corelibs-dev-bounces@swift.org <mailto:-----swift-corelibs-dev-bounces@swift.org> wrote: -----
To: swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>>
From: Tony Parker via swift-corelibs-dev
Sent by: swift-corelibs-dev-bounces@swift.org <mailto:swift-corelibs-dev-bounces@swift.org>
Date: 01/25/2017 01:13AM
Subject: [swift-corelibs-dev] URLSession test fest!

Hi everyone,

As we’re wrapping up Swift 3.1, I think it’d be a great opportunity to have a quality focus area on one of our most important APIs: URLSession. If you’ve been wondering how to make a meaningful contribution, this is the perfect time.

It’d be great to greatly expand both unit test and higher level test coverage of URLSession. For testing that can’t really be part of a unit test and CI like poorly performing networks or esoteric configurations, it would also be nice to build a different kind of test tool. That tool would provide a lot of knobs for controlling a URLSession. We could make this tool part of the swift-corelibs-foundation project.

Is anyone interested in chipping in here? We don’t have to start big. Even small contributions could get us moving in the right direction.

Thanks,
- Tony

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev


(Peter) #5

Just one outside data point on this: as someone who's been wanting to contribute to the test fest, I agree that being able to test against httpbin.org (or some other pre-existing server — as opposed to the loopback server) would make it much easier to make useful contributions.

Not that the loopback server is bad, but I trashed several attempts at a contribution due to lack of confidence about my changes. Specifically, with both "ends" of the test apparatus in flux (for example, having to add "features" to the loopback server just so that features of URLSession could be tested), I found it very hard to know if either end was correct.

So, just two cents from a lurker here, but this sounds like a great idea to me. Cheers,

Peter

···

On Mar 9, 2017, at 4:38 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Hi Pushkar,

Another option is to continue to use httpbin.org, but make the tests that use that network service not part of the normal unit tests but runnable on demand by us as we develop. That means we don’t have to reimplement the whole thing but we at least have some tools at our disposal to help debug and diagnose issues. As we make fixes we can improve our local unit test server’s infrastructure to prevent regressions.

- Tony

On Mar 9, 2017, at 7:02 AM, Pushkar N Kulkarni <pushkar.nk@in.ibm.com> wrote:

Hi Tony,

PRs 911 and 913 reminded me of this conversation you started.

The current unit testing infrastructure for URLSession may prove insufficient for future contributions. I've been looking at these tests written by Mike Ferenduros and they appear quite exhaustive! The only issue is that they connect to an external HTTP test service - http://httpbin.org.

A quick look at httpbin.org tells me it has sufficient functionality to aid in URLSession unit testing. Do you think it would make sense to emulate this functionality within TestFoundation?

Pushkar N Kulkarni,
IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

-----swift-corelibs-dev-bounces@swift.org wrote: -----
To: swift-corelibs-dev <swift-corelibs-dev@swift.org>
From: Tony Parker via swift-corelibs-dev
Sent by: swift-corelibs-dev-bounces@swift.org
Date: 01/25/2017 01:13AM
Subject: [swift-corelibs-dev] URLSession test fest!

Hi everyone,

As we’re wrapping up Swift 3.1, I think it’d be a great opportunity to have a quality focus area on one of our most important APIs: URLSession. If you’ve been wondering how to make a meaningful contribution, this is the perfect time.

It’d be great to greatly expand both unit test and higher level test coverage of URLSession. For testing that can’t really be part of a unit test and CI like poorly performing networks or esoteric configurations, it would also be nice to build a different kind of test tool. That tool would provide a lot of knobs for controlling a URLSession. We could make this tool part of the swift-corelibs-foundation project.

Is anyone interested in chipping in here? We don’t have to start big. Even small contributions could get us moving in the right direction.

Thanks,
- Tony

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev


(David Sweeris) #6

How hard it would be to fake enough of a web server to test URLSession against? Maybe involving the Swift Server APIs, once those are ready? They might even have the functionality to fake network issues for their own testing, which we should be testing against anyway to ensure that things fail correctly.

- Dave Sweeris

···

On Jan 25, 2017, at 2:02 AM, Pushkar N Kulkarni via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

That sounds like a great plan, Tony! I am keen on contributing.

We all know that current set of unit tests for URLSession are extremely trivial and nowhere close to it's real-world use cases! We've tried using external/public API endpoints in the tests, but intermittent connection failures (network delays) broke the CI. I believe a stronger testing framework for URLSession will be provide support in making the implementation robust.


(Pushkar N Kulkarni) #7

Peter: I agree. A stable testing infrastructure is absolutely necessary.

Tony: I am curious about where the sources for these tests (that use the external httpbin.org) will be placed? Will it makes sense to have them in TestFoundation and have a guard to exclude them during normal test execution?

Pushkar N Kulkarni,

IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

···

To: Tony Parker anthony.parker@apple.com
From: Peter Tomaselli vast.grapes@gmail.com
Date: 03/10/2017 04:47AM
Cc: Pushkar N Kulkarni pushkar.nk@in.ibm.com, swift-corelibs-dev swift-corelibs-dev@swift.org
Subject: Re: [swift-corelibs-dev] URLSession test fest!

Just one outside data point on this: as someone who’s been wanting to contribute to the test fest, I agree that being able to test against httpbin.org (or some other pre-existing server — as opposed to the loopback server) would make it much easier to make useful contributions.

Not that the loopback server is bad, but I trashed several attempts at a contribution due to lack of confidence about my changes. Specifically, with both “ends” of the test apparatus in flux (for example, having to add “features” to the loopback server just so that features of URLSession could be tested), I found it very hard to know if either end was correct.

So, just two cents from a lurker here, but this sounds like a great idea to me. Cheers,

Peter

On Mar 9, 2017, at 4:38 PM, Tony Parker via swift-corelibs-dev swift-corelibs-dev@swift.org wrote:

Hi Pushkar,

Another option is to continue to use httpbin.org, but make the tests that use that network service not part of the normal unit tests but runnable on demand by us as we develop. That means we don’t have to reimplement the whole thing but we at least have some tools at our disposal to help debug and diagnose issues. As we make fixes we can improve our local unit test server’s infrastructure to prevent regressions.

  • Tony

On Mar 9, 2017, at 7:02 AM, Pushkar N Kulkarni pushkar.nk@in.ibm.com wrote:

Hi Tony,

PRs 911 and 913 reminded me of this conversation you started.

The current unit testing infrastructure for URLSession may prove insufficient for future contributions. I’ve been looking at these tests written by Mike Ferenduros and they appear quite exhaustive! The only issue is that they connect to an external HTTP test service - http://httpbin.org.

A quick look at httpbin.org tells me it has sufficient functionality to aid in URLSession unit testing. Do you think it would make sense to emulate this functionality within TestFoundation?

Pushkar N Kulkarni,

IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

To: swift-corelibs-dev swift-corelibs-dev@swift.org
From: Tony Parker via swift-corelibs-dev
Sent by: swift-corelibs-dev-bounces@swift.org
Date: 01/25/2017 01:13AM
Subject: [swift-corelibs-dev] URLSession test fest!

Hi everyone,

As we’re wrapping up Swift 3.1, I think it’d be a great opportunity to have a quality focus area on one of our most important APIs: URLSession. If you’ve been wondering how to make a meaningful contribution, this is the perfect time.

It’d be great to greatly expand both unit test and higher level test coverage of URLSession. For testing that can’t really be part of a unit test and CI like poorly performing networks or esoteric configurations, it would also be nice to build a different kind of test tool. That tool would provide a lot of knobs for controlling a URLSession. We could make this tool part of the swift-corelibs-foundation project.

Is anyone interested in chipping in here? We don’t have to start big. Even small contributions could get us moving in the right direction.

Thanks,

  • Tony

swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

-----swift-corelibs-dev-bounces@swift.org wrote: -----


swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

-----Peter Tomaselli vast.grapes@gmail.com wrote: -----


(Tony Parker) #8

Peter: I agree. A stable testing infrastructure is absolutely necessary.

Tony: I am curious about where the sources for these tests (that use the external httpbin.org) will be placed? Will it makes sense to have them in TestFoundation and have a guard to exclude them during normal test execution?

Sure, or we could add a standalone tool like I mentioned at the top of this thread.

Either way, we should make it as easy as possible to enable the tests when you’re iterating on this code so you can verify the absence of regressions.

- Tony

···

On Mar 9, 2017, at 9:52 PM, Pushkar N Kulkarni <pushkar.nk@in.ibm.com> wrote:

Pushkar N Kulkarni,
IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

-----Peter Tomaselli <vast.grapes@gmail.com <mailto:vast.grapes@gmail.com>> wrote: -----
To: Tony Parker <anthony.parker@apple.com <mailto:anthony.parker@apple.com>>
From: Peter Tomaselli <vast.grapes@gmail.com <mailto:vast.grapes@gmail.com>>
Date: 03/10/2017 04:47AM
Cc: Pushkar N Kulkarni <pushkar.nk@in.ibm.com <mailto:pushkar.nk@in.ibm.com>>, swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>>
Subject: Re: [swift-corelibs-dev] URLSession test fest!

Just one outside data point on this: as someone who's been wanting to contribute to the test fest, I agree that being able to test against httpbin.org <http://httpbin.org/> (or some other pre-existing server — as opposed to the loopback server) would make it much easier to make useful contributions.

Not that the loopback server is bad, but I trashed several attempts at a contribution due to lack of confidence about my changes. Specifically, with both "ends" of the test apparatus in flux (for example, having to add "features" to the loopback server just so that features of URLSession could be tested), I found it very hard to know if either end was correct.

So, just two cents from a lurker here, but this sounds like a great idea to me. Cheers,

Peter

On Mar 9, 2017, at 4:38 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Hi Pushkar,

Another option is to continue to use httpbin.org <http://httpbin.org/>, but make the tests that use that network service not part of the normal unit tests but runnable on demand by us as we develop. That means we don’t have to reimplement the whole thing but we at least have some tools at our disposal to help debug and diagnose issues. As we make fixes we can improve our local unit test server’s infrastructure to prevent regressions.

- Tony

On Mar 9, 2017, at 7:02 AM, Pushkar N Kulkarni <pushkar.nk@in.ibm.com <mailto:pushkar.nk@in.ibm.com>> wrote:

Hi Tony,

PRs 911 and 913 reminded me of this conversation you started.

The current unit testing infrastructure for URLSession may prove insufficient for future contributions. I've been looking at these tests <https://github.com/mike-ferenduros/swift-corelibs-foundation/blob/more-urlsession-tests/TestFoundation/TestNSURLSession.swift> written by Mike Ferenduros and they appear quite exhaustive! The only issue is that they connect to an external HTTP test service - http://httpbin.org. <http://httpbin.org./>

A quick look at httpbin.org <http://httpbin.org/> tells me it has sufficient functionality to aid in URLSession unit testing. Do you think it would make sense to emulate this functionality within TestFoundation?

Pushkar N Kulkarni,
IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

-----swift-corelibs-dev-bounces@swift.org <mailto:-----swift-corelibs-dev-bounces@swift.org> wrote: -----
To: swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>>
From: Tony Parker via swift-corelibs-dev
Sent by: swift-corelibs-dev-bounces@swift.org <mailto:swift-corelibs-dev-bounces@swift.org>
Date: 01/25/2017 01:13AM
Subject: [swift-corelibs-dev] URLSession test fest!

Hi everyone,

As we’re wrapping up Swift 3.1, I think it’d be a great opportunity to have a quality focus area on one of our most important APIs: URLSession. If you’ve been wondering how to make a meaningful contribution, this is the perfect time.

It’d be great to greatly expand both unit test and higher level test coverage of URLSession. For testing that can’t really be part of a unit test and CI like poorly performing networks or esoteric configurations, it would also be nice to build a different kind of test tool. That tool would provide a lot of knobs for controlling a URLSession. We could make this tool part of the swift-corelibs-foundation project.

Is anyone interested in chipping in here? We don’t have to start big. Even small contributions could get us moving in the right direction.

Thanks,
- Tony

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev


(Pushkar N Kulkarni) #9

Thanks Tony. I revisited your first email in this regard and a standalone tool sounds like the better idea here, given we’d want to try all possible configurations of URLSession and also possibly subject it to some stress.

Pushkar N Kulkarni,

IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

Peter: I agree. A stable testing infrastructure is absolutely necessary.

Tony: I am curious about where the sources for these tests (that use the external httpbin.org) will be placed? Will it makes sense to have them in TestFoundation and have a guard to exclude them during normal test execution?

Sure, or we could add a standalone tool like I mentioned at the top of this thread.

Either way, we should make it as easy as possible to enable the tests when you’re iterating on this code so you can verify the absence of regressions.

  • Tony
···

On Mar 9, 2017, at 9:52 PM, Pushkar N Kulkarni pushkar.nk@in.ibm.com wrote:

Pushkar N Kulkarni,

IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

To: Tony Parker anthony.parker@apple.com
From: Peter Tomaselli vast.grapes@gmail.com
Date: 03/10/2017 04:47AM
Cc: Pushkar N Kulkarni pushkar.nk@in.ibm.com, swift-corelibs-dev swift-corelibs-dev@swift.org
Subject: Re: [swift-corelibs-dev] URLSession test fest!

Just one outside data point on this: as someone who’s been wanting to contribute to the test fest, I agree that being able to test against httpbin.org (or some other pre-existing server — as opposed to the loopback server) would make it much easier to make useful contributions.

Not that the loopback server is bad, but I trashed several attempts at a contribution due to lack of confidence about my changes. Specifically, with both “ends” of the test apparatus in flux (for example, having to add “features” to the loopback server just so that features of URLSession could be tested), I found it very hard to know if either end was correct.

So, just two cents from a lurker here, but this sounds like a great idea to me. Cheers,

Peter

On Mar 9, 2017, at 4:38 PM, Tony Parker via swift-corelibs-dev swift-corelibs-dev@swift.org wrote:

Hi Pushkar,

Another option is to continue to use httpbin.org, but make the tests that use that network service not part of the normal unit tests but runnable on demand by us as we develop. That means we don’t have to reimplement the whole thing but we at least have some tools at our disposal to help debug and diagnose issues. As we make fixes we can improve our local unit test server’s infrastructure to prevent regressions.

  • Tony

On Mar 9, 2017, at 7:02 AM, Pushkar N Kulkarni pushkar.nk@in.ibm.com wrote:

Hi Tony,

PRs 911 and 913 reminded me of this conversation you started.

The current unit testing infrastructure for URLSession may prove insufficient for future contributions. I’ve been looking at these tests written by Mike Ferenduros and they appear quite exhaustive! The only issue is that they connect to an external HTTP test service - http://httpbin.org.

A quick look at httpbin.org tells me it has sufficient functionality to aid in URLSession unit testing. Do you think it would make sense to emulate this functionality within TestFoundation?

Pushkar N Kulkarni,

IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

To: swift-corelibs-dev swift-corelibs-dev@swift.org
From: Tony Parker via swift-corelibs-dev
Sent by: swift-corelibs-dev-bounces@swift.org
Date: 01/25/2017 01:13AM
Subject: [swift-corelibs-dev] URLSession test fest!

Hi everyone,

As we’re wrapping up Swift 3.1, I think it’d be a great opportunity to have a quality focus area on one of our most important APIs: URLSession. If you’ve been wondering how to make a meaningful contribution, this is the perfect time.

It’d be great to greatly expand both unit test and higher level test coverage of URLSession. For testing that can’t really be part of a unit test and CI like poorly performing networks or esoteric configurations, it would also be nice to build a different kind of test tool. That tool would provide a lot of knobs for controlling a URLSession. We could make this tool part of the swift-corelibs-foundation project.

Is anyone interested in chipping in here? We don’t have to start big. Even small contributions could get us moving in the right direction.

Thanks,

  • Tony

swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

-----swift-corelibs-dev-bounces@swift.org wrote: -----


swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

-----Peter Tomaselli vast.grapes@gmail.com wrote: -----

To: Pushkar N Kulkarni pushkar.nk@in.ibm.com
From: Tony Parker
Sent by: anthony.parker@apple.com
Date: 03/11/2017 05:41AM
Cc: Peter Tomaselli vast.grapes@gmail.com, swift-corelibs-dev swift-corelibs-dev@swift.org
Subject: Re: [swift-corelibs-dev] URLSession test fest!

-----anthony.parker@apple.com wrote: -----