URLSession test fest!

Development Core Libraries
1

Hi Tony,

First post and excited to join the party.

I'm sure this may have been solved many times over by the community already. Maybe we can collect some ideas on this. In the spirit of a small step to get things moving forward I'm sharing my own approach which has proven very effective for making URLSession extremely testable and it's amazingly little code with zero weirdness. All credit for the technique comes from Michael Feathers "Working Effectively with Legacy Code"

Here's the Gist:

And an explanation:

Leveraging the 'Subclass and Override Method' Technique I create a subclass e.g. 'NSURLSessionTestable'. I then expose any hidden dependancies using NSURLSession.sharedSession() in the code and make this dependancy injected to create a seam. Under test we are then able to substitute uses of sharedSession with an instance of NSURLSessionTestable.

NSURLSessionTestable is trivial in that we simply override dataTaskWithRequest() -> NSURLSessionDataTask()

The returned instance of NSURLSessionDataTask() does nothing. Instead we immediately invoke the completionHandler returning local instance variables of Data & URLResponse - these are provided by the test.

With this approach you have completely removed any network calls and asynchronous behaviour. This leads to extremely concise easy to setup and run tests.

It's so trivial I wonder if it would even be worth adding to the corelib - I'm personally a little wary of special 'knobs and dials' intended for making it more testable that could be hard to find or worse become used for doing odd stuff in production.

I've never seen any explicit 'Testable' classes so this would certainly mark a break from tradition but I personally have found the approach to work extremely well and through use of an explicitly named class it's intention revealing + easily discovered.

- Paul

2

Hi Paul,

Hi Tony,

First post and excited to join the party.

Great, welcome!

I'm sure this may have been solved many times over by the community already. Maybe we can collect some ideas on this. In the spirit of a small step to get things moving forward I'm sharing my own approach which has proven very effective for making URLSession extremely testable and it's amazingly little code with zero weirdness. All credit for the technique comes from Michael Feathers "Working Effectively with Legacy Code"

Here's the Gist:

Testable NSURLSession using the 'Subclass and Override Method' technique. · GitHub

And an explanation:

Leveraging the 'Subclass and Override Method' Technique I create a subclass e.g. 'NSURLSessionTestable'. I then expose any hidden dependancies using NSURLSession.sharedSession() in the code and make this dependancy injected to create a seam. Under test we are then able to substitute uses of sharedSession with an instance of NSURLSessionTestable.

NSURLSessionTestable is trivial in that we simply override dataTaskWithRequest() -> NSURLSessionDataTask()

The returned instance of NSURLSessionDataTask() does nothing. Instead we immediately invoke the completionHandler returning local instance variables of Data & URLResponse - these are provided by the test.

With this approach you have completely removed any network calls and asynchronous behaviour. This leads to extremely concise easy to setup and run tests.

It's so trivial I wonder if it would even be worth adding to the corelib - I'm personally a little wary of special 'knobs and dials' intended for making it more testable that could be hard to find or worse become used for doing odd stuff in production.

I've never seen any explicit 'Testable' classes so this would certainly mark a break from tradition but I personally have found the approach to work extremely well and through use of an explicitly named class it's intention revealing + easily discovered.

- Paul
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

This all sounds very interesting. I think that ‘testable’ subclasses would be totally reasonable as part of this test tool or our testing framework.

I have had some challenges in the past trying to carefully manage the public API surface vs. what is required for testing. This is easier in ObjC, where there are a lot of interesting hacks one can do to get at internal stuff. =) If an outcome of this is that we can figure out how to leverage @testable for ourselves in swift-corelibs-foundation then that would be awesome as well.

Looking forward to seeing more ideas,

- Tony

···

On Jan 27, 2017, at 12:51 AM, Paul Stringer via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

3

Hi all, another new person here. This sounded like a lot of fun so I figured I would try and pitch in! However, I’m unable to get off the ground getting the current tests running. I’m sure it’s something I’m doing wrong, since I am completely new to the project, but I’d love it if anyone had any suggestions to get un-stuck.

I poked around in JIRA and there are at least two other tickets that have been filed describing the issue I also am having:

[SR-3703] Unable to run Foundation tests on master · Issue #4291 · apple/swift-corelibs-foundation · GitHub [I commented here with the specifics of my system, if you think that is important]
[SR-3775] Cannot build Foundation on macOS on master or swift-3.1-branch · Issue #4424 · apple/swift-corelibs-foundation · GitHub

Perhaps I am wrong, but it should be possible to use macOS to contribute tests for URLSession as described here, right? If so, does anyone have any suggestions for tackling the problem described in those two tickets? Thanks!

Cheers,

Peter

···

On Jan 27, 2017, at 6:31 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Hi Paul,

On Jan 27, 2017, at 12:51 AM, Paul Stringer via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Hi Tony,

First post and excited to join the party.

Great, welcome!

4

Hi Peter,

Hi all, another new person here.

Perfect, the more the merrier.

This sounded like a lot of fun so I figured I would try and pitch in! However, I’m unable to get off the ground getting the current tests running. I’m sure it’s something I’m doing wrong, since I am completely new to the project, but I’d love it if anyone had any suggestions to get un-stuck.

I poked around in JIRA and there are at least two other tickets that have been filed describing the issue I also am having:

[SR-3703] Unable to run Foundation tests on master · Issue #4291 · apple/swift-corelibs-foundation · GitHub [I commented here with the specifics of my system, if you think that is important]
[SR-3775] Cannot build Foundation on macOS on master or swift-3.1-branch · Issue #4424 · apple/swift-corelibs-foundation · GitHub

Perhaps I am wrong, but it should be possible to use macOS to contribute tests for URLSession as described here, right? If so, does anyone have any suggestions for tackling the problem described in those two tickets? Thanks!

Yup, we want building and running on macOS to be totally supported (so it’s easy to help contribute).

I’ll take a look and see if I can figure out what the problem is too…

- Tony

···

On Jan 29, 2017, at 11:05 AM, Peter Tomaselli via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Cheers,

Peter

On Jan 27, 2017, at 6:31 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Hi Paul,

On Jan 27, 2017, at 12:51 AM, Paul Stringer via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Hi Tony,

First post and excited to join the party.

Great, welcome!

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

5

I see what this problem is. I’ll have a PR up shortly to fix it.

- Tony

···

On Jan 30, 2017, at 2:33 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Hi Peter,

On Jan 29, 2017, at 11:05 AM, Peter Tomaselli via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Hi all, another new person here.

Perfect, the more the merrier.

This sounded like a lot of fun so I figured I would try and pitch in! However, I’m unable to get off the ground getting the current tests running. I’m sure it’s something I’m doing wrong, since I am completely new to the project, but I’d love it if anyone had any suggestions to get un-stuck.

I poked around in JIRA and there are at least two other tickets that have been filed describing the issue I also am having:

[SR-3703] Unable to run Foundation tests on master · Issue #4291 · apple/swift-corelibs-foundation · GitHub [I commented here with the specifics of my system, if you think that is important]
[SR-3775] Cannot build Foundation on macOS on master or swift-3.1-branch · Issue #4424 · apple/swift-corelibs-foundation · GitHub

Perhaps I am wrong, but it should be possible to use macOS to contribute tests for URLSession as described here, right? If so, does anyone have any suggestions for tackling the problem described in those two tickets? Thanks!

Yup, we want building and running on macOS to be totally supported (so it’s easy to help contribute).

I’ll take a look and see if I can figure out what the problem is too…

- Tony

Cheers,

Peter

On Jan 27, 2017, at 6:31 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Hi Paul,

On Jan 27, 2017, at 12:51 AM, Paul Stringer via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Hi Tony,

First post and excited to join the party.

Great, welcome!

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

6

Rad, thanks Tony. I can run all the tests now. Hoping to have some time soon to dig in and see if I can make a contribution. Love that this thing is spinning up its own HTTP server for this — seems real fun.

Cheers,

Peter

···

On Jan 30, 2017, at 5:46 PM, Tony Parker <anthony.parker@apple.com> wrote:

I see what this problem is. I’ll have a PR up shortly to fix it.

- Tony

7

Following up on this…

I’m wondering if anyone has any info on what happened with the (rather more elaborate) “loopback” server and set of tests from @danieleggert’s PR #299:

Is there some reason we ended up with a somewhat cut-down version of those (I understand that perhaps the URLSession implementation itself from this PR was not used, but perhaps we could pull some of the tests forward, or the more flexible loopback implementation) in master? Sorry if this is a silly question — I’m not very well-versed in the more low-level networking stuff involved here so there may be some obvious reason these are not suitable that I am missing.

Mailing list user Mike Ferenduros also put together what to my eye are some pretty comprehensive tests that have not ever seemed to be PR’d or merged, described here:

https://lists.swift.org/pipermail/swift-corelibs-dev/Week-of-Mon-20160912/000943.html

Seems like there was some concern over the use of httpbin.org <http://httpbin.org/&gt;, but these could be perhaps pulled in and simply redirected to hit the loopback, as the rest seems pretty well thought-out.

Just wondering if anyone has any of the background information here. I started writing some tests and loopback enhancements of my own but these other two gentlemen are way ahead of me, so maybe we should grab what we can from them, if it is possible to do so! Happy to take a crack at that if it seems like a sane idea.

Cheers,

Peter

···

On Jan 31, 2017, at 8:45 AM, Peter Tomaselli <vast.grapes@gmail.com> wrote:

Rad, thanks Tony. I can run all the tests now. Hoping to have some time soon to dig in and see if I can make a contribution. Love that this thing is spinning up its own HTTP server for this — seems real fun.

Cheers,

Peter

On Jan 30, 2017, at 5:46 PM, Tony Parker <anthony.parker@apple.com> wrote:

I see what this problem is. I’ll have a PR up shortly to fix it.

- Tony

8

Hi Peter,

I'll try to answer your questions to some extent!

I’m wondering if anyone has any info on what happened with the (rather more elaborate) “loopback” server and set of tests from @danieleggert’s PR #299:

https://github.com/apple/swift-corelibs-foundation/pull/299

Is there some reason we ended up with a somewhat cut-down version of those (I understand that perhaps the URLSession implementation itself from this PR was not used, but perhaps we could pull some of the tests forward, or the more flexible loopback implementation) in master? Sorry if this is a silly question — I’m not very well-versed in the more low-level networking stuff involved here so there may be some obvious reason these are not suitable that I am missing.

Yes, PR 299 from Daniel Eggert had a much mature loopback server and a better test coverage. However, it used functionality from the internal classes in the NSURLSession package (HTTPBodySource and friends), for which we needed to used a @testable import in TestFoundation. If I remember our discussions right, to support a @testable import you'd need to do build Foundation with the -enable-testing flag, which may not acceptable for Release builds. Of course, there was the possibility of rewriting this functionality (from the internal classes) all over again. But we decided to start with a simple loopback server and then build on it.

Mailing list user Mike Ferenduros also put together what to my eye are some pretty comprehensive tests that have not ever seemed to be PR’d or merged, described here:

https://lists.swift.org/pipermail/swift-corelibs-dev/Week-of-Mon-20160912/000943.html

>> Yes, I am aware of the extensive tests Mike Ferenduros has written. In my opinion, it will be great to have them contributed once we have a more mature loopback server infrastructure. Mike has also opened a dozen URLSession bugs!

Seems like there was some concern over the use of httpbin.org, but these could be perhaps pulled in and simply redirected to hit the loopback, as the rest seems pretty well thought-out.

The problem we have had with external URLs is the delays and intermittent failures which keep breaking the CI builds, causing much agony to many! Can you tell me more about this solution you propose?

Thanks!

Pushkar N Kulkarni,

IBM Runtimes

Simplicity is prerequisite for reliability - Edsger W. Dijkstra

I’m wondering if anyone has any info on what happened with the (rather more elaborate) “loopback” server and set of tests from @danieleggert’s PR #299:

https://github.com/apple/swift-corelibs-foundation/pull/299

Is there some reason we ended up with a somewhat cut-down version of those (I understand that perhaps the URLSession implementation itself from this PR was not used, but perhaps we could pull some of the tests forward, or the more flexible loopback implementation) in master? Sorry if this is a silly question — I’m not very well-versed in the more low-level networking stuff involved here so there may be some obvious reason these are not suitable that I am missing.

Mailing list user Mike Ferenduros also put together what to my eye are some pretty comprehensive tests that have not ever seemed to be PR’d or merged, described here:

https://lists.swift.org/pipermail/swift-corelibs-dev/Week-of-Mon-20160912/000943.html

Seems like there was some concern over the use of httpbin.org, but these could be perhaps pulled in and simply redirected to hit the loopback, as the rest seems pretty well thought-out.

Just wondering if anyone has any of the background information here. I started writing some tests and loopback enhancements of my own but these other two gentlemen are way ahead of me, so maybe we should grab what we can from them, if it is possible to do so! Happy to take a crack at that if it seems like a sane idea.

Cheers,

Peter

···

On Jan 31, 2017, at 8:45 AM, Peter Tomaselli vast.grapes@gmail.com wrote:

Rad, thanks Tony. I can run all the tests now. Hoping to have some time soon to dig in and see if I can make a contribution. Love that this thing is spinning up its own HTTP server for this — seems real fun.

Cheers,

Peter

On Jan 30, 2017, at 5:46 PM, Tony Parker anthony.parker@apple.com wrote:

I see what this problem is. I’ll have a PR up shortly to fix it.

  • Tony

swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

To: swift-corelibs-dev swift-corelibs-dev@swift.org
From: Peter Tomaselli via swift-corelibs-dev
Sent by: swift-corelibs-dev-bounces@swift.org
Date: 02/03/2017 08:11AM
Subject: Re: [swift-corelibs-dev] URLSession test fest!

Following up on this…

-----swift-corelibs-dev-bounces@swift.org wrote: -----

9

Hi Pushkar, thanks so much for the information! That does help to clear
everything up.

I'm afraid I don't have a very specific solution to propose. Only that, at
a high level, I think both of these earlier versions will serve as good
inspiration for new tests. For example, there are some test cases in both
of these earlier versions that I would never have thought of myself.

While it's not exactly my area of expertise, the existing loopback server
is very interesting to me, so I hope to have some time soon to improve it a
bit, adding URLSession tests as I go.

Of course, it is slightly tricky when both the system-under-test and the
mechanism by which it is tested are under construction at the same time! :)

Cheers,

Peter

···

On Fri, Feb 3, 2017 at 1:44 AM, Pushkar N Kulkarni <pushkar.nk@in.ibm.com> wrote:

Hi Peter,

I'll try to answer your questions to some extent!

*I’m wondering if anyone has any info on what happened with the (rather
more elaborate) “loopback” server and set of tests from @danieleggert’s PR
#299:*

*Basic NSURLSession implementation by danieleggert · Pull Request #299 · apple/swift-corelibs-foundation · GitHub*
<https://github.com/apple/swift-corelibs-foundation/pull/299&gt;

*Is there some reason we ended up with a somewhat cut-down version of
those (I understand that perhaps the URLSession implementation itself from
this PR was not used, but perhaps we could pull some of the tests forward,
or the more flexible loopback implementation) in master? Sorry if this is a
silly question — I’m not very well-versed in the more low-level networking
stuff involved here so there may be some obvious reason these are not
suitable that I am missing.*

>> Yes, PR 299 from Daniel Eggert had a much mature loopback server and a
better test coverage. However, it used functionality from the internal
classes in the NSURLSession package (HTTPBodySource and friends), for which
we needed to used a @testable import
<Basic NSURLSession implementation by danieleggert · Pull Request #299 · apple/swift-corelibs-foundation · GitHub
TestFoundation. If I remember our discussions right, to support a
`@testable import` you'd need to do build Foundation with the
`-enable-testing` flag, which may not acceptable for Release builds. Of
course, there was the possibility of rewriting this functionality (from the
internal classes) all over again. But we decided to start with a simple
loopback server and then build on it.

*Mailing list user Mike Ferenduros also put together what to my eye are
some pretty comprehensive tests that have not ever seemed to be PR’d or
merged, described here:*

*[swift-corelibs-dev] URLSession tests*
<https://lists.swift.org/pipermail/swift-corelibs-dev/Week-of-Mon-20160912/000943.html&gt;

*>> *Yes, I am aware of the extensive tests Mike Ferenduros has written.
In my opinion, it will be great to have them contributed once we have a
more mature loopback server infrastructure. Mike has also opened a dozen
URLSession bugs!

*Seems like there was some concern over the use of httpbin.org
<http://httpbin.org>, but these could be perhaps pulled in and simply
redirected to hit the loopback, as the rest seems pretty well thought-out.*
>> The problem we have had with external URLs is the delays and
intermittent failures which keep breaking the CI builds, causing much agony
to many! Can you tell me more about this solution you propose?

Thanks!

Pushkar N Kulkarni,
IBM Runtimes

*Simplicity is prerequisite for reliability - Edsger W. Dijkstra*

-----swift-corelibs-dev-bounces@swift.org wrote: -----
To: swift-corelibs-dev <swift-corelibs-dev@swift.org>
From: Peter Tomaselli via swift-corelibs-dev
Sent by: swift-corelibs-dev-bounces@swift.org
Date: 02/03/2017 08:11AM
Subject: Re: [swift-corelibs-dev] URLSession test fest!

Following up on this…

I’m wondering if anyone has any info on what happened with the (rather
more elaborate) “loopback” server and set of tests from @danieleggert’s PR
#299:

Basic NSURLSession implementation by danieleggert · Pull Request #299 · apple/swift-corelibs-foundation · GitHub

Is there some reason we ended up with a somewhat cut-down version of those
(I understand that perhaps the URLSession implementation itself from this
PR was not used, but perhaps we could pull some of the tests forward, or
the more flexible loopback implementation) in master? Sorry if this is a
silly question — I’m not very well-versed in the more low-level networking
stuff involved here so there may be some obvious reason these are not
suitable that I am missing.

Mailing list user Mike Ferenduros also put together what to my eye are
some pretty comprehensive tests that have not ever seemed to be PR’d or
merged, described here:

The swift-corelibs-dev Archives
Week-of-Mon-20160912/000943.html

Seems like there was some concern over the use of httpbin.org, but these
could be perhaps pulled in and simply redirected to hit the loopback, as
the rest seems pretty well thought-out.

Just wondering if anyone has any of the background information here. I
started writing some tests and loopback enhancements of my own but these
other two gentlemen are way ahead of me, so maybe we should grab what we
can from them, if it is possible to do so! Happy to take a crack at that if
it seems like a sane idea.

Cheers,

Peter

On Jan 31, 2017, at 8:45 AM, Peter Tomaselli <vast.grapes@gmail.com> > wrote:

Rad, thanks Tony. I can run all the tests now. Hoping to have some time
soon to dig in and see if I can make a contribution. Love that this thing
is spinning up its own HTTP server for this — seems real fun.

Cheers,

Peter

On Jan 30, 2017, at 5:46 PM, Tony Parker <anthony.parker@apple.com> wrote:

I see what this problem is. I’ll have a PR up shortly to fix it.

- Tony

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev