I am deeply interested in finding solutions for allowing unit-tests of preconditions. Without them, I believe we are leaving many holes in our tests and coverage. The solution found in the Swift project of forking the process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition function with a function that calls a configurable closure which defaults to the original precondition function. It would be great if the Standard Library allowed this by default so that XCTest could use it to offer full support for precondition unit tests.
What would precondition() if the condition evaluates to false, and the
special handler is installed? precondition() can't return, since it
can't allow the original code to continue. The original code
certainly does not expect to continue (it can even do something
memory-unsafe, or force-unwrap an nil optional, or advance an index
past endIndex, etc.)
Dmitri
···
On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:
I am deeply interested in finding solutions for allowing unit-tests of preconditions. Without them, I believe we are leaving many holes in our tests and coverage. The solution found in the Swift project of forking the process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition function with a function that calls a configurable closure which defaults to the original precondition function. It would be great if the Standard Library allowed this by default so that XCTest could use it to offer full support for precondition unit tests.
+1 to this being an important problem to solve, but I'm not sure about the
specific solution. Are there performance or security impacts to production
code by having this? Could the forked-process solution be implemented once
in a framework and then everybody just uses it transparently? The
forked-process approach also has the advantage of catching crashes for any
other unexpected reason (eg. division by zero), and of ensuring that test
executions are hermetically sealed without data leaking between test
instances.
···
On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution < swift-evolution@swift.org> wrote:
I am deeply interested in finding solutions for allowing unit-tests of
preconditions. Without them, I believe we are leaving many holes in our
tests and coverage. The solution found in the Swift project of forking the
process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition
function with a function that calls a configurable closure which defaults
to the original precondition function. It would be great if the Standard
Library allowed this by default so that XCTest could use it to offer full
support for precondition unit tests.
And, for the case of fatalError(), the compiler wouldn't even allow
you to return from a custom handler (since fatalError() is @noreturn).
I don't see a way around this. If the code needs to stop, it will
make all sorts of assumptions about it (that it won't continue), and
so will the optimizer.
Dmitri
···
On Sat, Mar 12, 2016 at 4:26 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:
On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution > <swift-evolution@swift.org> wrote:
I am deeply interested in finding solutions for allowing unit-tests of preconditions. Without them, I believe we are leaving many holes in our tests and coverage. The solution found in the Swift project of forking the process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition function with a function that calls a configurable closure which defaults to the original precondition function. It would be great if the Standard Library allowed this by default so that XCTest could use it to offer full support for precondition unit tests.
What would precondition() if the condition evaluates to false, and the
special handler is installed? precondition() can't return, since it
can't allow the original code to continue. The original code
certainly does not expect to continue (it can even do something
memory-unsafe, or force-unwrap an nil optional, or advance an index
past endIndex, etc.)
I completely agree that the approach that uses process isolation is
the right way to go, not just for testing preconditions, but for all
other reasons you mention.
Dmitri
···
On Sat, Mar 12, 2016 at 4:28 PM, Jonathan Tang via swift-evolution <swift-evolution@swift.org> wrote:
On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution > <swift-evolution@swift.org> wrote:
I am deeply interested in finding solutions for allowing unit-tests of
preconditions. Without them, I believe we are leaving many holes in our
tests and coverage. The solution found in the Swift project of forking the
process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition
function with a function that calls a configurable closure which defaults to
the original precondition function. It would be great if the Standard
Library allowed this by default so that XCTest could use it to offer full
support for precondition unit tests.
+1 to this being an important problem to solve, but I'm not sure about the
specific solution. Are there performance or security impacts to production
code by having this? Could the forked-process solution be implemented once
in a framework and then everybody just uses it transparently? The
forked-process approach also has the advantage of catching crashes for any
other unexpected reason (eg. division by zero), and of ensuring that test
executions are hermetically sealed without data leaking between test
instances.
I’m unsure about this; a precondition should validate that a method has been called correctly, so if it fails then it may indicate a flaw with your test, so it makes sense for it to stop. You can use assert() to also test for edge-cases within your method that could have unexpected results (due to implementation details). The unit test meanwhile should just pass in valid input, and test for expected output. At least that’s how I try to do it, i.e- with a unit test doing black-box testing only, and assert() handling white-box testing (implementation details and edge cases that could affect them). This leaves the precondition to check input; if one fails then it indicates a very basic error that may invalidate your remaining tests anyway.
If you do want to handle them however, then it seems like a compiler option to replace them with thrown errors might make most sense; this would mean duplicating code with preconditions (or that calls such methods) so that there’s one path with thrown preconditions and one without, the compiler can then select which path to use based upon the call-site in your unit test (e.g- if you have a catch block for PreconditionError, or an attribute like @preconditionError or something), and discard any paths that aren’t used. It’s a complex option, but it would definitely work, and would only be enabled when testing by default.
···
On 13 Mar 2016, at 00:32, Dmitri Gribenko via swift-evolution <swift-evolution@swift.org> wrote:
On Sat, Mar 12, 2016 at 4:28 PM, Jonathan Tang via swift-evolution > <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution >> <swift-evolution@swift.org> wrote:
I am deeply interested in finding solutions for allowing unit-tests of
preconditions. Without them, I believe we are leaving many holes in our
tests and coverage. The solution found in the Swift project of forking the
process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition
function with a function that calls a configurable closure which defaults to
the original precondition function. It would be great if the Standard
Library allowed this by default so that XCTest could use it to offer full
support for precondition unit tests.
+1 to this being an important problem to solve, but I'm not sure about the
specific solution. Are there performance or security impacts to production
code by having this? Could the forked-process solution be implemented once
in a framework and then everybody just uses it transparently? The
forked-process approach also has the advantage of catching crashes for any
other unexpected reason (eg. division by zero), and of ensuring that test
executions are hermetically sealed without data leaking between test
instances.
I completely agree that the approach that uses process isolation is
the right way to go, not just for testing preconditions, but for all
other reasons you mention.
Okay, perhaps this is not the optimal solution. But I tried to rally enthusiasm for finding a solution using forks a few months back and the complexity of the problem seemed to stall the progress. The issue is important enough for me that I tried to come at it from a different angle. If be more than happy if it was implemented using forks.
···
On 13 Mar 2016, at 01:28, Jonathan Tang <jonathan.d.tang@gmail.com> wrote:
On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:
I am deeply interested in finding solutions for allowing unit-tests of preconditions. Without them, I believe we are leaving many holes in our tests and coverage. The solution found in the Swift project of forking the process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition function with a function that calls a configurable closure which defaults to the original precondition function. It would be great if the Standard Library allowed this by default so that XCTest could use it to offer full support for precondition unit tests.
+1 to this being an important problem to solve, but I'm not sure about the specific solution. Are there performance or security impacts to production code by having this? Could the forked-process solution be implemented once in a framework and then everybody just uses it transparently? The forked-process approach also has the advantage of catching crashes for any other unexpected reason (eg. division by zero), and of ensuring that test executions are hermetically sealed without data leaking between test instances.
The forked-process approach also has the advantage of catching crashes for any other unexpected reason (eg. division by zero), and of ensuring that test executions are hermetically sealed without data leaking between test instances.
If by "forked-process approach" you mean putting code like this in XCTest:
func XCTAssertPreconditionsPass(_ expression: Void -> Void, _ message: String) {
let childPid = fork()
switch childPid {
case -1:
XCTFail("Forking during test \(message) failed")
case 0:
expression()
exit(0)
default:
var status: Int32 = 0
guard waitpid(childPid, &status, 0) > 0 else {
fatalError("XCTAssertPreconditionsPass waitpid() failed!")
}
if status != 0 {
XCTFail(message)
}
}
}
I don't think that's going to work in practice. Apple Foundation (and I believe CF as well) don't support forking unless you immediately exec() <http://lists.apple.com/archives/cocoa-dev/2007/Oct/msg01237.html>\. Even if XCTest could communicate with the subprocesses without using Foundation, many of the tests themselves would use Foundation; having tests fail because of the test harness's implementation is probably not acceptable.
It might be possible to run whole tests, rather than individual assertions, by fork/exec-ing our harness with a command-line flag indicating the test to run, but we would need a way to mark the tests which ought to be treated this way. (Or we could run every test in a sub-process, but that seems like it would introduce significant overhead.)
With the assistance of the compiler, we might be able to have precondition() and friends throw C++ exceptions in builds with testability enabled, and then have XCTest call into some C++ glue to wrap a try/catch block around the expression. This would leave a trail of leaked objects and broken invariants in its wake, but the damage might not matter for many simple tests.
Ultimately, I think what this proposal is about is catching universal errors <https://github.com/apple/swift/blob/master/docs/ErrorHandlingRationale.rst#universal-errors>\. There's no mechanism to do that in Swift right now, but there *is* a general understanding that we'll probably need something eventually. We may not have a better option than to defer precondition testing until we have those.
I don't agree because as soon as you have complicated preconditions, you will have bugs in them which need unit testing. As recently as Friday, the approach on Stack Overflow allowed me to test and uncover several bugs/holes in my preconditions.
···
On 13 Mar 2016, at 09:40, Haravikk via swift-evolution <swift-evolution@swift.org> wrote:
I’m unsure about this; a precondition should validate that a method has been called correctly, so if it fails then it may indicate a flaw with your test, so it makes sense for it to stop. You can use assert() to also test for edge-cases within your method that could have unexpected results (due to implementation details). The unit test meanwhile should just pass in valid input, and test for expected output. At least that’s how I try to do it, i.e- with a unit test doing black-box testing only, and assert() handling white-box testing (implementation details and edge cases that could affect them). This leaves the precondition to check input; if one fails then it indicates a very basic error that may invalidate your remaining tests anyway.
If you do want to handle them however, then it seems like a compiler option to replace them with thrown errors might make most sense; this would mean duplicating code with preconditions (or that calls such methods) so that there’s one path with thrown preconditions and one without, the compiler can then select which path to use based upon the call-site in your unit test (e.g- if you have a catch block for PreconditionError, or an attribute like @preconditionError or something), and discard any paths that aren’t used. It’s a complex option, but it would definitely work, and would only be enabled when testing by default.
On 13 Mar 2016, at 00:32, Dmitri Gribenko via swift-evolution <swift-evolution@swift.org> wrote:
On Sat, Mar 12, 2016 at 4:28 PM, Jonathan Tang via swift-evolution >> <swift-evolution@swift.org> wrote:
On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution >>> <swift-evolution@swift.org> wrote:
I am deeply interested in finding solutions for allowing unit-tests of
preconditions. Without them, I believe we are leaving many holes in our
tests and coverage. The solution found in the Swift project of forking the
process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition
function with a function that calls a configurable closure which defaults to
the original precondition function. It would be great if the Standard
Library allowed this by default so that XCTest could use it to offer full
support for precondition unit tests.
+1 to this being an important problem to solve, but I'm not sure about the
specific solution. Are there performance or security impacts to production
code by having this? Could the forked-process solution be implemented once
in a framework and then everybody just uses it transparently? The
forked-process approach also has the advantage of catching crashes for any
other unexpected reason (eg. division by zero), and of ensuring that test
executions are hermetically sealed without data leaking between test
instances.
I completely agree that the approach that uses process isolation is
the right way to go, not just for testing preconditions, but for all
other reasons you mention.
I think the forking approach is correct and would like to see more tooling support for that path in test runners.
···
On Mar 13, 2016, at 5:20 AM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:
Okay, perhaps this is not the optimal solution. But I tried to rally enthusiasm for finding a solution using forks a few months back and the complexity of the problem seemed to stall the progress. The issue is important enough for me that I tried to come at it from a different angle. If be more than happy if it was implemented using forks.
On 13 Mar 2016, at 01:28, Jonathan Tang <jonathan.d.tang@gmail.com> wrote:
On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:
I am deeply interested in finding solutions for allowing unit-tests of preconditions. Without them, I believe we are leaving many holes in our tests and coverage. The solution found in the Swift project of forking the process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition function with a function that calls a configurable closure which defaults to the original precondition function. It would be great if the Standard Library allowed this by default so that XCTest could use it to offer full support for precondition unit tests.
+1 to this being an important problem to solve, but I'm not sure about the specific solution. Are there performance or security impacts to production code by having this? Could the forked-process solution be implemented once in a framework and then everybody just uses it transparently? The forked-process approach also has the advantage of catching crashes for any other unexpected reason (eg. division by zero), and of ensuring that test executions are hermetically sealed without data leaking between test instances.
It might be possible to run whole tests, rather than individual assertions, by fork/exec-ing our harness
This is exactly what is being proposed.
with a command-line flag indicating the test to run, but we would need a way to mark the tests which ought to be treated this way. (Or we could run every test in a sub-process, but that seems like it would introduce significant overhead.)
I think we should measure first and then make conclusions. By the
way, I highly recommend everyone to take a look at StdlibUnittest,
where this approach is already implemented and works well for testing
preconditions in the standard library. StdlibUnittest just keeps a
worker process running, asking it to run tests one by one until it
crashes, and then starts a new one.
With the assistance of the compiler, we might be able to have precondition() and friends throw C++ exceptions in builds with testability enabled,
That would immediately lead to undefined behavior, not just leaks. If
the code does not expect to continue, then the optimizer will make all
sorts of assumptions about that, and we don't want to inflict
debugging that mess onto our users.
Dmitri
···
On Sun, Mar 13, 2016 at 3:32 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:
I think keeping the code running in a separate process with probes may be something really useful in the unit testing toolbox for sure. How far did your experiment/proposal with that went?
···
Sent from my iPhone
On 13 Mar 2016, at 14:27, Step C via swift-evolution <swift-evolution@swift.org> wrote:
I think the forking approach is correct and would like to see more tooling support for that path in test runners.
On Mar 13, 2016, at 5:20 AM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:
Okay, perhaps this is not the optimal solution. But I tried to rally enthusiasm for finding a solution using forks a few months back and the complexity of the problem seemed to stall the progress. The issue is important enough for me that I tried to come at it from a different angle. If be more than happy if it was implemented using forks.
On 13 Mar 2016, at 01:28, Jonathan Tang <jonathan.d.tang@gmail.com> wrote:
On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:
I am deeply interested in finding solutions for allowing unit-tests of preconditions. Without them, I believe we are leaving many holes in our tests and coverage. The solution found in the Swift project of forking the process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition function with a function that calls a configurable closure which defaults to the original precondition function. It would be great if the Standard Library allowed this by default so that XCTest could use it to offer full support for precondition unit tests.
+1 to this being an important problem to solve, but I'm not sure about the specific solution. Are there performance or security impacts to production code by having this? Could the forked-process solution be implemented once in a framework and then everybody just uses it transparently? The forked-process approach also has the advantage of catching crashes for any other unexpected reason (eg. division by zero), and of ensuring that test executions are hermetically sealed without data leaking between test instances.
As I'm not technically knowledgable enough in low-level process handling and about the intricacies of xctest, I can't write a proposal that stands on it's own feet. All I can do is talk about it until someone that is can work on it.
···
On 13 Mar 2016, at 20:36, Goffredo Marocchi <panajev@gmail.com> wrote:
I think keeping the code running in a separate process with probes may be something really useful in the unit testing toolbox for sure. How far did your experiment/proposal with that went?
Sent from my iPhone
On 13 Mar 2016, at 14:27, Step C via swift-evolution <swift-evolution@swift.org> wrote:
I think the forking approach is correct and would like to see more tooling support for that path in test runners.
On Mar 13, 2016, at 5:20 AM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:
Okay, perhaps this is not the optimal solution. But I tried to rally enthusiasm for finding a solution using forks a few months back and the complexity of the problem seemed to stall the progress. The issue is important enough for me that I tried to come at it from a different angle. If be more than happy if it was implemented using forks.
On 13 Mar 2016, at 01:28, Jonathan Tang <jonathan.d.tang@gmail.com> wrote:
On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:
I am deeply interested in finding solutions for allowing unit-tests of preconditions. Without them, I believe we are leaving many holes in our tests and coverage. The solution found in the Swift project of forking the process seems fairly complicated to implement in XCTest.
I found a solution online that works by overriding the precondition function with a function that calls a configurable closure which defaults to the original precondition function. It would be great if the Standard Library allowed this by default so that XCTest could use it to offer full support for precondition unit tests.
+1 to this being an important problem to solve, but I'm not sure about the specific solution. Are there performance or security impacts to production code by having this? Could the forked-process solution be implemented once in a framework and then everybody just uses it transparently? The forked-process approach also has the advantage of catching crashes for any other unexpected reason (eg. division by zero), and of ensuring that test executions are hermetically sealed without data leaking between test instances.