Testing Assertions


(Mohamed Ebrahim Afifi) #1

Right now, we cannot easily test failed assertions (assert,
assertionFailure, precondition, preconditionFailure, fatalError) in our own
code without some hacks in the code under test. Like this example
https://github.com/mohamede1945/AssertionsTestingExample

So, my suggestion is to add for XCTest something very similar to
*expectCrashLater* that is defined here
https://github.com/apple/swift/blob/9b15d03b73b9e8a6dbd3f71b5c78660a359e8e26/stdlib/private/StdlibUnittest/StdlibUnittest.swift.gyb
and used in tests like this example
https://github.com/apple/swift/blob/master/validation-test/stdlib/Assert.swift

What do you think?

Best Regards,
Mohamed Afifi


(Tony Parker) #2

Hi Mohamed,

I agree it’s very difficult to test assertions in XCTest today. This approach looks interesting, but I’m not sure how it’s possible to implement within XCTest’s current architecture. Do you have any idea how this would be implemented?

- Tony

···

On Dec 31, 2015, at 1:35 AM, Mohamed Ebrahim Afifi via swift-evolution <swift-evolution@swift.org> wrote:

Right now, we cannot easily test failed assertions (assert, assertionFailure, precondition, preconditionFailure, fatalError) in our own code without some hacks in the code under test. Like this example https://github.com/mohamede1945/AssertionsTestingExample

So, my suggestion is to add for XCTest something very similar to expectCrashLater that is defined here https://github.com/apple/swift/blob/9b15d03b73b9e8a6dbd3f71b5c78660a359e8e26/stdlib/private/StdlibUnittest/StdlibUnittest.swift.gyb
and used in tests like this example https://github.com/apple/swift/blob/master/validation-test/stdlib/Assert.swift

What do you think?

Best Regards,
Mohamed Afifi
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jerome Duquennoy) #3

It is true that testing an assert is not really possible, as it basically crashes the app.

But we have to take into account that the behaviour of the assert method is not strait-forward : it depends on what is the optimisation level. Here is an extract of the inline doc of the assert method :

* In playgrounds and -Onone builds (the default for Xcode's Debug
  configuration): if `condition` evaluates to false, stop program
  execution in a debuggable state after printing `message`.
* In -O builds (the default for Xcode's Release configuration),
  `condition` is not evaluated, and there are no effects.
* In -Ounchecked builds, `condition` is not evaluated, but the
  optimizer may assume that it *would* evaluate to `true`. Failure
  to satisfy that assumption in -Ounchecked builds is a serious
  programming error.

I have the feeling that assertions are not meant to be tested, as they are not meant to be executed in production code.
And I have to add that writing tests that would not behave the same depending on the optimisation level would make me feel rather uncomfortable.

Maybe throwing an error would be more adapted for a validation that must also occur on production builds ?
Then, testing it would be pretty easy, with an extension to XCTestCase like that :

extension XCTestCase {

  func XCTAssertThrows(file: String = __FILE__, line: UInt = __LINE__, _ closure:() throws -> Void) {
    do {
      try closure();
      XCTFail("Closure did not throw an error", file: file, line: line);
    } catch {
      // expected, nothing to do
    }
  }
  
  func XCTAssertNoThrow<T>(file: String = __FILE__, line: UInt = __LINE__, _ closure:() throws -> T) -> T? {
    do {
      return try closure();
    } catch let error {
      XCTFail("Closure throw unexpected error \(error)", file: file, line: line);
    }
    return nil;
  }
  
}

Jérôme

···

On 05 Jan 2016, at 01:42, Tony Parker via swift-evolution <swift-evolution@swift.org> wrote:

Hi Mohamed,

I agree it’s very difficult to test assertions in XCTest today. This approach looks interesting, but I’m not sure how it’s possible to implement within XCTest’s current architecture. Do you have any idea how this would be implemented?

- Tony

On Dec 31, 2015, at 1:35 AM, Mohamed Ebrahim Afifi via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Right now, we cannot easily test failed assertions (assert, assertionFailure, precondition, preconditionFailure, fatalError) in our own code without some hacks in the code under test. Like this example https://github.com/mohamede1945/AssertionsTestingExample

So, my suggestion is to add for XCTest something very similar to expectCrashLater that is defined here https://github.com/apple/swift/blob/9b15d03b73b9e8a6dbd3f71b5c78660a359e8e26/stdlib/private/StdlibUnittest/StdlibUnittest.swift.gyb
and used in tests like this example https://github.com/apple/swift/blob/master/validation-test/stdlib/Assert.swift

What do you think?

Best Regards,
Mohamed Afifi
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution