Swift-checkit: A library for validating common protocol semantics

I'd like to gauge the community's interest in developing a library for validating conformances to common protocols and algorithms which use them.

This all came about because I hit a bug recently: on one of my custom collections, incrementing startIndex by count returned nil, rather than the expected endIndex. I never noticed this before, because all of my real-world code incremented indices by +/-1, and never in steps larger than that. Nonetheless, there were expected, documented semantics and generic code was relying on that.

Swift's protocols aren't just bags of syntax; there is some expected behaviour attached to each of them. Sometimes those expectations can be reflected in the type-system, but sometimes they have to be explained in documentation and left to the programmer. That's fine - but as software becomes more complex, it's easy to make changes which accidentally violate some subtle semantics of the protocol. That's why we write tests.

A recent post on these forums brought attention to some of the implicit relationships between FloatingPoint properties that are described in IEEE-754 but can't be enforced statically. Others on that thread noted important edge-cases such as infinite recursion while converting -0, which other developers might not think of testing. Of course, there's also the infamous case of Sequence, which is documented as being a single-pass abstraction; but given how you almost never encounter a sequence which breaks with multiple passes in the wild, very few generic algorithms written against Sequence actually respect that.

This all got me thinking - these are common protocols, and their semantics and edge-cases are known. Why don't we create a library with generic validation methods for checking some of these basic properties? Wouldn't it be nice, if in one line, you could test your custom Collections against tricky cases? Or test that addition with your custom Float24 or Int128 type really is commutative? Or quickly validate that your algorithms really are only single-pass?

I've started a little library with the goal of helping you do that just that. I've called it swift-checkit.

Currently there isn't very much there: a collection-checker which tests incrementing/decrementing around start-/endIndex, and a single-pass sequence checker. I'm planning on adding more for the protocols I care about, but I think something like this has broad value to the community as a sort of executable knowledge-base. To that end, I'd like to invite everybody to fork/add/upstream new checks and checkers.

It's very simple to use. Simply add the package as a test dependency in your Package.swift, and add something like the following to your tests:

import Checkit

func testCollectionSemantics() {
  CollectionChecker.check("Hello everybody! 👋👨‍⚕️")
}

func testAlgorithm() {
  SinglePassSequenceChecker.check(0..<5) { sequence in sequence.sorted() }
}

I plan to add some collection mutation/index invalidation (MutableCollection, RangeReplaceableCollection) checkers in the near future, as well as some for custom Codable types and custom Encoder and Decoders (although feel free to beat me to it!). I think the numerics protocols are really good candidates for some high-level behaviour checking, and the Combine framework has lots of great candidates, too.

Check it out, and let me know if it helps you find any conformance issues in your code!

11 Likes

I adopted this pattern years ago. Whenever I write a protocol that I expect clients to conform to, then I enclose all its tests in such generic functions in a separate module provided alongside the main one. (Same for classes that will be subclassed.) Then my actual test targets simply call the test functions for each provided concrete conformer and nothing more. That way clients can apply the same set of tests with a single line, and automatically benefit from any new regression or edge case tests that pop up over time. (And hopefully they upstream tests of their own for any gaps they discover.)

Instead of a separate related project (which is still the next‐best thing), I actually recommend pitching the idea of vending the standard and core library’s native tests in a similar manner: SwiftTestUtilities, FoundationTestUtilities, etc.

6 Likes

SwiftNIO offers something similar as well for their ByteToMessageDecoder, where they provide a ByteToMessageDecoderVerifier for implementors to use for tests.

1 Like

One day we might consider making this a core library like XCTest, but that needs community involvement to add useful tests. Also, it's not obviously a good thing, as people may become over-reliant on them. They can help you detect bugs, but a passing test is not the same thing as being bug-free.

Another thing to note: tests which take an instance of a type (e.g. the collection checker and single-pass sequence checks) may pass/fail depending on your type's implementation details. Maybe a different collection, with a different set of values, will exercise different branches in your code. These kinds of tests should be combined with code-coverage reports to ensure you exercise all code paths. Floating-point tests are largely independent from any specific instance, because the type itself provides access to lots of interesting values (like zero and greatestFiniteMagnitude).

I did take a look at the standard library's existing validation suite, but it's heavily gyb-ed and isn't really designed to test 3rd-party conformances to protocols. For example, floating point tests don't check that fundamental type properties are consistent with IEE754, and collection tests check things like whether map only calls its closure once per element (it should). There are some tests that are worth including though, and I'll be copying those over. It just takes time (so patches welcome!)

My expectation is that we'll only copy sporadic bits and pieces of the existing validation suite, so it won't replace the one used right now. It could provide an extra level of validation, though.

2 Likes