Today I'm releasing the first alpha of PropertyTestingKit! PropertyTestingKit is a coverage-guided fuzzer available for Swift.
@Test
func testDatabaseQuery() async throws {
try await fuzz { table, limit in
let query = buildQuery(table: table, limit: limit)
let result = database.execute(query)
// Properties that should hold for all inputs
#expect(result.isValid || result.hasError)
if limit < 0 {
#expect(result.hasError, "Negative limit should error")
}
}
}
Intro
Fuzzing
Fuzzing is a testing strategy where random or semi-random data is fed into a program and output is observed. Use cases include property-based testing and security. While a developer may be able to hand pick edge cases, fuzzers assist in finding unknown edge cases that a developer may not think of. They're especially useful in extremely complex programs where it is infeasible or impossible to achieve full coverage using hand crafted examples. Things like browsers, compilers, and databases come to mind.
Property-Based Testing
Property-based testing (or property testing or PT) is a use case of fuzzers. Traditional example-based unit tests provide some input to a subject and then assert against its output. This works well when we know exactly the expected output for some input.
PT on the other hand establishes rules that are true for all inputs. For example, imagine a shopping cart type. We could generate grocery items at random and then also generate a random deletion index. If we insert our grocery items into the cart and then delete the item at the index, no matter what was generated we would expect the remaining items to still be in the cart. We can say that this is a "property" of the shopping cart type. That property should hold no matter what random inputs our fuzzer generates.
Coverage Guidance
This project uses a strategy called coverage guidance. Rather than having our fuzzer generate input entirely at random, it checks if an input produced new code coverage, and if it did it prioritizes testing similar inputs.
About the project
You can check out the readme for additional usage options, but I want to talk through some goals for the project.
Approachable
Property testing frameworks frequently require different setups from standard unit tests. I want property tests to feel familiar. With that goal in mind, PropertyTestingKit meshes easily with Swift Testing. It performs comfortably with parallel tests, and first party failed assertions are captured and recorded as input failures. By default it runs in regression mode. It does a single extended run, records a corpus of interesting inputs, and on subsequent runs only uses the corpus to keep tests deterministic and fast.
I also intentionally diverged from the many QuickCheck descendants. PropertyTestingKit is unopinionated about how the actual property tests are written. I'm hoping this results in developers writing their property tests in idiomatic imperative Swift.
Fast
The fuzzer is hand-optimized. By default it runs across all cores on the machine. On my local machine, performance is within striking distance of libFuzzer with libFuzzer at ~642k inputs/second/GHz and PropertyTestingKit capping out at ~587K inputs/second/GHz. As a point of reference, that's ~35m inputs/second on my MacBook Pro M3 Max. Of course a real test will be much slower, but at this point the overhead introduced by the fuzzer is minimal.
In addition to running a single fuzzing campaign extremely fast, the penalties for running multiple fuzzing campaigns in parallel, as would be expected with Swift Testing, are minimal. My benchmarks show ~537K inputs/sec/GHz total throughput with 16 parallel runs, which is only a 9% penalty.
Flexible
The library ships with good defaults for common data types, but it also allows for custom data sources with each data source defining seeds, a random generator and a mutator. The library allows users to additionally define what happens when edges are hit and how different coverage signatures are compared. It also includes a plugin system so users can introduce hooks that run on every iteration, input failure, etc.
Between all of this, I was able to implement strategies used by several other fuzzers or described in research papers[1].
A Big Asterisk
This project does not build with Swift nightly. As part of the development loop, I forked and patched the Swift compiler every time I ran into issues. The vast majority of those were issues with parameter packs. swift-testing has additionally been patched to expose a hook for failed assertions and allow failed assertions to be silenced.
Compiler fork here
Swift Testing fork here
I hope this project is the push needed for these bugs to be fixed! I’m not going to try to upstream the changes in my compiler fork because I don’t feel confident in their quality. The tests, however, are legitimate bug repros so I’ll open PRs with those to help out someone more familiar with the compiler architecture.
The Swift Testing patch, on the other hand, I plan on trying to upstream. It's more in my wheelhouse, so I'm confident enough to stand by those changes.
What's next
I'm calling this the 0.0.1 alpha release. It should not be considered stable. There will be breaking changes until I'm confident enough to tag 1.0.0. The first stable release will also not come out until this project can build and run on a stable release of the toolchain.
I'm excited to continue integrating strategies from the research into PropertyTestingKit! The one currently in the works is from a paper called ConFuzz which works by taking over the task scheduling system and fuzzing scheduling alongside the rest of your inputs. With this strategy, many programs that would be non-deterministic and poor fuzzing targets become deterministic and viable.
After that, I'll be looking to benchmark PropertyTestingKit against the Etna platform. I think this will provide a good benchmark target against which to evaluate further improvements to coverage guidance, mutation, and generation strategies.
[1] https://www.usenix.org/system/files/raid2019-wang-jinghan.pdf, https://dl.acm.org/doi/10.1145/3580596