I just published a mutation testing tool for Swift based on LLVM technology.
Mutation testing is a software testing technique that evaluates the quality of a test suite by mutation score. It creates several slightly modified versions of the original program and runs the test suite against them. A mutation score shows a ratio of killed vs total mutants. If you want more detail, please refer Wikipedia page.
Here is a demo GIF to show how to use it against SwiftPM project.
How it works
mull-xctest is implemented based on mull-project/mull project, which is a practical mutation testing tool for C and C++ based on LLVM. Mull finds and creates mutations of a program on the level of LLVM bitcode module. This LLVM-based approach saves a lot of time since it does not have to re-compile a program from source files.
The original Mull extracts bitcode from executable files, but XCTest bundle can't have embedded bitcode, so mull-xctest extracts and modifies bitcode while linking.
And since Swift is much higher-level language than C/C++, so not every mutations found at LLVM IR from Swift can be represented at the source level. These mutations, which exists at LLVM IR level, but cannot be achieved at source code level, are called junk mutation. mull-xctest filters junk mutations by looking back source code information retrieved from SwiftSyntax.
Since mutations are applied at LLVM IR level, there are some issues around junk mutations.
For example, mull-xctest finds
icmp eq i32 a, i32 b at LLVM level and checks that there is
a == b expression at the location from instruction debug info, then modifies it to
icmp ne i32 a, i32 b. This works well for simple types like
Bool.==, but doesn't work for
BinaryInteger.== because it emits a bunch of
icmp eq instructions from its definition.
If there is a way to disable mandatory-inlining for
@_transparent functions, mull can replace calls of
BinaryInteger.!=. (but I know some passes depend on mandatory-inlining...)
So I want to ask compiler folks whether there is a solution to address this issue.