LLVM-based mutation testing for Swift

Hi, everyone!

I just published a mutation testing tool for Swift based on LLVM technology.

Mutation testing is a software testing technique that evaluates the quality of a test suite by mutation score. It creates several slightly modified versions of the original program and runs the test suite against them. A mutation score shows a ratio of killed vs total mutants. If you want more detail, please refer Wikipedia page.

Here is a demo GIF to show how to use it against SwiftPM project.


How it works

mull-xctest is implemented based on mull-project/mull project, which is a practical mutation testing tool for C and C++ based on LLVM. Mull finds and creates mutations of a program on the level of LLVM bitcode module. This LLVM-based approach saves a lot of time since it does not have to re-compile a program from source files.

The original Mull extracts bitcode from executable files, but XCTest bundle can't have embedded bitcode, so mull-xctest extracts and modifies bitcode while linking.

And since Swift is much higher-level language than C/C++, so not every mutations found at LLVM IR from Swift can be represented at the source level. These mutations, which exists at LLVM IR level, but cannot be achieved at source code level, are called junk mutation. mull-xctest filters junk mutations by looking back source code information retrieved from SwiftSyntax.

Current Issues

Since mutations are applied at LLVM IR level, there are some issues around junk mutations.

For example, mull-xctest finds icmp eq i32 a, i32 b at LLVM level and checks that there is a == b expression at the location from instruction debug info, then modifies it to icmp ne i32 a, i32 b. This works well for simple types like Bool.==, but doesn't work for BinaryInteger.== because it emits a bunch of icmp eq instructions from its definition.

If there is a way to disable mandatory-inlining for @_transparent functions, mull can replace calls of BinaryInteger.== with BinaryInteger.!=. (but I know some passes depend on mandatory-inlining...)

So I want to ask compiler folks whether there is a solution to address this issue.




@kateinoigakukun can you start a thread in swift-dev on the transparent issue you are talking about? Lets leave this thread for the community show casing of your cool work! = ).

1 Like

This work deeply excites me, @kateinoigakukun, given the work I've done on Muter the last 2 years. I'm glad to see mutation testing is becoming practiced and discussed more widely within the community!

I'm looking forward to playing around with this and seeing how it performs.


Of course! :+1:

Muter is a great tool! I referred to it frequently while developing my tool. Thanks!

1 Like