2020 LLVM Developers’ Meeting: A. Denisov “LLVM-based mutation testing for C and C++”

https://llvm.org/devmtg/2020-09/
—
LLVM-based mutation testing for C and C++ - Alex Denisov

Slides:
—
Mutation Testing, a fault-based software testing technique, serves as a way to evaluate and improve the quality of software tests. A tool for mutation testing creates many slightly modified versions of the original program, so-called mutants, and runs a test suite against each of these programs. A mutant is said to be killed if the test suite detects a change to the program introduced by this mutant, or survived otherwise. To assess the quality of a test suite mutation testing uses a metric called mutation score, or mutation coverage.

 The description may sound like it is a form of fuzzing, but, unlike fuzzing, mutation testing doesn't modify the input to a program but modifies the program itself. Some examples include replacing addition with subtraction, inverting the condition of an if-statement, removing function calls, or replacing a call with a constant value.

In this talk, we want to present Mull[1], an open-source tool for mutation testing based on the LLVM framework. Mull works with LLVM IR, a low-level intermediate representation, to perform mutations, and uses LLVM JIT for just-in-time compilation. This design choice enables the following two capabilities of Mull: language independence and fine-grained control over the compilation and execution of a tested program and its mutations. Mull can work with code written in any programming language that supports compilation to LLVM IR, such as C, C++, Rust, or Swift. Direct manipulation of LLVM IR allows Mull to do less work to generate mutations: only modified fragments of IR code are recompiled, and this results in faster processing of mutated programs.

In this talk, we want to give an intro into the mutation testing and present the tool and its evolution since it was officially presented publicly several years ago[2].

The full list of available mutations and some examples can be found on the documentation page[3].


[1] https://github.com/mull-project/mull
[2] http://lowlevelbits.org/pdfs/Mull_Mutation_2018.pdf
[3] https://mull.readthedocs.io/en/latest/GettingStarted.html
—