llvm dev - Jan 2017 - LLVM-based Mutation Testing, first results.

Hello, everybody.

We are working on a tool for mutation testing. The work is still in progress and
far away from being done.
However, we have got some results already. And we would like to share them with
you.
But, let me give you a brief introduction first.

### Mutation Testing

In a nutshell, Mutation Testing is a way to evaluate a quality of a test suite.

The approach suggests introducing a slight change into the code under a test (so
called mutant). Then, if the test fails, mutant is considered as killed.
Otherwise we say that mutant has survived.
Along with that Mutation Testing introduces a metric, called Mutation Score:

  mutation score = (killed mutants) / (total mutants)
  
The higher the mutation score - the more adequate the test suite is. Hence, the
more mutants are killed, the better.

Consider the following code:

void test() {
  assert(sum(4, 5) > 0);
}

int sum(int a, int b) {
	return a + b;
}

Here, one can produce few mutants:

int sum(int a, int b) {
	return a * b;
}

int sum(int a, int b) {
	return a;
}

Both mutants will uncover that the test is not exhaustive enough.

Mutations

### Our Results

We run our tool against two test targets from LLVM: IRTests and ADTTests. Based
on our setup mutation score for them equals 43% and 66%, correspondingly.

We did not analyze all the results thoroughly, yet, but we plan to do so.
At the very moment, it is not clear for us how to process the data more
efficiently.
Also, it is a bit more complicated, since we do not know the domain of LLVM
internals very well.

Here I leave links to the results in case you are interested.

https://lowlevelbits.org/ADTTests/
https://lowlevelbits.org/IRTests/

Source code of the tool itself is here:

https://github.com/mull-project/mull

We hope it helps to improve the quality of LLVM even further.
Thanks for reading.
We appreciate any feedback.

Best regards,
Alex.
-- 
AlexDenisov
Software Engineer, http://lowlevelbits.org

> On Jan 5, 2017, at 3:33 PM, Alex Denisov via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hello, everybody.
> 
> We are working on a tool for mutation testing.
Nice!
> The work is still in progress and far away from being done.
> However, we have got some results already. And we would like to share them
with you.
> But, let me give you a brief introduction first.
> 
> ### Mutation Testing
> 
> In a nutshell, Mutation Testing is a way to evaluate a quality of a test
suite.
> 
> The approach suggests introducing a slight change into the code under a
test (so called mutant).
It is not clear to me if you’re only changing the test itself, or the program
the test exercise? (I’m more interested in the latter actually).

— 
Mehdi


> Then, if the test fails, mutant is considered as killed. Otherwise we say
that mutant has survived.
> Along with that Mutation Testing introduces a metric, called Mutation
Score:
> 
>  mutation score = (killed mutants) / (total mutants)
> 
> The higher the mutation score - the more adequate the test suite is. Hence,
the more mutants are killed, the better.
> 
> Consider the following code:
> 
> void test() {
>  assert(sum(4, 5) > 0);
> }
> 
> int sum(int a, int b) {
> 	return a + b;
> }
> 
> Here, one can produce few mutants:
> 
> int sum(int a, int b) {
> 	return a * b;
> }
> 
> int sum(int a, int b) {
> 	return a;
> }
> 
> Both mutants will uncover that the test is not exhaustive enough.
> 
> Mutations
> 
> ### Our Results
> 
> We run our tool against two test targets from LLVM: IRTests and ADTTests.
Based on our setup mutation score for them equals 43% and 66%, correspondingly.
> 
> We did not analyze all the results thoroughly, yet, but we plan to do so.
> At the very moment, it is not clear for us how to process the data more
efficiently.
> Also, it is a bit more complicated, since we do not know the domain of LLVM
internals very well.
> 
> Here I leave links to the results in case you are interested.
> 
> https://lowlevelbits.org/ADTTests/
> https://lowlevelbits.org/IRTests/
> 
> Source code of the tool itself is here:
> 
> https://github.com/mull-project/mull
> 
> We hope it helps to improve the quality of LLVM even further.
> Thanks for reading.
> We appreciate any feedback.
> 
> Best regards,
> Alex.
> -- 
> AlexDenisov
> Software Engineer, http://lowlevelbits.org
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

> It is not clear to me if you’re only changing the test itself, or the
program the test exercise? (I’m more interested in the latter actually).
At the moment just program, but it may make sense to mutate a test as well.
> On 6 Jan 2017, at 00:36, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> 
>> 
>> On Jan 5, 2017, at 3:33 PM, Alex Denisov via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> 
>> Hello, everybody.
>> 
>> We are working on a tool for mutation testing.
> 
> Nice!
> 
>> The work is still in progress and far away from being done.
>> However, we have got some results already. And we would like to share
them with you.
>> But, let me give you a brief introduction first.
>> 
>> ### Mutation Testing
>> 
>> In a nutshell, Mutation Testing is a way to evaluate a quality of a
test suite.
>> 
>> The approach suggests introducing a slight change into the code under a
test (so called mutant).
> 
> It is not clear to me if you’re only changing the test itself, or the
program the test exercise? (I’m more interested in the latter actually).
> 
> — 
> Mehdi
> 
> 
> 
>> Then, if the test fails, mutant is considered as killed. Otherwise we
say that mutant has survived.
>> Along with that Mutation Testing introduces a metric, called Mutation
Score:
>> 
>> mutation score = (killed mutants) / (total mutants)
>> 
>> The higher the mutation score - the more adequate the test suite is.
Hence, the more mutants are killed, the better.
>> 
>> Consider the following code:
>> 
>> void test() {
>> assert(sum(4, 5) > 0);
>> }
>> 
>> int sum(int a, int b) {
>> 	return a + b;
>> }
>> 
>> Here, one can produce few mutants:
>> 
>> int sum(int a, int b) {
>> 	return a * b;
>> }
>> 
>> int sum(int a, int b) {
>> 	return a;
>> }
>> 
>> Both mutants will uncover that the test is not exhaustive enough.
>> 
>> Mutations
>> 
>> ### Our Results
>> 
>> We run our tool against two test targets from LLVM: IRTests and
ADTTests. Based on our setup mutation score for them equals 43% and 66%,
correspondingly.
>> 
>> We did not analyze all the results thoroughly, yet, but we plan to do
so.
>> At the very moment, it is not clear for us how to process the data more
efficiently.
>> Also, it is a bit more complicated, since we do not know the domain of
LLVM internals very well.
>> 
>> Here I leave links to the results in case you are interested.
>> 
>> https://lowlevelbits.org/ADTTests/
>> https://lowlevelbits.org/IRTests/
>> 
>> Source code of the tool itself is here:
>> 
>> https://github.com/mull-project/mull
>> 
>> We hope it helps to improve the quality of LLVM even further.
>> Thanks for reading.
>> We appreciate any feedback.
>> 
>> Best regards,
>> Alex.
>> -- 
>> AlexDenisov
>> Software Engineer, http://lowlevelbits.org
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev