Introduce mutation testing

[li-boxuan]

We don't have mutation testing used in any coala repository.

As http://pitest.org/ points out,

Traditional test coverage (i.e line, statement, branch, etc.) measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code. It is therefore only able to identify code that is definitely not tested.

The most extreme examples of the problem are tests with no assertions. Fortunately these are uncommon in most code bases. Much more common is code that is only partially tested by its suite. A suite that only partially tests code can still execute all its branches (examples).

As it is actually able to detect whether each statement is meaningfully tested, mutation testing is the gold standard against which all other types of coverage are measured.

As far as I am concerned, checkstyle community uses mutation testing, which seems to be quite powerful.

See available mutation testing tools for python: https://github.com/theofidry/awesome-mutation-testing Unfortunately, none of them is very popular, and I have no idea how well they work.

[realJema]

I will like to work on this project. are you the mentor @li-boxuan ?

[li-boxuan]

@realJema No, at least not for this year. And I am not sure whether this can be a suitable gsoc project.

[KVGarg]

In context to Implement metrics for coala, can this be added as a software metric for coala?