Python Specific Mutation Operators

Description:

This repository contains the implementation of Flame, a mutation testing tool for Python. Flame contains a set of mutation operators that are specific to Python.

The mutation operators implemented in Flame are:

Mutation Operator (Abbr. Name)	Before Mutation	After Mutation
1. Remove A Function Argument (RemFuncArg)	func(a₁, a₂, a₃, ...)	func(a₁, a₃, ...)
2. Remove Conversion Functions (RemConvFunc)	conv(a)	a
3. Remove Elements From Containers (RemElCont)	C = [e₁, ..., eᵢ, ..., eₙ]	[e₁, ..., eᵢ₋₁, eᵢ₊₁, ... eₙ]
4. Remove Expressions from Conditions (RemExpCond)	S = [e₁, ..., eᵢ, ..., eₙ]	S = [e₁, ..., eᵢ₋₁, eᵢ₊₁, ... eₙ]
5. Change Used Attribute (ChUsedAttr)	object.attr	object.new_attr
6. Remove Attribute Access (RemAttrAcc)	object.attr	object
7. Remove Method Call (RemMetCall)	object.method()	object

Flame injects these mutations into the source code of a Python project and runs the test suite of the project to determine the effectiveness of the test suite.

To Run Over a Project:

Using Docker:

There is a Docker file available for Flame. And all of the benchmark projects are in benchmarks directory. To run Flame on all projects, and reproduce the results, you can use the following command:

docker build -t flame .
docker run flame

Then in the container, you can run the following command to reproduce the results for one project:

bash run_in_docker.sh <project_name>

For example:

bash run_in_docker.sh blinker

To run for all projects, you can use the following command:

Note that this command will run Flame on all projects, and it may take a long time.

bash run_in_docker.sh

This script will make a venv for the project, install the requirements, run Flame, run Mutmut and get the summary of the results.

Without Docker:

If you want to run Flame on a specific project, you can follow the steps below:

1. Clone the project
2. Install Flame on the project

pip install /path/to/flame

3. Run the following command:

flame -r <project_root> -i <include_paths> -e <exclude_paths>

For example:

flame -r . -i src -e src/test

To use subset selection, you can use the following command:

flame flame -r . -i src -ss operator_random_threshold

And to set timeout for test execution, you can use the following command:

flame flame -r . -i src -ss operator_random_threshold -to 10

Suggested values for each project can be found in the benchmarks/benchmarks.csv file.

To Install:

1. Install poetry:

curl -sSL https://install.python-poetry.org | python3 -

On Mac, you can also use the following:

brew install poetry

2. Now clone and run:

poetry install

Structure:

The repository is structured as follows:

• mutation_testing: Contains the implementation of Flame.
  • mutation_testing/mutaion_operators: Contains the implementation of the mutation operators.
  • mutation_testing/dynamic_analysis: Contains the implementation of the dynamic operator analysis.
  • mutation_testing/command_line_interface.py: Contains the implementation of the CLI and all of the options can be found here.
  • mutation_testing/mutation.py: Main file that runs the mutation testing process.
• benchmarks: Contains the benchmark projects
  • Each project is in a separate directory and contains the project source code on specific commit hashes.
  • Each project directory contains:
    • install_reqs.sh: A script to install the requirements of the project.
    • flame_run.sh: A script to run Flame on the project.
    • mutmut_run.sh: A script to run Mutmut on the project. (for RQ2)
    • get_results.sh: A script to get the summary of the results.
• scripts: Contains the scripts to summarize the results generated by Flame and Mutmut.

Benchmark Projects:

Benchmark projects are in the benchmarks directory, all of the projects are cloned from their repositories and are on specific commit hashes. The list of all benchmark projects and their commit hashes can be found in the benchmarks/benchmarks.csv file. The following table shows the summary of the benchmark projects and the links to their repositories:

Project	LOC	#Functions	#Tests	Line Coverage%	Mutation Score%	Flame Time (min)
blinker	745	34	25	83	65.12	0.42
schedule	945	57	81	98	79.45	1.31
pyjwt	2235	116	271	94	85.71	4.36
pyquery	2350	142	150	89	85.98	26.15
funcy	2369	262	202	96	79.31	1.51
wtforms	3685	222	352	99	91.5	3.23
python_patterns	3779	352	65	92	67.39	0.35
marshmallow	5105	264	1129	97	92.86	5.34
pyvips	6180	249	320	74	79.13	8.55
structlog	6260	274	822	99	80	5.55
flask	8979	391	1223	95	77.87	14.88
graphene	12255	764	468	96	87.6	96.35
supervisor	15775	1082	1388	89	86.75	93.75
drf	16279	972	1539	96	84.92	462.93
praw	19729	727	1029	99	80.18	276.74
Average	7111.33	393.86	586.46	93	81.58	66.76