Add pytest-cov for reporting test coverage (huggingface#641)

As discussed, this adds line coverage to the tests. This will allow us
to identify parts of the code that are missing coverage and make it
easier to ensure newly added code is well covered.

At the moment, CI is not set up to report if new, uncovered code is
being added. We could add codecov to the CI to get this functionality,
but having 100% coverage for new code is not always desired, so it's
debatable if it is needed.

Right now, there are multiple test commands (normal, single, multi GPU).
For each individual command, the coverage report would only include the
lines covered by that command, so the total coverage would be
underreported. It is possible to combine multiple coverage reports into
a single report:

https://coverage.readthedocs.io/en/stable/cmd.html#cmd-combine

Combining the reports will be added in a future PR.

.github/workflows/tests.yml

@@ -43,7 +43,7 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           # cpu version of pytorch
-          pip install .[test]
+          pip install -e .[test]
       - name: Test with pytest
         run: |
           make test

pyproject.toml

@@ -33,4 +33,7 @@ doctest_optionflags = [
     "NORMALIZE_WHITESPACE",
     "ELLIPSIS",
     "NUMBER",
-]
+]
+
+[tool.pytest.ini_options]
+addopts = "--cov=src/peft --cov-report=term-missing"

setup.py

@@ -18,7 +18,7 @@
 extras["quality"] = ["black ~= 22.0", "ruff>=0.0.241", "urllib3<=2.0.0"]
 extras["docs_specific"] = ["hf-doc-builder"]
 extras["dev"] = extras["quality"] + extras["docs_specific"]
-extras["test"] = extras["dev"] + ["pytest", "pytest-xdist", "parameterized", "datasets", "diffusers"]
+extras["test"] = extras["dev"] + ["pytest", "pytest-cov", "pytest-xdist", "parameterized", "datasets", "diffusers"]
 
 setup(
     name="peft",