CA-390883: Add docs and improve comments on pytest and Coverage.py

Signed-off-by: Bernhard Kaindl <[email protected]>

doc/content/python/_index.md

@@ -0,0 +1,135 @@
+---
+title: "Python"
+---
+
+Introduction
+------------
+
+Most Python3 scripts and plugins shall be located below the `python3` directory.
+The structure of the directory is as follows:
+
+- `python3/bin`: This contains files installed in `/opt/xensource/bin`
+   and are meant to be run by users
+- `python3/libexec`: This contains files installed in `/opt/xensource/libexec`
+   and are meant to only be run by `xapi` and other daemons.
+- `python3/packages`: Contains files to be installed in python's `site-packages`
+   are meant to be modules and packages to be imported by other scripts
+   or executed via `python3 -m`
+- `python3/plugins`: This contains files that
+   are meant to be `xapi` plugins
+- `python3/tests`: Tests for testing and covering the Python scripts and plugins
+
+Dependencies for development and testing
+----------------------------------------
+
+In GitHub CI and local testing, we can use [pre-commit] to execute the tests in
+a dedicated, clearly defined and always consistent Python environment.
+
+The easiest way to run all tests and checks is simply to run [pre-commit]. The
+example commands below assume that you have Python3 in your PATH and are running
+on recent Linux host with Python 3.11 installed and available in your PATH.
+
+> By default, CentOS 8 provides Python3.6, which is too old to run some tests:
+>> Capturing logs in `python3/tests/observer/it_traces.py` needs Python>=3.7.
+
+```bash { title="Installing and running pre-commit" }
+pip3 install pre-commit
+pre-commit run -av
+# Or, to just run the pytest hook:
+pre-commit run -av pytest
+```
+
+Alternatively, you can of course tests in any suitable environment,
+given that you install the supported versions of all dependencies.
+
+The dependencies are defined in [additional_dependencies] of the [pytest] hook
+in the [pre-commit] configuration file [.pre-commit-config.yaml].
+{{% expand title=
+"Example `pytest` hook from `.pre-commit-config.yaml` (expand)" %}}
+
+```yaml
+    hooks:
+    -   id: pytest
+        files: python3/
+        name: check that the Python3 test suite in passes
+        entry: sh -c 'coverage run && coverage xml &&
+            coverage html && coverage report &&
+            diff-cover --ignore-whitespace --compare-branch=origin/master
+            --show-uncovered --html-report .git/coverage-diff.html
+            --fail-under 50 .git/coverage3.11.xml'
+        require_serial: true
+        pass_filenames: false
+        language: python
+        types: [python]
+        additional_dependencies:
+        - coverage
+        - diff-cover
+        - future
+        - opentelemetry-api
+        - opentelemetry-exporter-zipkin-json
+        - opentelemetry-sdk
+        - pytest-mock
+        - mock
+        - wrapt
+        - XenAPI
+```
+
+{{% /expand %}}
+
+Coverage
+--------
+
+Code moved to the python3 directory tree shall have good code coverage using
+tests that are executed, verified and covered using [pytest] and [Coverage.py].
+
+The `coverage` tool and [pytest] are configured in `pyproject.toml` and
+`coverage run` is configured to run [pytest] by default.
+
+`coverage run` collects coverage from the run and stores it in its database.
+
+The most simple command line to run and report coverage to stdout is:
+`coverage run && coverage report`
+
+Other commands (also used in the [pytest] hook example above):
+
+- `coverage xml`: Generates an XML report from the coverage database to
+  `.git/coverage3.11.xml`. It is needed for upload to <https://codecov.io>
+- `coverage html`: Generates an HTML report from the coverage database to
+  `.git/coverage_html/`
+
+For more information see: <https://coverage.readthedocs.io>
+
+We configure the file paths used for the generated database and other coverage
+configuration in the sections `[tool.coverage.run]` and `[tool.coverage.report]`
+of `pyproject.toml`.
+
+Pytest
+------
+
+If your Python environment has the [dependencies for the tests] installed, you
+can run [pytest] in this environment without any arguments to use the defaults.
+
+For development using tests, you can run [pytest] with coverage (or without).
+
+To run a specific [pytest] command, [pytest] and pass the test case (example):
+
+```bash
+pytest python3/tests/test_perfmon.py
+```
+
+Same example with coverage collection and reporting:
+
+```bash
+coverage run -m pytest python3/tests/test_perfmon.py && coverage report
+```
+
+[coverage.py]: https://coverage.readthedocs.io
+"coverage.py is the coverage collector for Python"
+[dependencies for the tests]: #dependencies-for-development-and-testing
+"Installation of the dependencies for development and testing"
+[pytest]: https://docs.pytest.org "Pytest documentation"
+[pre-commit]: https://pre-commit.com "pre-commit commit hook framework"
+[.pre-commit-config.yaml]: https://pre-commit.com/#adding-pre-commit-plugins-to-your-project
+"project-specific configuration file of pre-commit, found in the project's top directory"
+[additional_dependencies]: https://pre-commit.com/#pre-commit-configyaml---hooks
+"dependencies that will be installed in the environment where this hook gets to run"

pyproject.toml

@@ -30,6 +30,20 @@ line-length = 88
 
 # -----------------------------------------------------------------------------
 # Coverage.py - https://coverage.readthedocs.io/en/coverage-5.5/config.html
+#
+# [tool.coverage.run] and [tool.coverage.report] configure these commands:
+# coverage run && coverage report
+#
+# These work in conjunction with [tool.pytest.ini_options] to set defaults
+# for running pytest (on its own) and for running Coverage.py with pytest:
+#
+# Examples for Python test development with Coverage.py:
+#
+# Run the default tests and check coverage:
+#   coverage run && coverage report
+#
+# Run a custom set of tests and check coverage:
+#   coverage run -m pytest python3/tests/test_*.py && coverage report
 # -----------------------------------------------------------------------------
 
 [tool.coverage.report]
@@ -198,22 +212,12 @@ exclude              = [
 
 [tool.pytest.ini_options]
 # -----------------------------------------------------------------------------
-# Options to enable for pytest by default:
+# addopts: Options to add to all pytest calls:
 # -v                           show what happens
 # -ra                          show short summary after running tests
-# Other options should not be passed using addopts, as addopts forces those
-# options to be used every time pytest is run, which is very restrictive.
-# Instead, use `coverage run` to configure coverage options, and support
-# running specific tests by passing them as arguments to pytest:
-# For example:
-# coverage run -m pytest python3/tests/test_xenapi.py
-# Adding specific --cov options using addopts is not recommended as it would
-# require to use the pytest-cov plugin, which would conflict with the use of
-# `coverage run`. Instead, use `coverage` to configure coverage options.
-# Specifying directories to test is better done using the testpaths option,
-# as testpaths sets the default directories to search for tests, but does not
-# force them to be run, so you can still run specific tests files by just
-# passing them as arguments to pytest: pytest python3/tests/test_xenapi.py
+#
+# addopts are added to all pytest calls. We don't add options that would force
+# testing specific paths. To be flexible, we use use testpaths instead(see below)
 # -----------------------------------------------------------------------------
 addopts = "-v -ra"
 
@@ -223,7 +227,9 @@ addopts = "-v -ra"
 # log_cli_level:    log level to show
 # python_files:     pattern for test files
 # python_functions: pattern for test functions
-# testpaths:        directories to search for tests
+# testpaths:        directories to search for tests(by default, used for CI)
+#                   For development, developers can test only specific files:
+#                   Example: pytest python3/tests/test_perfmon.py
 # minversion:       this config requires pytest>=7 to configure pythonpath
 # pythonpath:       path to stub files and typing stubs for tests
 # xfail_strict:     require to remove pytext.xfail marker when test is fixed