How to get coverage reporting when testing an Ansible lookup plugin

[andy-maier]

I am developing an Ansible lookup plugin and test code for it. Both the lookup plugin and the test code for the plugin work fine.

I am using "pytest" with "pytest-ansible" for that test, and the test function uses the ansible_module fixture provided by "pytest-ansible" to invoke the builtin "set_fact" module to invoke the lookup. The test function looks like this (simplified):

import pytest

@pytest.mark.ansible(host_pattern='localhost', connection='local')
def test_mylookup(ansible_module):
    expr = "lookup('myorg.mycoll.mylookup', 'someterm')"   # Jinja2 expression to call the lookup to be tested
    contacted = ansible_module.set_fact(result=f"{{{{ {expr} }}}}")    # Call the lookup
    contacted_host = contacted['localhost']
    result = contacted_host['ansible_facts']['result']   # result value returned by the lookup
    # ... assertions on result

Again, this all works fine. The problem is that this test does not add to the test coverage (0%). Other Python modules in my project do add to the test coverage, just the tests for Ansible plugins do not.

I suppose this is because the ansible_module object at some point uses Ansible mechanisms to execute the module, and that probably lets my lookup code run in a different thread or process. But this is speculation.

I am using the "pytest-cov" and "coverage" packages to have "pytest" count the coverage, and run the test by invoking pytest with --cov* options.

Relevant Python package versions (on Python 3.9):

ansible                   8.7.0
ansible-core              2.15.13
pytest                    7.4.4
pytest-ansible            4.1.1
pytest-cov                6.0.0
coverage                  7.6.4

I would like to know how I can get the test coverage properly accounted for when testing an Ansible lookup plugin with "pytest" and "pytest-ansible".

I also posted this question on SO: https://stackoverflow.com/q/79174901/1424462

[jon-nfc]

@andy-maier, did you get any further with this? I'm also looking at using pytest-ansible for e2e tests.

Some randoms that may be helpful to look at:

• check .coveragerc to ensure the code paths are included and not excluded
• check pytest.ini to ensure that you don't have any settings defined there that may be contrary to testing the code in question
• is there any output from coverage? check specified output dir .coverage or the output reports (will require that you output reports. I use pytest --cov --cov-report term --cov-report xml:../artifacts/coverage_unit.xml --cov-report html:../artifacts/coverage/unit/ --junit-xml=../artifacts/unit.JUnit.xml this gives: STDOUT, xml, JUnit and HTML reports)
• This one may be chirping to the choir, however I'm sure we've all done it. If using a python venv to run the tests ensure it's the venv that has the test dependencies installed to. i.e. accidental usage of wrong venv.

[bastantoine]

Hello,
@andy-maier I don't know if you still have the issue, I had the exact same one, so for anyone facing the same problem, here's how I was able to solve it.

Internally Ansible uses subprocesses to run the tasks, so I had to configure Coverage to use the multiprocessing mode:

# pyproject.toml

[tool.coverage.run]
# The test for the ansible plugin is using ansible itself to run. Ansible works
# with multiprocesses, so we need to configure coverage to work with it.
concurrency = ["multiprocessing"]
# Config needs to be set here and not in the addopts where it would be added to the command.
# When dealing with multiprocessing and subprocesses, only the options can only be set in
# the config file, and not passed by the command line.
# https://coverage.readthedocs.io/en/7.6.10/subprocess.html
branch = true

I did it in my pyproject.toml file, but any configuration of coverage will work (.coveragerc, setup.cfg or tox.ini)

[source]

I also had to make sure that the source code of my plugin was included in the sources with the --cov option.

Btw thank you very much for the excrept of code showing how to write the test, I basically copy-pasted it to write mine 😅.