Using python_function to find pytest tests

[martvald]

Hi,

I'm struggling with having my test pass. I have been using a pytest.ini:

[pytest]
python_files = *.py
python_functions = test_* _test_*

I would then normally run something like:

pytest my_package

Which would test all python files in my_package starting with "test_" or "test". I have not been able to duplicate this behavior in pants. It does find and run with the same pytest.ini file. This is my current BUILD:

python_requirements(name="reqs")

python_sources(name="root")
python_sources(name="my_package", sources=["my_package/**/*.py"])

python_tests(
    name="my_package_test",
    sources=["my_package/**/*.py"],
    dependencies=[":my_package"],
)

Hopefully this is a quick fix I haven't been able to find.

[benjyw]

The issue is that even before pytest.ini is applied, Pants has to decide what to run pytest on. It does so by looking at the sources of python_tests targets. In your case pants pytest path/to:my_package_test for example.

However -

Typically you don't need to explicitly specify sources= on targets, because the defaults are sensible. E.g., the default for python_tests() is ('test_*.py', '*_test.py', 'tests.py'), and the default for python_sources() is ('*.py', '*.pyi', '!test_*.py', '!*_test.py', '!tests.py', '!conftest.py', '!test_*.pyi', '!*_test.pyi', '!tests.pyi').

But you've specified them explicitly (presumably because you want to recurse into subdirs), and they overlap, so that may be causing issues. I would try making sure the sources don't overlap, for starters.

So what exactly is the issue you're having, that some tests aren't being run when you expect them to? Which ones? I'm guessing the _test_* ones?

[martvald]

The issue is that it runs on all files, and if that files does not contain a test it will fail with pytest exit code 5:

Exit code 5:
No tests were collected

I think the issue is that the python functions flag is what says to ignore files without tests and that this is not correctly set up when using pants.

[benjyw]

Ah, I think what is happening is that those pytest.ini stanzas configure pytest's test discovery, but because Pants passes a list of files to it explicitly, pytest doesn't apply test discovery logic to them, and just takes all those files as given.

So the workaround is to fix your sources= to include just the tests. At least to evaluate this hypothesis. If this is the case we can consider a more robust solution.

[martvald]

Yeah, I have struggled with this and probably just need to skip pants for this part...:( Sad since pytest allows it.

Alot of our files look something like:

# File named f.ex summation.py
import ...

def sum(a: float, b: float) -> float: ...

def _test_sum(): ...

This means that pants test does not run the test since the file does not contain test in their name. It does work if I ask it to test specific files or "**.*py" in the appropriate folder, but then it will fail since there are files that does not contain any tests.

[benjyw]

Ah, gotcha. That explains why you need to do things this way. Can you document your case in #20897? It would be great if the new python backend could handle this sensibly (we generally want the new backend to be a lot less intrusive and opinionated)