Improve (beginner) UX around self vs. fixtures

[The-Compiler]

Something I see sometimes in my pytest company trainings is people accidentally doing:

def test_something(self):
    ...

because they're using to writing things in classes (e.g. from Java?). That results in a

E       fixture 'self' not found
>       available fixtures: [quite a long list here]
>       use 'pytest --fixtures [testpath]' for help on them.

which is not exactly useful if you don't know yet what a fixture is. IMHO, we should detect the self name here, and output an additional hint ("hint: Remove self as this is not a method?" or somesuch).

---

Similarly, when people start moving tests from functions to classes, they forget to add self, which can be very confusing in combination with fixtures:

class TestSomething:

    def test_x(monkeypatch):
        monkeypatch.setenv("DEBUG", "true")

results in:

    def test_x(monkeypatch):
>       monkeypatch.setenv("DEBUG", "true")
E       AttributeError: 'TestSomething' object has no attribute 'setenv'

which is still somewhat clear, but depending on what you do with a fixture (e.g. passing data to a function under test), it can be quite confusing if you've never seen this kind of message before.

Maybe a bit more tricky here, but it would be nice if pytest could detect that we're in a test method, the first argument is not self, and the first argument is a valid fixture name. If all of those are true, that seems to be worth warning about as well.

[nicoddemus]

I'm 👍 on the first case.

Not so sure on the 2nd case because that's Python 101 and not sure how far we should go about helping with the language itself -- not that I oppose it if it is simple to implement, but I suspect it is not (which plays a part on why I think this might not be a good idea).

[The-Compiler]

Agreed the second case is trickier - but at the same time, the error is very confusing, and I think it's more confusing when pytest fixtures come into play.

In normal Python code, if we do:

class Wrong:

    def wrong1(arg):
        pass

    def wrong2():
        pass

    def wrong3(arg=42):
        print(arg)
        print(self.x)

w = Wrong()
w.wrong1(23)

in all cases, the failure mode is quite clear:

• Case 1: we get TypeError: Wrong.wrong1() takes 1 positional argument but 2 were given. Maybe a tiny bit confusing, but it at least happens immediately at the calling site, and is a loud error.
• Case 2: similar to case 1
• Case 3: If we pass an argument, similar to above. If not, arg is now self, which is maybe closest to what happens with pytest.

The main problem I see with it that it can be a almost silent failure (comparing some data to data from a fixture with ==, or worse, !=), or a failure that only gets noticed far away (passing some fixture data into a function under test, which then does operations on it that might or might not explode).

[nicoddemus]

Well, if 2nd case gets implemented, I would say it cannot rely on the name self... for example, I think this would be the wrong approach:

• If it is a method, it must start with self, otherwise fail collection.

I think, if implemented, it should be implemented as:

• If it is a method, and the first argument is not a known fixture, fail collection (as usual), but then look if the failure is due to it being the first argument, and it is not named self, then add to the error message "this is a method, perhaps missing self argument" (or something along those lines).