Tests no longer pass

[cxw42]

Steps:

1. Install deps on an Ubuntu 22.04 machine
2. Create and activate a Python 3.8.10 venv
3. Check out ad558b0
4. Run prove

Expected:

• Tests pass

Observed:

• numerous test failures

Test Summary Report
-------------------
t/100-basic.t       (Wstat: 512 Tests: 3 Failed: 1)
  Failed test:  3
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
t/200-api.t         (Wstat: 256 Tests: 0 Failed: 0)
  Non-zero exit status: 1
  Parse errors: No plan found in TAP output
t/300-doc-comments.t (Wstat: 256 Tests: 0 Failed: 0)
  Non-zero exit status: 1
  Parse errors: No plan found in TAP output
t/400-web.t         (Wstat: 256 Tests: 0 Failed: 0)
  Non-zero exit status: 1
  Parse errors: No plan found in TAP output
Files=5, Tests=9,  1 wallclock secs ( 0.03 usr  0.00 sys +  0.80 cusr  0.10 csys =  0.93 CPU)
Result: FAIL

It looks like the tests have not been kept up to date with code changes, e.g., 9e18f26.

[cxw42]

I am working on this on branch test-fixes-2025-01-cwhite.

[fstachura]

Hello, thanks for the report. Indeed, the code changed a lot, and tests were kind of left behind. Some probably won't be easy to fix - IIRC some were running Elixir as a CGI script, but it's longer that straightforward, because Elixir was moved to WSGI (this can be worked around with some wrappers).

I was planning to replace the current test code at some point in the future. If you are able to fix it easily then that's good, but I'm not sure how much can be updated without too much hassle.

[tleb]

@fstachura said it all! I'll add the main reason we left the tests behind: neither @fstachura nor me know how to read a single line of Perl.

In the long-term, we envisioned going to Python for testing:

• To unify the codebase,
• Current maintainers (@fstachura & I) know about it,
• More people know it, maximizing the chance fly-by contributors and potential future maintainers know it as well. (This is pretty much 100% guaranteed as contributors must be able to talk Python to contribute.)

We appreciate the effort though, thanks! It might not be an easy thing, how big do you expect the changes to be?

[cxw42]

Well, even independent of the Perl tests, the existing Python tests don't pass either :) .

But, yes, if you think it's best for the project to rework the tests, I certainly won't argue with you! However, if it were up to me (which it's not), I would get the existing tests running first to guard against regressions and loss of coverage.

Edit I got t/200 working

[cxw42]

Also, the query.py sample in the README doesn't work for me on latest master (973d006) --- is that expected?

$ ./elixir/query.py v4.10 ident raw_spin_unlock_irq C
Traceback (most recent call last):
  File "./elixir/query.py", line 21, in <module>
    from .lib import script, scriptLines, decode
ImportError: attempted relative import with no known parent package

(this causes all the query.py tests to fail)

[tleb]

Try replacing ./elixir/query.py by python -m elixir.query. The way you start the script changes the import paths!

[tleb]

However, if it were up to me (which it's not), I would get the existing tests running first to guard against regressions and loss of coverage.

I would agree. But pragmatism tells me that taking 2 days to fix tests is less time efficient than taking 30 minutes here and there to fix the odd bugs that get reported (often internally & quickly as we are big users of Elixir). Anyway, tests would bring peace of mind during deployments, so could be beneficial. They just shouldn't take too long to write and run.