Test Coverage Reporting

[maennchen]

Changes

Implements test coverage reporting.

Requirements

• Can show coverage summary in CI
• Offers details for download (HTML?)
• Can show coverage summary in CLI
• Follows same rules as mix test.coverage for reporting (generated lines etc.)

TODOs / Issues

• mix test.coverage Summaries depend on mix being present. Find a way to avoid duplicating code.
• Some test break (Like FunctionClauseError message)
• Poluted IO from https://github.com/erlang/otp/blob/05737d130706c7189a8e6750d9c2252d2cc7987e/lib/tools/src/cover.erl#L1896

[maennchen]

@josevalim Do you have any preference on how to deal with the FunctionClauseError issues?

(See https://github.com/elixir-lang/elixir/actions/runs/13929976188/job/38984368756?pr=14343)

[josevalim]

@maennchen probably the best option is to check if coverage is enabled and perform only a partial assertion (i.e. break this assertion in two), the second one wrapped by a conditional.

[maennchen]

@josevalim Ok, I will do that. Do you btw. know the reason why this doesn't work while cover compiled?

[josevalim]

@josevalim Ok, I will do that. Do you btw. know the reason why this doesn't work while cover compiled?

We traverse the stored ast when analyzing the error message but when cover compiling the information is lost.

[maennchen]

@josevalim I got it down to one test failure:

  1) test structs in lists (ExUnit.DiffTest)
     test/ex_unit/diff_test.exs:943
     Assertion with =~ failed
     code:  assert diff_left =~ expected_left
     left:  "[\n  -%ExUnit.DiffTest.Customer{\n    address: %{\"street\" => \"123 Main St\", \"zip\" => \"62701\"},\n    language: \"en-US\",\n    age: 30,\n    first_name: \"John\",\n    last_name: \"Doe\",\n    notifications: true\n  }-\n]"
     right: "[\n  -%ExUnit.DiffTest.Customer{\n    address: %{\"street\" => \"123 Main St\", \"zip\" => \"62701\"},\n    age: 30,\n    first_name: \"John\",\n    language: \"en-US\",\n    last_name: \"Doe\",\n    notifications: true\n  }-\n]"
     stacktrace:
       test/ex_unit/diff_test.exs:123: ExUnit.DiffTest.refute_diff/5
       test/ex_unit/diff_test.exs:956: (test)

The ExUnit.DiffTest.Customer module is defined inside the test and therefore should also not be cover compiled. Do you have a clue why the order is different?

[josevalim]

@maennchen i would have to investigate locally which order is being printed. It seems it was supposed to be alphabetical.

[maennchen]

Splitting PR into multiple PRs. Marking as draft until done.