Silence console output on passing tests

[denis-sokolov]

It would be really cool if Ava silenced console output that happens as a side-effect in passing tests. After all, if a test passes, there is nothing actionable. Console output in those cases is only noise.

This is particularly valuable with tests that validate error throwing with t.throws. It is common for code that throws to also output diagnostic and guidance messages to console.error. As a specific example, React adds this additional information to console.error:

The above error occurred in the component:

at Foo (/Foo.tsx:13:24)

Consider adding an error boundary to your tree to customize error handling behavior.
Visit https://reactjs.org/link/error-boundaries to learn more about error boundaries.

If a test passes, we have no value in this output, everything works as expected. Thus I think it could be a nice addition to buffer calls to console methods during a test execution and only output them if the test fails.

I suspect it’s possible this has already been considered, and there are some scenarios where such behavior is undesirable, and adding more configuration was decided against. In that case, I apologize. I tried to look for any discussion of this in issues, discussions, and did not find it.

[novemberborn]

Yes that is interesting. I'm not sure how it would work during non-serial tests though. We won't be able to attribute output to a specific passing test, so if there's a failure you may suddenly get a lot of unrelated output. Which I suppose is no worse than what we have now.

I'm also not sure how synchronous the logs are and whether we have seen all of it by the time the test ends.

Logs can also be used for debugging purposes even when all tests pass.

What do you think?

[denis-sokolov]

Hm, yeah, attributing logs in non-serial tests is a blocking issue, possibly an unsolveable one. :-( The only way I see it resolved is to re-run failed tests in isolated environments. But perhaps this would be beyond the scope of changes that we can realistically expect for this feature.

[turadg]

One use case for this is CI where we run many tests in the same job and the logs are cluttered with useless info. There I only want output for failed tests. I've tried disabling "verbose" but it looks like that option is no longer observed.

@novemberborn WDYT of verbose=false making t.log only output for failing tests? That doesn't have the concurrency problem because each test has its own.

I wouldn't want this to be the default because it's helpful while watching tests to see how they're progressing, and failure can't be known until it's reached. (Though I suppose you could do something with the TTY to output and then hide what was output once it's known to be passing.)

[novemberborn]

We did make some improvements in AVA 4.3. Really though the whole reporter infrastructure needs overhauling to make these kinds of things possible.

[croconut]

i've had decent success rerouting the process's write streams to alternate files. it doesn't catch t.log() and it worked really well for splitting up logs so i can read just the test failure / success outputs in the terminal output, since rewiring process.stdout.write doesn't effect the top level coverage / test failure + pass output.

i imagine it could be used as an internal ava technique, but this file is run in a required module (the overwrites are specifically only run in the pretest hook, btw)

const fs = require('node:fs');
const path = require('node:path');

// could change directory path to e.g. a temp directory or a user given filename
fs.mkdirSync(path.resolve(__dirname, '../coverage/'), { recursive: true });
let outfile = path.resolve(__dirname, '../coverage/testout.log');
let errfile = path.resolve(__dirname, '../coverage/testerr.log');

fs.writeFileSync(outfile, '');
fs.writeFileSync(errfile, '');

const standard = fs.createWriteStream(outfile, { flags: 'a' });
process.stdout.write = standard.write.bind(standard);
const errors = fs.createWriteStream(errfile, { flags: 'a' });
process.stderr.write = errors.write.bind(errors);

its even simpler if we just want to throw away the logs or just throw away stdout and keep stderr in the normal ava output since we dont have to worry about file overwriting / file appending.

this method seems usable inside ava's framework since we could save the actual process.stdout.write, make a separate filename for each test using file path -> suite -> test name in some temp directory (or hold in memory if we dont want to require write permissions), then on test failure send the data to actual process.stdout.write