Reduce Noise for Persistent Test Failures

[ralphcallaway]

Often enough there is a failing test method that isn't urgent to fix and I'll leave it failing for a while. But then I'm getting notifications every day about the test failure which can drown out when another test method fails that is important. It'd be nice to have an easy way to reduce that noise.

Perhaps there could be an application level notification setting to only send for new test failures. Or maybe the ability to specify that a specify test method failure can be ignored. For that route it'd be good to have that skip setting get cleared once the test starts passing again.

[ChuckJonas]

I agree.

We definitely have the ability to send on new test failures only. It's something I had already discussed with @fishygeek91 when we started tracking "Is New Failure". I think it might even be a configuration only change.

I think the other options could work as well, but would require additional dev.