How to speed up feedback from autotests

[slinlee]

@EvgeniyEA - I was thinking about how to get faster feedback from the auto tests. Let me know if this makes sense:
if there are failed tests for a PR, can we have the follow-up test start with only the previously failed tests? That way the dev can see if their changes resolved those tests.

It could be a separate step:
[previously failed tests] -> smoke tests -> full tests

If those still fail, it will stop the whole test, just the same as the behavior if smoke tests don't pass.

cc @xneyder @kepricon @fionnachan @alexamakarov