Feature/skip remaining tests being executed #1029
Conversation
- convert certain statuses into enum - introduce cross-executor messaging - fix config clone
zhenyu-ms
force-pushed
the
skip-remaining
branch
from
a5f7fc2 to
265257b
Compare
testplan/runners/base.py
Outdated
| self._msg_handlers[msg.mark](msg) | ||
| time.sleep(interval) | ||
|
|
||
| def _handle_expected_abort(self, _): |
There was a problem hiding this comment.
Between abort & stop, i think this feature shall re-use stop logic of entities. But I'd rather not having a 3rd mode (expected_abort).
This can be useful in interactive mode, that after triggering an entire testsuite/multitest, we can stop the on-going execution with the same flag.
There was a problem hiding this comment.
2nd thought... stop_plan is going to discard all on-going tests, so maybe it's good enough to just reuse the timeout workflow, which is essentially abort().
testplan/runners/local.py
Outdated
| self._results[next_uid] = result | ||
| self.ongoing.pop(0) | ||
|
|
||
| if thres and result.report.status <= thres: |
There was a problem hiding this comment.
I think we can just call a method of TestRunner here to set the flag, something like
if self.cfg.stop_strategy.should_stop_plan(result):
self.parent.stop_plan()
Then TestRunner shall distribute the should_stop flag to all executors (including this one).
Executors just discard all pending_work and pretend we have finished, and triggers normal stop workflow.
There was a problem hiding this comment.
I think pool size code change shall be in task result handler, every time pool receives a task result, check if we shall stop plan. Then the handling logic shall be exactly as testplan timeout handler.
The nested child pool on the remote side needs some special handing, need to disable this feature.
zhenyu-ms
force-pushed
the
skip-remaining
branch
from
1be7c92 to
2f0fe1a
Compare
zhenyu-ms
force-pushed
the
skip-remaining
branch
2 times, most recently
from
35151a2 to
f3d6dea
Compare
zhenyu-ms
force-pushed
the
skip-remaining
branch
from
f3d6dea to
f0dee31
Compare
| @@ -1000,6 +1008,7 @@ def _post_run_checks(self, start_threads, start_procs): | |||
| :param start_procs: processes before run | |||
| :type start_procs: ``list`` of ``Process`` | |||
| """ | |||
| # XXX: do we want to suppress process/thread check for tests? | |||
There was a problem hiding this comment.
i use XXX for something that's not TODO and not FIXME and not NOTE
suppression is from intuition, use XXX instead of FIXME here
| @@ -95,6 +98,7 @@ def __init__(self, **options) -> None: | |||
| self.event_recorder = EventRecorder( | |||
| name=str(self), event_type="Worker" | |||
| ) | |||
| self._discard_running = threading.Event() | |||
There was a problem hiding this comment.
What is the difference of using threading.Event() v.s boolean.
And _discard_running v.s _discard_pending?
There was a problem hiding this comment.
in this simple use case i think they could be equivalent even in the GIL-free world, threading.Event() might only exist as a reminder
i tend to choose different variable names for the ease of grep
| @@ -13,6 +13,9 @@ | |||
| from testplan.common.utils import logger | |||
| from testplan.runners.pools.communication import Message | |||
|
|
|||
| if TYPE_CHECKING: | |||
| from testplan.runners.pools.base import Worker | |||
There was a problem hiding this comment.
previously missing, see L291
| @@ -0,0 +1,3 @@ | |||
| Added a new command line argument ``--skip-remaining``, a new argument ``skip_strategy`` to MultiTest, allowing certain test execution being skipped for a better user experience. | |||
There was a problem hiding this comment.
allow early return of test execution and skip remaining Testcases/Testsuites/Multitests being executed when a Testcase has failed or raised exception.
zhenyu-ms
force-pushed
the
skip-remaining
branch
from
bf0e8b6 to
08bdb98
Compare
Bug / Requirement Description
Clearly and concisely describe the problem.
Solution description
Describe your code changes in detail for reviewers.
Checklist: