Improve testing harness using psutil process handling

[piotrm-nvidia]

No description provided.

[piotrm-nvidia]

You can run test with debug logs to see both pytest logs from test and also logs from subscriber.py and publisher.py processes:

pytest -v icp/tests/python/event_plane/test_multiprocess.py --log-cli-level DEBUG

The sample of log:

INFO     icp.tests.python.event_plane.publisher_subscriber_utils:publisher_subscriber_utils.py:275 Checking if worker needs kill for WorkerRecord(name='publisher_0', command=['/usr/bin/python3', '-u', './icp/examples/python/event_plane/publisher.py', '--publisher-id', '1', '--save-events-path', '/tmp/publisher_1_xz2ax8hc.json'], needs_kill_to_stop=False, json_file='/tmp/publisher_1_xz2ax8hc.json', tag='publisher') worker to finish
INFO     icp.tests.python.event_plane.publisher_subscriber_utils:publisher_subscriber_utils.py:277 Waiting for WorkerRecord(name='publisher_0', command=['/usr/bin/python3', '-u', './icp/examples/python/event_plane/publisher.py', '--publisher-id', '1', '--save-events-path', '/tmp/publisher_1_xz2ax8hc.json'], needs_kill_to_stop=False, json_file='/tmp/publisher_1_xz2ax8hc.json', tag='publisher') worker to finish
INFO     subscriber_0:publisher_subscriber_utils.py:89 selector_events.py: DEBUG: __init__(): 64:	Using selector: EpollSelector
INFO     subscriber_0:publisher_subscriber_utils.py:89 nats_event_plane.py: DEBUG: connect(): 156:	Connecting to NATS server: nats://localhost:4222
INFO     subscriber_0:publisher_subscriber_utils.py:89 nats_event_plane.py: DEBUG: connect(): 178:	Connected to NATS server: nats://localhost:4222

The logs with icp.tests.python.event_plane.publisher_subscriber_utils are generated by test harness and lines liken this:

INFO     subscriber_0:publisher_subscriber_utils.py:89 nats_event_plane.py: DEBUG: connect(): 178:	Connected to NATS server: nats://localhost:4222

Are generated by ScriptThread class and forwarded from sub-process subscriber.py with id 0 to main process.

Full log:

verbose_test_log.txt

[piotrm-nvidia]

It seems that this test dead-locks at CI:

https://gitlab-master.nvidia.com/dl/triton/triton-distributed/-/jobs/141090309#L318

[piotrm-nvidia]

Working example.

[piotrm-nvidia]

It won't be used.