Bugfix in the federated launch script to let failures be reported correctly

[petervdonovan]

This fixes another way tests can pass when they should be failing.

The behavior which apparently is currently in master is that if all federates terminate then the test passes, even if some federates terminated with nonzero exit code.

[petervdonovan]

The following are now failing for Python in master:

• src/federated/DistributedCountDecentralizedLate.lf (added to this list 7/18 after running tests again to account for flakiness)
• src/federated/DistributedLoopedAction.lf
• src/federated/DistributedLoopedPhysicalAction.lf
• src/federated/DistributedNoReact.lf (added 7/18)
• src/federated/DistributedStop.lf
• src/federated/DistributedStopZero.lf
• src/federated/PingPongDistributed.lf (added 7/18)

We will have to either move them to failing or fix them before this gets merged.

Fortunately it looks like the C and TypeScript tests are still passing. We're lucky that it was only four Python tests that are failing.

[petervdonovan]

Actually, this is a nondeterministic segmentation fault (sometimes it happens, usually it doesn't), so probably it affects other Python federated tests as well. It happens during garbage collection; here is the stack trace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  gc_reset_refs (
    refs=<error reading variable: Cannot access memory at address 0x10>, g=0x0)
    at ../Modules/gcmodule.c:105
105     ../Modules/gcmodule.c: No such file or directory.
[Current thread is 1 (Thread 0x7f78704c4000 (LWP 45104))]
(gdb) bt
#0  gc_reset_refs (
    refs=<error reading variable: Cannot access memory at address 0x10>, g=0x0)
    at ../Modules/gcmodule.c:105
#1  update_refs (containers=containers@entry=0x565479446e30)
    at ../Modules/gcmodule.c:427
#2  deduce_unreachable (base=base@entry=0x565479446e30, 
    unreachable=unreachable@entry=0x7ffe0006e530) at ../Modules/gcmodule.c:1104
#3  0x00005654786b9625 in gc_collect_main (tstate=0x565479462f40, generation=2, 
    n_collected=0x7ffe0006e5f8, n_uncollectable=0x7ffe0006e5f0, nofail=0)
    at ../Modules/gcmodule.c:1239
#4  0x00005654787c0f50 in gc_collect_with_callback (tstate=0x565479462f40, 
    generation=2) at ../Modules/gcmodule.c:1413
#5  0x00005654787f44be in PyGC_Collect () at ../Modules/gcmodule.c:2099
#6  0x00005654787f1efe in Py_FinalizeEx () at ../Python/pylifecycle.c:1781
#7  0x00005654787e1af3 in Py_RunMain () at ../Modules/main.c:668
#8  0x00005654787b7bcd in Py_BytesMain (argc=<optimized out>, argv=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--
    at ../Modules/main.c:720
#9  0x00007f7870229d90 in __libc_start_call_main (
    main=main@entry=0x5654787b7b90 <main>, argc=argc@entry=4, argv=argv@entry=0x7ffe0006e8d8)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#10 0x00007f7870229e40 in __libc_start_main_impl (main=0x5654787b7b90 <main>, argc=4, 
    argv=0x7ffe0006e8d8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7ffe0006e8c8) at ../csu/libc-start.c:392
#11 0x00005654787b7ac5 in _start ()

Could this be related to the problem that @jackykwok2024 is working on?

[lhstrh]

I would be in favor of moving these tests to failing and creating an issue documenting this. I agree that it seems that @jackykwok2024 would be the right person to look into this...

[jackyk02]

Hi @petervdonovan, thanks for the thorough description you've provided. From my understanding, the nondeterministic segmentation fault seems to be caused by the unordered main reactions occurring in Python's federated execution. This does correlate with the memory leak issue I'm currently examining. I will look into it next week.

[petervdonovan]

The reason why this has not been merged yet with the failing tests moved into failing is that I am worried that other tests will also prove to be flaky. This means that I am basically allowing all Python tests to be more or less disabled in master (because that is the status quo) so that we do not have to be bothered with more test failures.

[lhstrh]

I see. I guess the question is: is a fix for the memory leak within reach? And if not, what is the status, @jackykwok2024? Can we help?

[edwardalee]

I think this is also related to the fact that the Python programs segfault with Python 3.11 (issue #1458). I started looking into this, and put some notes in #1458, including some instructions on how to use the VS Code debugger with mixed Python and C. In my setup, I can set breakpoints in our C code and see variable values, etc.

[petervdonovan]

This change has already made it into master by way of the remove-unordered-reactions PR, so we can close this PR.