KillingInvocation seems to not properly work with Killed invocation status

[tillrohrmann]

In this test run https://github.com/restatedev/restate/actions/runs/13764395844/job/38488506837?pr=2873#step:11:434, the test KillInvocation is stuck. From the looks of the sys_invocation table, we are not properly killing all recursive calls. It seems as if one child is still running while the direct parent and the startCallTree are completed. Since the final verification call runs on the same VO as the recursive call, the test is stuck.

{"rows":[{"id":"inv_12dw5UBHriBL6zDR4IfF2Ev9TdQHvK6rsZ","target":"KillTestSingleton/c735fb3a-4e2c-44dd-9d42-ad9c932eb107/recursiveCall","target_service_name":"KillTestSingleton","target_service_key":"c735fb3a-4e2c-44dd-9d42-ad9c932eb107","target_handler_name":"recursiveCall","target_service_ty":"virtual_object","invoked_by":"service","invoked_by_service_name":"KillTestSingleton","invoked_by_id":"inv_12dw5UBHriBL2DJ0pgfNmE46yAptW2rrMJ","invoked_by_target":"KillTestSingleton/c735fb3a-4e2c-44dd-9d42-ad9c932eb107/recursiveCall","journal_size":1,"created_at":"2025-03-10T12:36:10.914Z","modified_at":"2025-03-10T12:36:11.211Z","inboxed_at":"2025-03-10T12:36:10.914Z","running_at":"2025-03-10T12:36:11.211Z","retry_count":1,"last_start_at":"2025-03-10T12:36:11.212Z","last_attempt_deployment_id":"dp_10tsC1MOgC0nVFsmaix9uHn","last_attempt_server":"restate-sdk-java/2.0.0-SNAPSHOT_624fd2fa","status":"running"},
{"id":"inv_12dw5UBHriBL4FkVUseKdvH5CqopfY5eTS","target":"KillTestRunner/c735fb3a-4e2c-44dd-9d42-ad9c932eb107/startCallTree","target_service_name":"KillTestRunner","target_service_key":"c735fb3a-4e2c-44dd-9d42-ad9c932eb107","target_handler_name":"startCallTree","target_service_ty":"virtual_object","idempotency_key":"449df2b9-ceb8-4ebd-898b-cf37de34e464","invoked_by":"ingress","created_at":"2025-03-10T12:36:06.998Z","modified_at":"2025-03-10T12:36:11.208Z","running_at":"2025-03-10T12:36:06.999Z","completed_at":"2025-03-10T12:36:11.208Z","status":"completed","completion_result":"failure","completion_failure":"[409 Aborted] killed"},
{"id":"inv_12dw5UBHriBL5asgtMrkbz2gWGD29ThwA8","target":"AwakeableHolder/c735fb3a-4e2c-44dd-9d42-ad9c932eb107/unlock","target_service_name":"AwakeableHolder","target_service_key":"c735fb3a-4e2c-44dd-9d42-ad9c932eb107","target_handler_name":"unlock","target_service_ty":"virtual_object","idempotency_key":"2d5aa487-076c-4925-b092-836cdf5e0ff2","invoked_by":"ingress","created_at":"2025-03-10T12:36:10.837Z","modified_at":"2025-03-10T12:36:10.900Z","running_at":"2025-03-10T12:36:10.837Z","completed_at":"2025-03-10T12:36:10.900Z","status":"completed","completion_result":"success"},
{"id":"inv_12dw5UBHriBL76f2hy65GbO0KVfkDU6Sdj","target":"KillTestSingleton/c735fb3a-4e2c-44dd-9d42-ad9c932eb107/isUnlocked","target_service_name":"KillTestSingleton","target_service_key":"c735fb3a-4e2c-44dd-9d42-ad9c932eb107","target_handler_name":"isUnlocked","target_service_ty":"virtual_object","invoked_by":"ingress","created_at":"2025-03-10T12:36:11.422Z","modified_at":"2025-03-10T12:36:11.422Z","inboxed_at":"2025-03-10T12:36:11.422Z","status":"pending"}]}

[tillrohrmann]

cc @slinkydeveloper

[slinkydeveloper]

This bug exists only with the experimental feature Killed status turned on.

For now I won't fix this bug, I'll re-assess the situation when i start working on the trim and restart feature.

[tillrohrmann]

Here is potentially another instance: https://github.com/restatedev/restate/actions/runs/13784250758/job/38549770575#step:11:252

[tillrohrmann]

Another instance https://github.com/restatedev/restate/actions/runs/13817073865/job/38653893288