Investigate failed performance test

[tillrohrmann]

The performance test https://github.com/restatedev/e2e-verification-runner/actions/runs/13777670369 failed. What we could gather from the logs is that at some point the test driver was no longer able to send new invocations to the runtime. A few of the idempotent sends timed out consistently for 1,5h. The timeout is set to 1s. The system was still processing other invocations based on the log output.

Based on this, we assume that the system was no longer able to ingest and respond to an idempotent send within 1 second. Somewhere on the way from the ingress to the PP, writing to Bifrost and applying the invocation, we accumulated too much delay. Maybe the system slowed down or there were too many unprocessed messages in Bifrost to respond in time.

We need to look into this problem to better understand what's causing the apparent slow down.