[Serve] Add per-request replica failure cache in LB to reduce redundant retries

[cblmemo]

Previously, the retry on our load balancer is just select a ready replica, which is possible to select the same replica that failed last time, and its status might not be updated to the LB as the sync with controller only happens every 20s. This PR try to avoid those replica that failed once and trying other instead.

Also, update the controller LB sync interval to 10s to keep align with the probing interval (which is also 10s).

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[Michaelvll]

Thanks for the fix @cblmemo! Looks mostly good to me.

[Michaelvll]

Thanks for the update @cblmemo!

[cblmemo]

@Michaelvll bump for this

[cblmemo]

bump for review @Michaelvll

[Michaelvll]

I am wondering how effective the new failed_repica_urls is compared to the original globally increased index? Can we simulate the case when a replica went down and there is a different load and see the success rate / latency?

[Michaelvll]

Sorry, did not mean to approve this PR. I feel we need some benchmark to see whether this handling is actually effective

[github-actions]

This PR is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[andylizf]

I conducted a local simulation benchmark, and here are some results:

• With purely random failures, the new per-request failed-replica skipping shows a small improvement (~+1%–2% success).
• In scenarios with a permanently failing node, the new approach achieves higher success (up to +2%–3%) and slightly lower latency.
• Under combined spot-like downscaling and a permanently failing node, the new approach shows a clear edge (+1.8% success rate improvement in our tests).

For details, please check the benchmark code: andylizf/skypilot-lb-benchmark. Cc'ing @cblmemo