[Heartbeat] Adjust State loader timeout to reasonable value

[vigneshshanmugam]

Summary

Heartbeat uses state loader to get the last status from the ES cluster and loads the current monitor state after the monitor has been successfully ran. The ES loader has a backoff of 3 retries with a 90 second Default Request timeout which in case of connection failures extend the duration of a monitor run leading to skipped executions.

Proposal

These higher values of timeouts cannot be applied on the SAAS Service where we are running in tight loops and want the next successive check to be run for a given monitor within the next scheduled interval which could be 1 or 3m. To overview these scenarios, we have to adjust the ES default transport timeout to the reasonable timeout that is similar to the State loader connection establishment timeouts.

I proposed, we keep the search timeout to less than 3 seconds and keep the 3 retries as we have currently. Thoughts?

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)