need a metric to see if overlap is being hit

[shantanugadgil]

Proposal

For periodic job which have prohibit_overlap set to true, we need a method to detect that the overlap threshold is being hit.
With this metric it would be easy to detect if the configured schedule is too soon, based on runtime of the job.
Also, if the threshold is being hit often, it could be an indicator that something has changed (for the negative) in the job itself.

Use-cases

If there was an explicit metric which indicated that job FOO was hitting threshold often, we could increase it's schedule duration for longer timeframes and setup an alert in the monitoring system like DataDog, etc.

Attempted Solutions

Currently we monitor pending allocations, but cannot pinpoint if the allocation is pending due to insufficient resources or if certain jobs are hitting their overlap threshold.

[jrasell]

Hi @shantanugadgil and thanks for taking the time to write up this issue. I think it would be a good addition, although we probably want to take a minute a think about exactly what labels to add, if any, to the data points.

I wonder if it would currently be possible to alert on periodic jobs which run for longer than expected, as a way to mitigate this pending any work on this request.

[shantanugadgil]

I hadn't given a thought to what the label(s) might be named. Nothing much comes to mind immediately. Something like overlap_threshold_crossed, maybe? Also the next requirement could be to record the amount of time by which it crossed the threshold (which though I think can be figured out by the timestamp itself, not sure)

I wonder if it would currently be possible to alert on periodic jobs which run for longer than expected, as a way to mitigate this pending any work on this request.

This, I think is a separate problem about batch job not having a timeout, correct?

We are mitigating this currently by running a loop to check which periodic jobs "appear stuck"
#1782 (comment)