Reset jobs to pending when memory is low

[madwort]

Stop jobs to pending if we dip below a threshold of free memory on a backend. Based on the most recent incident, my initial suggestion is 45GB free. We should try to use the same principle as db maintenance mode, i.e. reset to pending so that they automatically restart when resources become available (instead of erroring & requiring the end-user/researcher to re-run the job). My idea is that the most recently started job should be killed (i.e. FIFO).

sibling ticket of #699

see discussion in slack thread

[madwort]

some low free mem occurences recently, but no outages afaik

https://ui.honeycomb.io/bennett-institute-for-applied-data-science/environments/production/datasets/jobrunner-metrics/result/sVV9A2Ti16R/a/sm1Np2Wj7kY/Total-free-memory?omitMissingValues

https://ui.honeycomb.io/bennett-institute-for-applied-data-science/environments/production/datasets/jobrunner-metrics/result/Jkq2JzEHjcs/a/sm1Np2Wj7kY/Total-free-memory?omitMissingValues

https://ui.honeycomb.io/bennett-institute-for-applied-data-science/environments/production/datasets/jobrunner-metrics/result/vvVzb6my9h3/a/sm1Np2Wj7kY/Total-free-memory?omitMissingValues

https://ui.honeycomb.io/bennett-institute-for-applied-data-science/environments/production/datasets/jobrunner-metrics/result/AfDKEXtyGL1/a/sm1Np2Wj7kY/Total-free-memory?omitMissingValues