Incorrect job status reported due to caching

[warownia1]

For performance reasons, results of qstat for Grind Engine and squeue for Slurm are cached for 5 seconds. It is fine for existing jobs, however, whenever a new job or batch of jobs is submitted to the system, their entries are missing from the cache. This may make the scheduler assume that the job was completed or deleted even though it is an old cache result.

Possible solution include

• invalidate caches every time new jobs are submitted
• re-run qstat/squeue if the requested job id was not found (may result in unnecessary cache invalidations for completed jobs which disappeared from the jobs list)

[warownia1]

cachetools library provides many caching tools including ttl cache with capability to invalidate cache on demand.

[warownia1]

Fixed by a41d060