Improve performance with lots of jobs

[sampsyo]

Several operations in Polyphemus are linear in the total job count. For example, to find the latest job, you have to list all the jobs on the filesystem, open their JSON documents, and sort the result.

It would be great to someday not slow down too much in the presence of lots of old jobs, perhaps by caching some metadata in an index.

(Expanded from #1.)

[rachitnigam]

Note to future self: This is hard because of concurrent worker execution. Workers modifying a meta-data file without locks will corrupt the index file.