need a way to adminstratively hold a job

[grondo]

Flux doesn't currently support administrative hold for jobs, i.e. a hold placed by the instance owner which can only be removed by the instance owner. The instance owner can set urgency to 0, but there is nothing at this time that prevents the user from releasing their own job.

@garlick suggested that we could add a locked flag for jobs. This would be similar to the immutable flag that currently prevents jobspec updates. If the instance owner updates the urgency of a job, then this could have a side effect of locking the job so that the user can no longer modify the urgency. This may also require a new clear-flags event in RFC 21 to clear the locked flag when urgency is set nonzero.

Another idea would be to allow administrator-only access to explicitly setting priority of a job. We discarded this option because it would complicate priority plugins which would have to somehow be told the priority has been explicitly set. Administrators can already expedite jobs (by setting the urgency to its highest value) or give jobs a priority bump by setting urgency > default (16).

To summarize the work needed here:

• Add a clear-flags event to RFC 21
• Document existing flags in RFC 21 (alloc-bypass, immutable)
• Add a locked flag or similar to RFC 21
• Set locked flag on guest jobs when administrator sets urgency=0
• Clear locked flag on guest jobs when administrator sets urgency>0 and locked flag is set

Edit: BTW, the proposed difference between locked and immutable is that the first affects the urgency (for now, possibly other job attributes later), and immutable refers explicitly to jobspec updates.