fluxion cancels all running jobs in a removed queue

[grondo]

Problem: On a production system, an overlapping queue was removed with all resources in the removed queue moved into another queue. Upon a scheduler reload, Fluxion raised a fatal exception on all jobs that had started in the old queue, e.g.:

[  +2.470985] sched-fluxion-qmanager[0]: jobmanager_hello_cb: unknown queue name
 (id=166204075275191296 queue=rabbit)
[  +2.471005] sched-fluxion-qmanager[0]: raising fatal exception on running job id=fPNq8E3xAYX
[  +2.475535] job-exec[0]: exec aborted: id=fPNq8E3xAYX
[  +2.477140] sched-fluxion-qmanager[0]: jobmanager_hello_cb: unknown queue name (id=166204048498754560 queue=rabbit)
[  +2.477161] sched-fluxion-qmanager[0]: raising fatal exception on running job id=fPNq7XFovnB

Fluxion should allow these jobs to continue running. It isn't clear why Fluxion needs to validate the queue for jobs in the RUN state which already have assigned resources.

[trws]

That's pretty much what I would have expected to happen. I'm sure we can fix this, but as it sits we don't just validate it, the queue is the actual object in qmanager that defines behavior for various transitions, there's no notion of a job that doesn't have a queue so there are a number of places where we unconditionally invoke something on the queue of a job.

As you correctly point out, if the job is already in the run state, or even allocated, we could make it valid for the queue not to exist, but it may take extra checks in a number of places to avoid trying to update a queue object when a job transitions.

[grondo]

For now we can caution administrators to avoid completely removing queues if there are jobs still using the queue (especially running jobs, for which there is no way to update the queue before the change) AIUI, this was a one-off case where a queue was accidentally left enabled.