Proposal

The current situation is that we create individual 'schedulers' for every organisation, this amounts to 3 schedulers for every organisation.

The addition of queues in the scheduler being persisted as a database table allows us to explore the possibility of running a dedicated schedulers for all organisations.

Create one BoefjeScheduler, NormalizerScheduler and ReportScheduler for all organisations instead of individual schedulers for every organisation. One message queue for all scan profile mutations to which scan profile mutations are posted of all organisations for the BoefjeScheduler, and one message queue for raw file creation of all organisations for the NormalizerScheduler, the ReportScheduler will reference its internal database table.

Advantages

• Less overhead of creating and removing schedulers for every organisation
• Stops continuous checking of available organisations
• Allows to replicate scheduler application instances when used with a blocking messaging queue
• Additionally we need to support popping multiple tasks from the endpoint

Disadvantages

• the allow_updates, allow_replace, allow_priority_updates for individual organisations can't be used, however this function doesn't seem to be used and is for every type scheduler the same
• we should investigate how we solve when organisations wan to disable a scheduler, and even if this has a use-case

Impact

• now we have several different rabbitmq messaging queues for every organisation, this needs to be reduced to one
• organisation ids need to be passed in the messages on the queue
• changes to task runner to pop off tasks
• rocky likely needs to be update to interface with changes from the scheduler api

PR

#3839