Refactor in-memory schedulers to postgresql table

[jpbruinsslot]

Current situation

The running schedulers (boefje, normalizer, and report) can be accessed from the rest API. The detailed information about these schedulers are kept in memory in an application-wide dict.

nl-kat-coordination/mula/scheduler/app.py

Lines 61 to 67 in bf3267a

	self.schedulers: dict[
	str,
	schedulers.Scheduler
	| schedulers.BoefjeScheduler
	| schedulers.NormalizerScheduler
	| schedulers.ReportScheduler,
	] = {}

When referencing what available queues there are a /queues endpoint can be called (the task runner does this to gather what queues to pop from)

nl-kat-coordination/mula/scheduler/server/handlers/queues.py

Lines 56 to 57 in bf3267a

	def list(self) -> Any:
	return [models.Queue(**s.queue.dict(include_pq=False)) for s in self.schedulers.copy().values()]

This will iterate over all the available schedulers and construct queue representations. The same goes for the available schedulers:

nl-kat-coordination/mula/scheduler/server/handlers/schedulers.py

Lines 45 to 46 in bf3267a

	def list(self) -> Any:
	return [models.Scheduler(**s.dict()) for s in self.schedulers.values()]

Currently, we don't have any filtering possibilities for these endpoints, meaning a task runner needs to poll the scheduler for available schedulers to pop from and iterate over them.

Suggested changes

• consolidate /queues and /schedulers endpoints, they are interchangeable
• move schedulers configuration and settings to a postgres table
• implement filtering of available schedulers from the rest API
• change /pop endpoint to support popping of multiple tasks (batches), and add more filtering options (e.g. pop tasks for multiple organisations)
• optional leveraging ETag (Entity Tag) or Last-Modified headers of scheduler endpoint
• NEW create one BoefjeScheduler , NormalizerScheduler and ReportScheduler for all organisations instead individual schedulers for every organisation. One message queue for all scan profile mutations, and raw file creation (Combine all schedulers for all organisations #3838)

New Functionality

• faster overview and querying of all available scheduler without relying on iterating over the in-memory schedulers
• rest API filtering options allow for specific retrieval of schedulers (e.g. filtering by created_at to retrieve schedulers that have been created since a specific timestamp)
• speed up start-up times, for already defined schedulers we can reference the database in order to create running schedulers

Considerations

• Since the current way of referencing organisations in OpenKAT we're still bound by a start-up to check how what are organisations are available in the katalogus
• Additionally we're still bound by periodically checking the katalogus for new or removed organisations. This can be optimized by sending a signal (either rest, or aqmp) to the scheduler to create scheduler for a new organisation.

[jpbruinsslot]

superseded by #3838