Background Processes API (core)

[jbee]

Motivation

The server is performing different types of "long running processes" that aren't easily observable for users.
This ranges from processes that are entirely "invisible" to those that can be observed by querying different APIs in combination.
However, this requires deep knowledge about how each type can be observed and which values to look at, which IDs to copy and insert into further queries.
Even for people with that knowledge a simple goal of observing becomes something that takes minutes rather than seconds.
This makes working on features with such "background processes" unnecessary difficult and time consuming.

Proposal

The proposed solution to the above dilemma is to introduce a "background processes" API to the core.
This API would give the user an instant high level overview of what the server is working on in the background.
This would be no means be a complete picture. Only processes that explicitly have been added and which progress is maintained are visible.

A good example of such processes would be the job scheduler execution. Jobs running in the scheduler are observable in detail using the according APIs but getting an idea of what happens in the server right now and some minutes ago is not easy to aggregate quickly.

Following this example the background processes API is not about measuring performance or having a detailed insight into the processes. It is about getting an overview at a glance of what happens now and what did happen a while ago.

This is very much comparable to the background processes tab in an IDE except that it would also have a sliding window letting you see the historic (finished) processes.

A crud mockup could look like this

UID	description	from...to
a1234567890	Data Integrity Checks	12:34...now
b0987654321	Data Import	12:33...now
x1234567890	Analytics Table Generation	04:00...04:34

This tabular view would be sorted by starting time and show the process with a UID, a name and the start and finish times.
To prevent consuming significant resources the entries would be held in a fixed size buffer of some sorts providing a "sliding window" which discards the oldest entries to make room for new ones.
Entries are identified by the UID of the underlying object, like the job they represent.
This is used to identify that an entry should be updated with the end time.

API

Just to flesh out the idea a bit more concrete the API could look something like the following.

A GET request can be made to /system/backgroundProcesses resulting in a JSON array of entries

[
{ "id": "...", "description": "A...", "from": <timestamp> "to": <timestamp> },
{ "id": "...", "description": "B...", "from": <timestamp> "to": <timestamp> },
...
]

Internally this is maintained in a fixed size queue.
With the datastore having a the fixed collection feature and partial updates it could be used to hold the entries.
This would make the information available even after a restart.
A BackgroundProcessService would merely be a thin layer on top of the datastore.
Source of background processes would explicitly call the service to make the process visible.
The interface could look something like this

interface BackgroundProcessService {
  void start(UID process, String description);
  void finished(UID process);
}

Implementation Notes

The datastore has a "append to collection" feature with size cap.
However, this does not allow to update existing entries, for example to complement an entry with the end.
The solution here is to now update entries ever. Instead each "event" is recorded as is and the related events (by UID) are then merged to the object that is visible in the API.
So internally this { "id": "X", "description": "bla", "from": 0 "to": 100 } entry would consist of two

[
  { "id": "X", "description": "bla", "from": 0 },
  { "id": "X", "to": 100 }
]

This also leaves room to extend the information kept.

Given that some types of events happen very frequent and other just once a day it makes sense to not have a strictly linear history as the majority of the history would consist of entries with high frequency. Instead the history might be per type.
For example, jobs would use their job type as type. The entire processes data would be an object with members that each represent a type and have a array value as described earlier.

{
  "HOUSEKEEPING": [{ "id": "X", "description": "bla", "from": 0 }, ...],
  "DATA_INTEGRITY": [...],
  "METADATA_IMPORT": [...],
  ...
}

Such data is no longer strictly discarding the oldest entry in the entire data structure but the oldest for the type. This way the interesting information for the different types of events is kept even if some events are very frequent while others are rare.
In the API the event type level would be folded into a flat list again, e.g.

[
{ "id": "...", "event": "HOUSEKEEPING", "description": "A...", "from": <timestamp> "to": <timestamp> },
{ "id": "...", "event": "DATA_INTEGRITY", "description": "B...", "from": <timestamp> "to": <timestamp> },
...
]

This means the actual interface for the service would be

interface BackgroundProcessService {
  void start(String event, UID process, String description);
  void finished(String event, UID process);
}