MODSOURMAN-1195 Save job execution progress in batches

[okolawole-ebsco]

Purpose

Save job execution progress in batches to improve performance and scalability.

Approach

Instead of each record attempt to update job execution progress, a message is sent to an address over the Vert.x event bus. Upon successful placement of the message in the event bus, processing of the record continues.
On the consumer side, messages are group in batches by tenant ID and job execution ID. The group of messages are then reduced into a single job execution progress which is then saved into the database and overall job execution status is updated.
In local testing, update of job execution progress happens 10,000 times for a data import job of 10,000 records. Total execution time for all 10,000 database calls was 53 seconds(mean execution time for each call was 5.3 milliseconds). After this optimization, there were only 352 database calls with a total execution time of 17 milliseconds.

Advantages

• Less database overhead due to less commits and less contention for the same database row.
• Less database connection pool thrashing since a connection is not required to update job execution progress for each record.
• Faster processing of records since processing is no longer waiting for update of job execution progress to continue.

Disadvantages

• If for any reason the update of the reduced job execution fails, the records responsible for the update would have already been discarded. This is due to the asynchronous nature of this optimization.
• Very small data import jobs with minimal number of records can see total duration be extended by < 1000 milliseconds.

[okolawole-ebsco]

These lines were moved to the JobExecutionProgressVerticle file.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
89.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud