feat: improved data integrity execution [DHIS2-16223] #15789

jbee · 2023-11-29T10:56:36Z

Summary

Various improvements around the data integrity execution.

details jobs run as a new job type DATA_INTEGRITY_DETAILS so they are not blocked by a summary still running
the execution time of jobs is remembered to allow sorting them fast to slow when executing a set of checks
slow checks always run last (unless we know their execution time and it is fast)
checks maintain a transient averageExecutionTime property that is also readable in the API
checks have a isProgrammatic flag if they are not SQL based
all programmatic checks are considered slow now (because of that * now also expands to slow tests)
SQL based checks take precedence over programmatic checks of the same name

API

The GET /api/dataIntegrity endpoint has two added filters

slow: e.g. /api/dataIntegrity?slow=true to get only slow checks
programmatic: e.g. /api/dataIntegrity?programmatic=false to get only SQL based checks

The new filters can be combined with each other and with the existing section filter (AND logic).

Automatic Testing

New unit tests were added to tests the execution time tracking in DataIntegrityCheck as well as the fast to slow sorting.

New integration tests were added to test the added slow and programmatic filters for the API.

The execution of the new DATA_INTEGRITY_DETAILS job type is already covered by many tests that do details checks.

Manual Testing

Testing filters, try different combinations and observe the result items, e.g.

/api/dataIntegrity?slow=true vs /api/dataIntegrity?slow=true vs /api/dataIntegrity
/api/dataIntegrity?programmatic=true vs /api/dataIntegrity?programmatic=true vs /api/dataIntegrity
/api/dataIntegrity?programmatic=true&section=Categories

david-mackessy

nice work. nice to be able to visualise the programmatic checks.

codecov · 2023-11-29T15:27:26Z

Codecov Report

Merging #15789 (377587e) into master (997692a) will increase coverage by 0.00%.
The diff coverage is 93.47%.

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #15789   +/-   ##
=========================================
  Coverage     66.32%   66.33%           
- Complexity    31414    31429   +15     
=========================================
  Files          3488     3489    +1     
  Lines        130069   130103   +34     
  Branches      15198    15201    +3     
=========================================
+ Hits          86272    86300   +28     
- Misses        36716    36719    +3     
- Partials       7081     7084    +3

Flag	Coverage Δ
integration	`50.04% <93.47%> (+<0.01%)`	⬆️
integration-h2	`32.34% <10.86%> (-0.01%)`	⬇️
unit	`30.35% <30.43%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...rg/hisp/dhis/dataintegrity/DataIntegrityCheck.java	`100.00% <100.00%> (ø)`
...ava/org/hisp/dhis/scheduling/JobConfiguration.java	`76.11% <ø> (ø)`
...his/dataintegrity/DefaultDataIntegrityService.java	`76.75% <100.00%> (+0.67%)`	⬆️
...hisp/dhis/dataintegrity/jobs/DataIntegrityJob.java	`72.22% <ø> (-11.12%)`	⬇️
...rc/main/java/org/hisp/dhis/scheduling/JobType.java	`94.93% <66.66%> (+0.06%)`	⬆️
...is/dataintegrity/jobs/DataIntegrityDetailsJob.java	`83.33% <83.33%> (ø)`
...his/webapi/controller/DataIntegrityController.java	`88.63% <92.30%> (+2.52%)`	⬆️

... and 4 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 997692a...377587e. Read the comment docs.