Stateful data checks for ingestion and other types of jobs

[yruslan]

Background

TBD

Feature

Stateful data checks for ingestion and other types of jobs

Example

• Check that the number of records in the ingested snapshot today does not deviate too much from the one ingested for the previous day
• Similar checks for transformers and sinks

Proposed Solution

• Should be configuration-only solution

[jirifilip]

Quickly looked into this for fun (was solving something similar at the previous job).

Looks like Deequ has some useful functionality we could steal 😄

• Basic guide
• There are multiple strategies you can detect anomalies with, like absolute difference, seasonal etc.

[yruslan]

Nice! Thanks for the links, they have nice ideas for inspiration!