Data contracts

[karolisg]

Current implementation

The current dozer version (v0.3.0) supports dynamic schema generation in protobof and openapi formats. Schema is generated based on input sources and transformations.

Contracts features

Different articles might be defining data contracts slightly differently, but essentially data contract needs to solve 4 key agreements between producers and consumers:

• schema
• semantics
• SLA
• metadata (data governance)

To ensure that, contract needs to provide:

• validations (for schema and semantics)
• defined users, roles and ACLs
• SLA commitments

Validations (for schema and semantics)

protobuf automaticaly generates schema level validations, while for json validation is done on consumer side and dozer needs to make sure that json schema contains all possible information (like min, max values for UInt and Int types)

Semantics validations are used for defining business logic and validation of data. They can be differentiated into several categories:

• Value integrity - find unacceptable, illogical data
• Outliers - find data which doesn't fit to provided condition (usually it is average, mean or something similar + thresholds)
• Referential integrity - find incomplete data. This check is similar to foreign key between two or more tables.
• Event order and state transitions - find inconsistent data. Data might not be sent in correct order or data is missing in particular state

Validation errors have several handling solutions:

• Alerts only - Only alert user if validation failed
• All or none - Make API unavailable
• Ignore failed records
• Dead letter queue - Send record to another system, which would allow user to fix data

Users, roles and ACLs

Data contracts should have clear visibility and rules on data access. This can be separated into several aspects:

• users - list of users which has access to data source
• roles - list of roles with rules of data access
• ACL - list of rules

SLA

This part of contract is used to define data quality.

Potential improvements

• Improve json schema generation to have more explicit data validations
• Allow user to define column, tables descriptions
• Implement data integrity validations support
• Implement outliers validation support
• Implement references validation
• Implement different validation error handling strategies
• Improve user access management
• Expand current security implementation with ACL and Roles and reflect that in data contract
• Allow user to define/import SLA metrics
• Store data contract for application even when application is deleted or terminated

Examples

open data contract standard full example
datacontract

Resources:

https://atlan.com/data-contracts/#how-to-handle-contract-validation-failures
https://medium.com/profitoptics/data-contract-101-568a9adbf9a9
https://sherinthomas.medium.com/data-contracts-what-is-it-and-why-should-you-care-bc19be951ec2
https://mlops.community/an-engineers-guide-to-data-contracts-pt-1/