Add a very high level comparison tool to diff the results of entire branches of the ETL

[danyx23]

#280 adds a high level diff tool for one table at a time. What we would like to have for staging (or feature branches) is a tool that can efficiently diff everything against master. To do so we don't want to calculate diffs for every file in the catalog twice, so I think we should first check md5 hashes (do we have those on the file level already?) and then compare them against a target catalog. It would be great if that catalog could be our main remote one as well (I think S3 can provide md5 hashes for files without having to download them?).

It would also be good to have the ability to check only a subset of all tables with this tool (e.g. everything in garden/OWID)

[Marigold]

check md5 hashes (do we have those on the file level already?)

We have them as column checksum in our catalog, so it's enough to download both production and staging catalog and compare them for out of sync data.

I think S3 can provide md5 hashes for files without having to download them?

True!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Marigold]

@danyx23 how about reopening this one? It's quite relevant for our ETL CI efforts.

[Marigold]

Implemented with datadiff.