Add compare tool #280
Merged
Add compare tool #280
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
As I mentioned above, most of the diffing logic is done in the other PR. If you can, please focus on the cli itself in this one. To run it, do poetry install
poetry shell
compare dataframes DF1 DF2
# or
compare etl_catalog CHANNEL NAMESPACE DATASET TABLE |
This was referenced Jul 11, 2022
|
Hi Daniel, thanks a lot for this! The CLI looks very neat. I can't go through the PR now in detail, but I have some general preliminary comments:
|
Marigold
approved these changes
Jul 12, 2022
| def yield_list_lines( | ||
| description: str, items: Iterable[Any] | ||
| ) -> Generator[str, None, None]: | ||
| sublines = [item for item in items] |
There was a problem hiding this comment.
(Could be list(items) I think)
| ) | ||
|
|
||
|
|
||
| def load_table(path_str: str) -> catalog.Table: |
There was a problem hiding this comment.
(There's a similar method that could be used instead, but it'd probably look weirder than your method)
| help="Print truncated lists if they are longer than the given length.", | ||
| ) | ||
| def etl_catalog( | ||
| channel: str, |
There was a problem hiding this comment.
(For me it would be more natural to write it as a path / URI channel/namespace/dataset/table instead of having them separated)
pabloarosado
approved these changes
Jul 15, 2022
larsyencken
force-pushed
the
compare-tool
branch
from
1dbf7bb to
f29bc79
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements the first version of a compare CLI that allows easy comparison of either arbitrary dataframes (feather, csv, parquet) or etl files (in that case against the version currently uploaded in the production catalog). Much of the logic for the diffing lives in the data-utils-py repo and is managed in this PR: owid/owid-datautils-py#29 .
This PR currently includes a temporary copy of the new HighlevelDiff class for development. It will be removed here and imported from data-utils-py when the PR over there is merged.