Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task/3 data validation #19

Merged
merged 7 commits into from
Jul 11, 2020
Merged

Task/3 data validation #19

merged 7 commits into from
Jul 11, 2020

Conversation

OldMetalmind
Copy link
Contributor

Close #3 #4

Description

This pull requests implements big changes on the script.

Changes:

  • Creates two classes: MortalityScrapping, MortalityReport
    -- MortalityScrapping handles everything regarding data scrapping, processing and writing to file.
    -- MortalityReport verifies data structure and writes a small json report.
  • Github Action mortality_data_extraction.yml now runs every hour and it will ran the scraping.py and the respective tests
  • Github Action data_extraction_validation will also run the scraping.py script at the end of every other step

@OldMetalmind OldMetalmind requested a review from piresn July 11, 2020 16:13
Copy link
Contributor

@piresn piresn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

O refactoring com as 2 classes faz bastante sentido. Só detectei um problema com o logging (todos os dias têm missing values, mas eles são de esperar em várias colunas. Tb tenho uma sugestão para o scheduling do Actions. De resto parece-me bem!

@OldMetalmind OldMetalmind merged commit 2e4293b into master Jul 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Validação e normalização dos dados
3 participants