Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(airflow): OWID, FOPH and Colombia dag. #133

Merged
merged 30 commits into from
Oct 14, 2022
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
5eefb19
Begin of owid workflow
luabida Aug 24, 2022
3d25ae1
Add logger and clean data files do owid dag
luabida Aug 25, 2022
7eb5266
Colombia data DAG
luabida Aug 26, 2022
66d214e
Add foph DAG
luabida Sep 1, 2022
27a338f
Run black
luabida Sep 1, 2022
c757d4f
DEBUG log, some table sizes won't ever match
luabida Sep 1, 2022
eee97b6
Remove duplicated file
luabida Sep 1, 2022
7514ec8
DRAFT test epigraphhub_py import
luabida Sep 2, 2022
0a7be2d
foph dag
luabida Sep 2, 2022
2aaa60d
Remove old data
luabida Sep 5, 2022
dcf79fe
Finish colombia data
luabida Sep 6, 2022
e502605
Reduce comparation bugs
luabida Sep 6, 2022
22dfc71
Minor fixes
luabida Sep 6, 2022
bb45e1f
Add stdout logging to DAGs
luabida Sep 8, 2022
ec337b5
Start of Airflow Documentation
luabida Sep 8, 2022
08e8f98
Configuring the postgres connection with .config/epigraphhub.yaml file
luabida Sep 23, 2022
caa2e13
Add documentation for owid dag
luabida Sep 23, 2022
0a207f9
FOPH docstrings plus minor adjustments
luabida Sep 26, 2022
b1af26f
Import epigraphhub from pypi
luabida Oct 10, 2022
36f08c2
chore(airflow): add initial tests
luabida Sep 26, 2022
720478c
run black
luabida Sep 26, 2022
3ec0715
Add unitary test for dags, check existence
luabida Sep 26, 2022
edf8743
Add tests to verify if table exists
luabida Oct 11, 2022
9264a6e
Merge pull request #1 from luabida/airflow-tests
luabida Oct 11, 2022
81caf6e
Fix EOF
luabida Oct 11, 2022
4f4ee21
Remove database tests
luabida Oct 11, 2022
ae3e2f8
Fix permissions for legacy Cron scripts
luabida Oct 11, 2022
e595420
Removing legacy Data Collection Cron scripts and setting Email on Fai…
luabida Oct 13, 2022
589dcb4
Removing forecast code. Deprecated.
luabida Oct 13, 2022
806b381
Fix typos
luabida Oct 14, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .github/workflows/main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -96,9 +96,12 @@ jobs:
docker ps
make docker-wait-all

- name: test cron scripts
- name: Trigger all DAGs
run: |
make docker-run-cron SERVICE=superset
make docker-console SERVICE=airflow CONSOLE='airflow dags trigger owid'
make docker-console SERVICE=airflow CONSOLE='airflow dags trigger foph'
make docker-console SERVICE=airflow CONSOLE='airflow dags trigger colombia'
make docker-console SERVICE=airflow CONSOLE='airflow dags trigger web_status_test'

- name: Generate logs
if: ${{ failure() }}
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,4 @@ Documentation/.Rhistory

# airflow
docker/airflow/files/logs/*
docker/airflow/logs/*
5 changes: 0 additions & 5 deletions Data_Collection/CRON_scripts/colombia.sh

This file was deleted.

120 changes: 0 additions & 120 deletions Data_Collection/CRON_scripts/colombia_data.py

This file was deleted.

5 changes: 0 additions & 5 deletions Data_Collection/CRON_scripts/foph.sh

This file was deleted.

47 changes: 0 additions & 47 deletions Data_Collection/CRON_scripts/foph_fetch.py

This file was deleted.

5 changes: 0 additions & 5 deletions Data_Collection/CRON_scripts/forecast.sh

This file was deleted.

Loading