📊 human fertility database

[lucasrodes]

tracking issue: https://github.com/owid/owid-issues/issues/1412

This is a first attempt to import the Human Fertility Database to ETL. It contains several indicators, so in this PR I'll focus on the so-called "summary indicators" (see section 6 in their Methods Protocol report).

Progress

• Import snapshot
• Import Meadow
  • Inspect each file
  • Read each file as table and format
• Import garden
  • Read all tables
  • Consolidate all tables into fewer ones
  • Minimal metadata
• Import Grapher
• Export to DB
• Work on metadata

[owidbot]

Quick links (staging server):

Site Dev	Site Preview	Admin	Wizard	Docs

Login: ssh owid@staging-site-fertility-data

chart-diff: ✅

• 10/10 reviewed charts
• Modified: 0/0
• New: 10/10
• Rejected: 0

data-diff: ✅ No differences found

+ Dataset garden/hmd/2024-11-19/hfd
+ + Table cohort
+   + Column cmab
+   + Column cmab40
+   + Column sdcmab
+   + Column sdcmab40
+   + Column ccf
+   + Column ccf40
+ + Table period
+   + Column adjtfr
+   + Column cbr
+   + Column mab
+   + Column mab40
+   + Column patfr
+   + Column tmab
+   + Column sdmab
+   + Column sdmab40
+   + Column tfr
+   + Column tfr40
+   + Column b
+ Dataset garden/war/2024-11-22/ucdp_preview
+ + Table ucdp_preview
+   + Column number_deaths_ongoing_conflicts
+   + Column number_deaths_ongoing_conflicts_high
+   + Column number_deaths_ongoing_conflicts_low
+   + Column number_deaths_ongoing_conflicts_civilians
+   + Column number_deaths_ongoing_conflicts_unknown
+   + Column number_deaths_ongoing_conflicts_combatants
+   + Column number_ongoing_conflicts
+   + Column number_new_conflicts
+   + Column number_ongoing_conflicts_per_country
+   + Column number_ongoing_conflicts_per_country_pair
+   + Column number_new_conflicts_per_country
+   + Column number_new_conflicts_per_country_pair
+   + Column number_deaths_ongoing_conflicts_per_capita
+   + Column number_deaths_ongoing_conflicts_high_per_capita
+   + Column number_deaths_ongoing_conflicts_low_per_capita
+ + Table ucdp_preview_country
+   + Column participated_in_conflict
+   + Column number_participants
+ + Table ucdp_preview_locations
+   + Column number_deaths
+   + Column number_deaths_high
+   + Column number_deaths_low
+   + Column number_deaths_civilians
+   + Column number_deaths_unknown
+   + Column number_deaths_combatants
+   + Column is_location_of_conflict
+   + Column death_rate
+   + Column death_rate_high
+   + Column death_rate_low
+   + Column number_locations


Legend: +New  ~Modified  -Removed  =Identical  Details
Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet

Automatically updated datasets matching weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included

Edited: 2024-11-22 15:24:44 UTC
Execution time: 317.48 seconds

[lucasrodes]

@paarriagadap doesn't need a thorough revision.

Maybe a high-level review of the metadata, etc.

The Garden step is complicated because the original data comes in a particular format. Meadow has about 60 tables, and in Garden, I try to consolidate as many as possible into fewer. Feel free also to skim the Garden step rapidly.

Thanks!

[paarriagadap]

@lucasrodes It seems you haven't push the latest changes? I don't see much metadata nor code in garden

[paarriagadap]

Very good! Here we have some comments

[paarriagadap]

You can also define Total fertility rate (TFR) somewhere and then use TFR in the following texts

[lucasrodes]

I don't fully understand your suggestion here. There are various TFR types, each with a different definition. E.g. for this one, adjtfr it needs a slight variant of the original tfr definition.