📊 emissions: Update emissions and energy datasets (#3582)

* 📊 emissions: Update emissions and energy datasets * Refactor emissions steps to create new release of CO2 dataset * Add snapshot links to GCB data * Archive unused steps * Improve owid_co2 * Write data and codebook to github repos as well * Create export steps to write to S3 * Create codebook in garden, and improve github and s3 steps * Fix order of rows in codebook * Add sanity checks * Minor improvements in README * Update Global Carbon Budget data * Improve format * Fix small bug in anomalist * Archive unused steps * Add commented code for sanity checks * Fix wrong link * Warn that S3 export step does not work, but for now do nothing
owid · Nov 22, 2024 · b8564d3 · b8564d3
1 parent 892fcef
commit b8564d3
Show file tree

Hide file tree

Showing 25 changed files with 3,985 additions and 404 deletions.
diff --git a/apps/anomalist/anomalist_api.py b/apps/anomalist/anomalist_api.py
@@ -121,14 +121,14 @@ def renormalize_score(
 
 # Function to format population numbers.
 def pretty_print_number(number):
-    if number >= 1e9:
+    if pd.isna(number):
+        return "?"
+    elif int(number) >= 1e9:
         return f"{number/1e9:.1f}B"
     elif number >= 1e6:
         return f"{number/1e6:.1f}M"
     elif number >= 1e3:
         return f"{number/1e3:.1f}k"
-    elif pd.isna(number):
-        return "?"
     else:
         return f"{int(number)}"
 

diff --git a/dag/archive/emissions.yml b/dag/archive/emissions.yml
@@ -48,3 +48,72 @@ steps:
   data://garden/emissions/2024-02-26/gdp_and_co2_decoupling:
     - data://garden/worldbank_wdi/2024-05-20/wdi
     - data://garden/gcp/2023-12-12/global_carbon_budget
+  #
+  # GCP - Global Carbon Budget (published on 2023-12-05, updated on 2023-12-12 to use the latest primary energy data).
+  #
+  data://meadow/gcp/2023-12-12/global_carbon_budget:
+    - snapshot://gcp/2023-12-12/global_carbon_budget_fossil_co2_emissions.csv
+    - snapshot://gcp/2023-12-12/global_carbon_budget_national_emissions.xlsx
+    - snapshot://gcp/2023-12-12/global_carbon_budget_land_use_change_emissions.xlsx
+    - snapshot://gcp/2023-12-12/global_carbon_budget_global_emissions.xlsx
+  #
+  # GCP - Global Carbon Budget (published on 2023-12-05, updated on 2024-06-20 to use the latest primary energy data).
+  #
+  data://garden/gcp/2024-06-20/global_carbon_budget:
+    - data://garden/wb/2024-03-11/income_groups
+    - data://garden/demography/2023-03-31/population
+    - data://garden/energy/2024-06-20/primary_energy_consumption
+    - data://garden/regions/2023-01-01/regions
+    - data://meadow/gcp/2023-12-12/global_carbon_budget
+    - data://garden/ggdc/2024-04-26/maddison_project_database
+  #
+  # Decoupling of GDP and CO2 (2023).
+  #
+  data://garden/emissions/2024-06-20/gdp_and_co2_decoupling:
+    - data://garden/gcp/2024-06-20/global_carbon_budget
+    - data://garden/worldbank_wdi/2024-05-20/wdi
+  #
+  # GCP - Global Carbon Budget.
+  #
+  data://grapher/gcp/2024-06-20/global_carbon_budget:
+    - data://garden/gcp/2024-06-20/global_carbon_budget
+  #
+  # GCP - Global Carbon Budget.
+  #
+  data://meadow/gcp/2024-11-13/global_carbon_budget:
+    - snapshot://gcp/2024-11-13/global_carbon_budget_national_emissions.xlsx
+    - snapshot://gcp/2024-11-13/global_carbon_budget_fossil_co2_emissions.csv
+    - snapshot://gcp/2024-11-13/global_carbon_budget_global_emissions.xlsx
+    - snapshot://gcp/2024-11-13/global_carbon_budget_land_use_change_emissions.xlsx
+  #
+  # GCP - Global Carbon Budget.
+  #
+  data://garden/gcp/2024-11-13/global_carbon_budget:
+    - data://garden/ggdc/2024-04-26/maddison_project_database
+    - data://garden/wb/2024-07-29/income_groups
+    - data://garden/demography/2024-07-15/population
+    - data://meadow/gcp/2024-11-13/global_carbon_budget
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/energy/2024-06-20/primary_energy_consumption
+  #
+  # Emissions - CO2 dataset.
+  #
+  data://garden/emissions/2024-11-13/owid_co2:
+    - data://garden/ggdc/2024-04-26/maddison_project_database
+    - data://garden/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/gcp/2024-11-13/global_carbon_budget
+    - data://garden/emissions/2024-04-08/national_contributions
+    - data://garden/climate_watch/2023-10-31/emissions_by_sector
+    - data://garden/energy/2024-06-20/primary_energy_consumption
+  #
+  # Decoupling of GDP and CO2 (2023).
+  #
+  data://garden/emissions/2024-11-13/gdp_and_co2_decoupling:
+    - data://garden/gcp/2024-11-13/global_carbon_budget
+    - data://garden/worldbank_wdi/2024-05-20/wdi
+  #
+  # GCP - Global Carbon Budget.
+  #
+  data://grapher/gcp/2024-11-13/global_carbon_budget:
+    - data://garden/gcp/2024-11-13/global_carbon_budget
diff --git a/dag/emissions.yml b/dag/emissions.yml
@@ -24,14 +24,6 @@ steps:
   data://grapher/andrew/2019-12-03/co2_mitigation_curves_2celsius:
     - data://garden/andrew/2019-12-03/co2_mitigation_curves
   #
-  # GCP - Global Carbon Budget (published on 2023-12-05, updated on 2023-12-12 to use the latest primary energy data).
-  #
-  data://meadow/gcp/2023-12-12/global_carbon_budget:
-    - snapshot://gcp/2023-12-12/global_carbon_budget_fossil_co2_emissions.csv
-    - snapshot://gcp/2023-12-12/global_carbon_budget_global_emissions.xlsx
-    - snapshot://gcp/2023-12-12/global_carbon_budget_national_emissions.xlsx
-    - snapshot://gcp/2023-12-12/global_carbon_budget_land_use_change_emissions.xlsx
-  #
   # RFF - World Carbon Pricing (2022-09-14).
   #
   data://meadow/rff/2023-10-19/world_carbon_pricing:
@@ -93,63 +85,52 @@ steps:
   data://grapher/emissions/2024-04-08/national_contributions:
     - data://garden/emissions/2024-04-08/national_contributions
   #
-  # GCP - Global Carbon Budget (published on 2023-12-05, updated on 2024-06-20 to use the latest primary energy data).
+  # GCP - Global Carbon Budget.
   #
-  data://garden/gcp/2024-06-20/global_carbon_budget:
-    - data://garden/ggdc/2024-04-26/maddison_project_database
-    - data://meadow/gcp/2023-12-12/global_carbon_budget
-    - data://garden/demography/2023-03-31/population
-    - data://garden/regions/2023-01-01/regions
-    - data://garden/wb/2024-03-11/income_groups
-    - data://garden/energy/2024-06-20/primary_energy_consumption
+  data://meadow/gcp/2024-11-21/global_carbon_budget:
+    - snapshot://gcp/2024-11-21/global_carbon_budget_fossil_co2_emissions.csv
+    - snapshot://gcp/2024-11-21/global_carbon_budget_land_use_change_emissions.xlsx
+    - snapshot://gcp/2024-11-21/global_carbon_budget_global_emissions.xlsx
+    - snapshot://gcp/2024-11-21/global_carbon_budget_national_emissions.xlsx
   #
   # GCP - Global Carbon Budget.
   #
-  data://grapher/gcp/2024-06-20/global_carbon_budget:
-    - data://garden/gcp/2024-06-20/global_carbon_budget
+  data://garden/gcp/2024-11-21/global_carbon_budget:
+    - data://garden/demography/2024-07-15/population
+    - data://meadow/gcp/2024-11-21/global_carbon_budget
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-07-29/income_groups
+    - data://garden/energy/2024-06-20/primary_energy_consumption
+    - data://garden/ggdc/2024-04-26/maddison_project_database
   #
   # Decoupling of GDP and CO2 (2023).
   #
-  data://garden/emissions/2024-06-20/gdp_and_co2_decoupling:
+  data://garden/emissions/2024-11-21/gdp_and_co2_decoupling:
     - data://garden/worldbank_wdi/2024-05-20/wdi
-    - data://garden/gcp/2024-06-20/global_carbon_budget
+    - data://garden/gcp/2024-11-21/global_carbon_budget
   #
-  # GCP - Global Carbon Budget (published on 2023-12-05, updated on 2023-12-12 to use the latest primary energy data).
+  # GCP - Global Carbon Budget.
   #
-  data://meadow/gcp/2024-11-13/global_carbon_budget:
-    - snapshot://gcp/2024-11-13/global_carbon_budget_national_emissions.xlsx
-    - snapshot://gcp/2024-11-13/global_carbon_budget_land_use_change_emissions.xlsx
-    - snapshot://gcp/2024-11-13/global_carbon_budget_fossil_co2_emissions.csv
-    - snapshot://gcp/2024-11-13/global_carbon_budget_global_emissions.xlsx
+  data://grapher/gcp/2024-11-21/global_carbon_budget:
+    - data://garden/gcp/2024-11-21/global_carbon_budget
   #
-  # GCP - Global Carbon Budget (published on 2023-12-05, updated on 2024-06-20 to use the latest primary energy data).
+  # Emissions - CO2 dataset.
   #
-  data://garden/gcp/2024-11-13/global_carbon_budget:
-    - data://garden/wb/2024-07-29/income_groups
-    - data://garden/regions/2023-01-01/regions
+  data://garden/emissions/2024-11-21/owid_co2:
     - data://garden/demography/2024-07-15/population
-    - data://garden/ggdc/2024-04-26/maddison_project_database
-    - data://meadow/gcp/2024-11-13/global_carbon_budget
+    - data://garden/emissions/2024-04-08/national_contributions
+    - data://garden/gcp/2024-11-21/global_carbon_budget
+    - data://garden/regions/2023-01-01/regions
     - data://garden/energy/2024-06-20/primary_energy_consumption
+    - data://garden/climate_watch/2023-10-31/emissions_by_sector
+    - data://garden/ggdc/2024-04-26/maddison_project_database
   #
-  # GCP - Global Carbon Budget.
-  #
-  data://grapher/gcp/2024-11-13/global_carbon_budget:
-    - data://garden/gcp/2024-11-13/global_carbon_budget
-  #
-  # Decoupling of GDP and CO2 (2023).
+  # Emissions - CO2 dataset.
   #
-  data://garden/emissions/2024-11-13/gdp_and_co2_decoupling:
-    - data://garden/worldbank_wdi/2024-05-20/wdi
-    - data://garden/gcp/2024-11-13/global_carbon_budget
+  export://github/co2_data/latest/owid_co2:
+    - data://garden/emissions/2024-11-21/owid_co2
   #
   # Emissions - CO2 dataset.
   #
-  export://github/co2_data/latest/owid_co2:
-    - data://garden/ggdc/2024-04-26/maddison_project_database
-    - data://garden/demography/2023-03-31/population
-    - data://garden/regions/2023-01-01/regions
-    - data://garden/emissions/2024-04-08/national_contributions
-    - data://garden/gcp/2024-11-13/global_carbon_budget
-    - data://garden/climate_watch/2023-10-31/emissions_by_sector
-    - data://garden/energy/2024-06-20/primary_energy_consumption
+  export://s3/co2_data/latest/owid_co2:
+    - data://garden/emissions/2024-11-21/owid_co2