📊 switch WDI from sources to origins #2696
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Marigold
force-pushed
the
wdi-sources-to-origins
branch
2 times, most recently
from
63762a9 to
1582d5d
Compare
|
Quick links (staging server):
Login: chart-diff: ✅
data-diff:= Dataset garden/education/2023-07-17/education_barro_lee_projections
= Table education_barro_lee_projections
= Dataset garden/education/2023-07-17/education_lee_lee
= Table education_lee_lee
~ Dim country
+ + New values: 231 / 12737 (1.81%)
year country
2022 Middle-income countries
2022 Myanmar
2022 Romania
2022 Sint Maarten (Dutch part)
2022 United Kingdom
~ Dim year
+ + New values: 231 / 12737 (1.81%)
country year
Middle-income countries 2022
Myanmar 2022
Romania 2022
Sint Maarten (Dutch part) 2022
United Kingdom 2022
~ Column f_adults__25_64_years__percentage_of_no_education (new data)
+ + New values: 231 / 12737 (1.81%)
country year f_adults__25_64_years__percentage_of_no_education
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column f_primary_enrollment_rates_combined_wb (changed metadata, new data)
- - description: |-
+ + description_from_producer: |-
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year f_primary_enrollment_rates_combined_wb
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column f_secondary_enrollment_rates_combined_wb (changed metadata, new data)
- - description: |-
+ + description_from_producer: |-
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year f_secondary_enrollment_rates_combined_wb
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column f_tertiary_enrollment_rates_combined_wb (changed metadata, new data, changed data)
- - description: |-
+ + description_from_producer: |-
- - - producer: UNESCO (via World Bank)
+ + - producer: UNESCO Institute for Statistics
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year f_tertiary_enrollment_rates_combined_wb
Middle-income countries 2022 43.13060
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 5.66911
United Kingdom 2022 NaN
~ Changed values: 4437 / 12737 (34.84%)
country year f_tertiary_enrollment_rates_combined_wb - f_tertiary_enrollment_rates_combined_wb +
Ethiopia 1997 0.333790 0.325650
Lesotho 1995 2.506810 2.460640
Namibia 2020 35.555290 36.926449
Spain 1989 34.452919 34.603401
Turkey 1987 6.931130 6.468240
~ Column f_youth_and_adults__15_64_years__average_years_of_education (new data)
+ + New values: 231 / 12737 (1.81%)
country year f_youth_and_adults__15_64_years__average_years_of_education
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column f_youth_and_adults__15_64_years__percentage_of_no_education (new data)
+ + New values: 231 / 12737 (1.81%)
country year f_youth_and_adults__15_64_years__percentage_of_no_education
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column female_over_male_enrollment_rates_primary (changed metadata, new data)
- - description: |-
+ + description_from_producer: |-
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year female_over_male_enrollment_rates_primary
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column female_over_male_enrollment_rates_secondary (changed metadata, new data)
- - description: |-
+ + description_from_producer: |-
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year female_over_male_enrollment_rates_secondary
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column female_over_male_enrollment_rates_tertiary (changed metadata, new data, changed data)
- - description: |-
+ + description_from_producer: |-
- - - producer: UNESCO (via World Bank)
+ + - producer: UNESCO Institute for Statistics
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year female_over_male_enrollment_rates_tertiary
Middle-income countries 2022 1.122979
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 4.986945
United Kingdom 2022 NaN
~ Changed values: 4438 / 12737 (34.84%)
country year female_over_male_enrollment_rates_tertiary - female_over_male_enrollment_rates_tertiary +
Liechtenstein 2012 0.553084 0.538128
Mexico 2019 1.055538 1.062011
Thailand 2016 1.407331 1.430745
Turkey 2005 0.730853 0.748266
Zambia 1988 0.376277 0.363264
~ Column m_primary_enrollment_rates_combined_wb (changed metadata, new data)
- - description: |-
+ + description_from_producer: |-
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year m_primary_enrollment_rates_combined_wb
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column m_secondary_enrollment_rates_combined_wb (changed metadata, new data)
- - description: |-
+ + description_from_producer: |-
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year m_secondary_enrollment_rates_combined_wb
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column m_tertiary_enrollment_rates_combined_wb (changed metadata, new data, changed data)
- - description: |-
+ + description_from_producer: |-
- - - producer: UNESCO (via World Bank)
+ + - producer: UNESCO Institute for Statistics
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year m_tertiary_enrollment_rates_combined_wb
Middle-income countries 2022 38.407299
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 1.136790
United Kingdom 2022 NaN
~ Changed values: 4441 / 12737 (34.87%)
country year m_tertiary_enrollment_rates_combined_wb - m_tertiary_enrollment_rates_combined_wb +
Chile 2019 86.068123 85.028748
Lower-middle-income countries 1992 8.983160 8.612840
Montenegro 2005 17.220909 17.013590
Slovakia 2011 44.489342 45.312408
Ukraine 2020 NaN 78.205231
~ Column mf_adults__25_64_years__percentage_of_tertiary_education (new data)
+ + New values: 231 / 12737 (1.81%)
country year mf_adults__25_64_years__percentage_of_tertiary_education
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column mf_primary_enrollment_rates_combined_wb (changed metadata, new data)
- - description: |-
+ + description_from_producer: |-
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year mf_primary_enrollment_rates_combined_wb
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column mf_secondary_enrollment_rates_combined_wb (changed metadata, new data)
- - description: |-
+ + description_from_producer: |-
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year mf_secondary_enrollment_rates_combined_wb
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column mf_tertiary_enrollment_rates_combined_wb (changed metadata, new data, changed data)
- - description: |-
+ + description_from_producer: |-
- - - producer: UNESCO (via World Bank)
+ + - producer: UNESCO Institute for Statistics
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
+ + New values: 231 / 12737 (1.81%)
country year mf_tertiary_enrollment_rates_combined_wb
Middle-income countries 2022 40.68354
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 3.38303
United Kingdom 2022 NaN
~ Changed values: 4985 / 12737 (39.14%)
country year mf_tertiary_enrollment_rates_combined_wb - mf_tertiary_enrollment_rates_combined_wb +
Belgium 1993 44.851688 45.476151
Dominican Republic 2017 59.915588 59.948189
Paraguay 1986 8.766680 8.980240
Romania 2017 49.381870 50.510818
Switzerland 1995 31.338699 32.363091
~ Column mf_youth_and_adults__15_64_years__average_years_of_education (new data)
+ + New values: 231 / 12737 (1.81%)
country year mf_youth_and_adults__15_64_years__average_years_of_education
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column mf_youth_and_adults__15_64_years__percentage_of_no_education (new data)
+ + New values: 231 / 12737 (1.81%)
country year mf_youth_and_adults__15_64_years__percentage_of_no_education
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
= Dataset garden/emdat/2024-04-11/natural_disasters
= Table natural_disasters_decadal
~ Column gdp (changed metadata, changed data)
- - description: |-
+ + description_from_producer: |-
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
~ Changed values: 2527 / 42952 (5.88%)
country year type gdp - gdp +
Chile 2000 volcanic_activity 1.796634e+10 1.798946e+10
Iran 2020 extreme_weather 8.992829e+10 8.977423e+10
Portugal 2020 all_disasters 0.000000e+00 6.379916e+10
Romania 1990 flood 2.466673e+10 2.465151e+10
Russia 2020 all_disasters 5.929275e+11 1.019329e+12
~ Column insured_damages_per_gdp (changed metadata, changed data)
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
~ Changed values: 469 / 42952 (1.09%)
country year type insured_damages_per_gdp - insured_damages_per_gdp +
Australia 2010 extreme_weather 0.063099 0.063071
Low-income countries 2010 extreme_weather 0.003131 0.003369
Lower-middle-income countries 1990 all_disasters 0.006631 0.006910
Mozambique 2010 all_disasters_excluding_extreme_temperature 0.097466 0.096695
Puerto Rico 2020 earthquake 0.077655 0.077571
~ Column reconstruction_costs_per_gdp (changed metadata, changed data)
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
~ Changed values: 148 / 42952 (0.34%)
country year type reconstruction_costs_per_gdp - reconstruction_costs_per_gdp +
Belarus 1990 extreme_weather 0.106169 0.106153
Croatia 2020 all_disasters_excluding_extreme_temperature 0.498831 0.372258
European Union (27) 1990 all_disasters_excluding_earthquakes 0.003091 0.003091
European Union (27) 2020 earthquake 0.001399 0.001398
Upper-middle-income countries 2000 flood 0.000587 0.000561
~ Column total_damages_per_gdp (changed metadata, changed data)
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
~ Changed values: 1540 / 42952 (3.59%)
country year type total_damages_per_gdp - total_damages_per_gdp +
Mexico 2000 flood 0.028544 0.027258
New Zealand 2020 wildfire 0.000945 0.000941
Nicaragua 1970 extreme_weather 0.004597 0.000000
World 1980 wildfire 0.002877 0.002842
World 2010 all_disasters_excluding_earthquakes 0.165644 0.165441
= Table natural_disasters_decadal_deaths
= Table natural_disasters_yearly
~ Column gdp (changed metadata, changed data)
- - description: |-
+ + description_from_producer: |-
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
~ Changed values: 9640 / 37481 (25.72%)
country year type gdp - gdp +
Botswana 1965 all_disasters_excluding_extreme_temperature 4.579087e+07 4.578870e+07
European Union (27) 1980 extreme_weather 3.303153e+12 3.302960e+12
Italy 2020 extreme_weather 1.896755e+12 1.897462e+12
Japan 1993 earthquake 4.454144e+12 4.536940e+12
Peru 2015 all_disasters_excluding_earthquakes 1.898053e+11 1.898030e+11
~ Column insured_damages_per_gdp (changed metadata, changed data)
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
~ Changed values: 2493 / 37481 (6.65%)
country year type insured_damages_per_gdp - insured_damages_per_gdp +
European Union (27) 2020 all_disasters_excluding_earthquakes 0.000547 0.000546
India 2020 all_disasters_excluding_earthquakes 0.029989 0.029945
Italy 2022 glacial_lake_outburst_flood NaN 0.000000
Upper-middle-income countries 1998 all_disasters 0.018726 0.018136
World 2011 flood 0.017754 0.017728
~ Column reconstruction_costs_per_gdp (changed metadata, changed data)
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
~ Changed values: 1278 / 37481 (3.41%)
country year type reconstruction_costs_per_gdp - reconstruction_costs_per_gdp +
Argentina 1975 extreme_weather 0.0 NaN
Brazil 1970 drought 0.0 NaN
Haiti 2022 all_disasters_excluding_extreme_temperature NaN 0.0
Low-income countries 1979 flood 0.0 NaN
Nicaragua 1979 all_disasters_excluding_extreme_temperature 0.0 NaN
~ Column total_damages_per_gdp (changed metadata, changed data)
- - attribution: Multiple sources compiled by World Bank (2023)
? ^
+ + attribution: Multiple sources compiled by World Bank (2024)
? ^
- - date_accessed: '2023-05-29'
? ^ ^
+ + date_accessed: '2024-05-20'
? ^ ^
- - date_published: '2023-05-11'
? ^ ^^
+ + date_published: '2024-05-20'
? ^ ^^
- - name: CC BY 4.0
+ + name: Creative Commons Attribution 4.0
~ Changed values: 5873 / 37481 (15.67%)
country year type total_damages_per_gdp - total_damages_per_gdp +
Lower-middle-income countries 1988 flood 0.332519 0.361345
Saudi Arabia 2015 all_disasters 0.000306 0.000299
Upper-middle-income countries 1972 extreme_weather 0.004243 0.004078
World 1997 earthquake 0.015677 0.015593
World 2018 all_disasters_excluding_earthquakes 0.146659 0.146513
= Table natural_disasters_yearly_impact
= Table natural_disasters_yearly_deaths
= Table natural_disasters_decadal_impact
2024-05-31 08:10:56 [error ] Traceback (most recent call last):
File "/home/owid/etl/etl/datadiff.py", line 421, in cli
lines = future.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/owid/etl/etl/datadiff.py", line 414, in func
differ.summary()
File "/home/owid/etl/etl/datadiff.py", line 252, in summary
self._diff_tables(self.ds_a, self.ds_b, table_name)
File "/home/owid/etl/etl/datadiff.py", line 237, in _diff_tables
out = _data_diff(
File "/home/owid/etl/etl/datadiff.py", line 570, in _data_diff
both = samp_a.merge(samp_b, on=dims, suffixes=(" -", " +"))
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 617, in merge
return merge(left=self, right=right, *args, **kwargs)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 1193, in merge
tb.metadata = combine_tables_metadata(tables=[left, right], short_name=short_name)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 1708, in combine_tables_metadata
dataset = combine_tables_datasetmeta(tables=tables)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 1702, in combine_tables_datasetmeta
return _get_metadata_value_from_tables_if_all_identical(tables=tables, field="dataset")
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 1682, in _get_metadata_value_from_tables_if_all_identical
unique_values = set(
TypeError: unhashable type: 'dict'
= Dataset garden/growth/2024-05-16/gdp_historical
= Table gdp_historical
~ Dim country
+ + New values: 1 / 48 (2.08%)
year country
2022 World
~ Dim year
+ + New values: 1 / 48 (2.08%)
country year
World 2022
~ Column gdp (changed metadata, new data, changed data)
- - description: |-
- - PPP GDP is gross domestic product converted to international dollars using purchasing power parity rates. An international dollar has the same purchasing power over GDP as the U.S. dollar has in the United States. GDP is the sum of gross value added by all resident producers in the country plus any product taxes and minus any subsidies not included in the value of the products. It is calculated without making deductions for depreciation of fabricated assets or for depletion and degradation of natural resources. Data are in constant 2017 international dollars.
- -
- - Statistical concept and methodology: For the concept and methodology of 2017 PPP, please refer to the International Comparison Program (ICP)’s website (https://www.worldbank.org/en/programs/icp).
+ + New values: 1 / 48 (2.08%)
country year gdp
World 2022 1.393578e+14
~ Changed values: 47 / 48 (97.92%)
country year gdp - gdp +
World 1700 7.500170e+11 7.515850e+11
World 1993 5.373394e+13 5.383247e+13
World 2003 7.435855e+13 7.455494e+13
World 2005 8.166562e+13 8.187848e+13
World 2008 9.265324e+13 9.289912e+13
~ Column gdp_per_capita (changed metadata, new data, changed data)
- - description: |-
- - GDP per capita based on purchasing power parity (PPP). PPP GDP is gross domestic product converted to international dollars using purchasing power parity rates. An international dollar has the same purchasing power over GDP as the U.S. dollar has in the United States. GDP at purchaser's prices is the sum of gross value added by all resident producers in the country plus any product taxes and minus any subsidies not included in the value of the products. It is calculated without making deductions for depreciation of fabricated assets or for depletion and degradation of natural resources. Data are in constant 2017 international dollars.
- -
- - Statistical concept and methodology: For the concept and methodology of 2017 PPP, please refer to the International Comparison Program (ICP)’s website (https://www.worldbank.org/en/programs/icp).
+ + New values: 1 / 48 (2.08%)
country year gdp_per_capita
World 2022 17527.189453
~ Changed values: 34 / 48 (70.83%)
country year gdp_per_capita - gdp_per_capita +
World 1990 9697.507812 9718.007812
World 1999 10749.005859 10776.479492
World 2005 12463.140625 12495.378906
World 2008 13622.655273 13658.740234
World 2013 14789.622070 14820.272461
= Dataset garden/health/2024-04-22/gpei_funding
= Table gpei_funding
~ Column domestic_resources (changed metadata)
- - description: |-
+ + description_short: |-
? ++++++
+ + Domestic resources for polio eradication, including government contributions and other domestic funding. This data is expressed in US dollars, adjusted for inflation.
+ + description_from_producer: |-
- - description_short: |-
- - Domestic resources for polio eradication, including government contributions and other domestic funding. This data is expressed in US dollars, adjusted for inflation.
+ + - producer: International Monetary Fund (via World Bank)
+ + title: World Development Indicators
+ + description: |-
+ + The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
+ + citation_full: World Bank's World Development Indicators (WDI).
+ + attribution: Multiple sources compiled by World Bank (2024)
+ + url_main: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
+ + url_download: http://databank.worldbank.org/data/download/WDI_csv.zip
+ + date_accessed: '2024-05-20'
+ + date_published: '2024-05-20'
+ + license:
+ + name: Creative Commons Attribution 4.0
- - sources:
- - - name: International Monetary Fund (via World Bank)
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - published_by: World Bank
~ Column g7_countries__and__european_commission (changed metadata)
- - description: |-
+ + description_short: |-
? ++++++
+ + Funding from G7 countries and the European Commission for polio eradication. This data is expressed in US dollars, adjusted for inflation.
+ + description_from_producer: |-
- - description_short: |-
- - Funding from G7 countries and the European Commission for polio eradication. This data is expressed in US dollars, adjusted for inflation.
+ + - producer: International Monetary Fund (via World Bank)
+ + title: World Development Indicators
+ + description: |-
+ + The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
+ + citation_full: World Bank's World Development Indicators (WDI).
+ + attribution: Multiple sources compiled by World Bank (2024)
+ + url_main: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
+ + url_download: http://databank.worldbank.org/data/download/WDI_csv.zip
+ + date_accessed: '2024-05-20'
+ + date_published: '2024-05-20'
+ + license:
+ + name: Creative Commons Attribution 4.0
- - sources:
- - - name: International Monetary Fund (via World Bank)
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - published_by: World Bank
~ Column multilateral_sector (changed metadata)
- - description: |-
+ + description_short: |-
? ++++++
+ + Funding from multilateral organizations for polio eradication. This data is expressed in US dollars, adjusted for inflation.
+ + description_from_producer: |-
- - description_short: |-
- - Funding from multilateral organizations for polio eradication. This data is expressed in US dollars, adjusted for inflation.
+ + - producer: International Monetary Fund (via World Bank)
+ + title: World Development Indicators
+ + description: |-
+ + The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
+ + citation_full: World Bank's World Development Indicators (WDI).
+ + attribution: Multiple sources compiled by World Bank (2024)
+ + url_main: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
+ + url_download: http://databank.worldbank.org/data/download/WDI_csv.zip
+ + date_accessed: '2024-05-20'
+ + date_published: '2024-05-20'
+ + license:
+ + name: Creative Commons Attribution 4.0
- - sources:
- - - name: International Monetary Fund (via World Bank)
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - published_by: World Bank
~ Column non_g7_oecd_countries (changed metadata)
- - description: |-
+ + description_short: Funding from non-G7 OECD countries for polio eradication. This data is expressed in US dollars, adjusted
+ + for inflation.
+ + description_from_producer: |-
- - description_short: Funding from non-G7 OECD countries for polio eradication. This data is expressed in US dollars, adjusted
- - for inflation.
+ + - producer: International Monetary Fund (via World Bank)
+ + title: World Development Indicators
+ + description: |-
+ + The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
+ + citation_full: World Bank's World Development Indicators (WDI).
+ + attribution: Multiple sources compiled by World Bank (2024)
+ + url_main: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
+ + url_download: http://databank.worldbank.org/data/download/WDI_csv.zip
+ + date_accessed: '2024-05-20'
+ + date_published: '2024-05-20'
+ + license:
+ + name: Creative Commons Attribution 4.0
- - sources:
- - - name: International Monetary Fund (via World Bank)
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - published_by: World Bank
~ Column other_donor_countries (changed metadata)
- - description: |-
+ + description_short: Funding from other donor countries for polio eradication. This data is expressed in US dollars, adjusted
+ + for inflation.
+ + description_from_producer: |-
- - description_short: Funding from other donor countries for polio eradication. This data is expressed in US dollars, adjusted
- - for inflation.
+ + - producer: International Monetary Fund (via World Bank)
+ + title: World Development Indicators
+ + description: |-
+ + The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
+ + citation_full: World Bank's World Development Indicators (WDI).
+ + attribution: Multiple sources compiled by World Bank (2024)
+ + url_main: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
+ + url_download: http://databank.worldbank.org/data/download/WDI_csv.zip
+ + date_accessed: '2024-05-20'
+ + date_published: '2024-05-20'
+ + license:
+ + name: Creative Commons Attribution 4.0
- - sources:
- - - name: International Monetary Fund (via World Bank)
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - published_by: World Bank
~ Column private_sector__non_governmental_donors (changed metadata)
- - description: |-
+ + description_short: |-
? ++++++
+ + Funding from the private sector and non-governmental donors for polio eradication. This data is expressed in US dollars, adjusted for inflation.
+ + description_from_producer: |-
- - description_short: |-
- - Funding from the private sector and non-governmental donors for polio eradication. This data is expressed in US dollars, adjusted for inflation.
+ + - producer: International Monetary Fund (via World Bank)
+ + title: World Development Indicators
+ + description: |-
+ + The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
+ + citation_full: World Bank's World Development Indicators (WDI).
+ + attribution: Multiple sources compiled by World Bank (2024)
+ + url_main: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
+ + url_download: http://databank.worldbank.org/data/download/WDI_csv.zip
+ + date_accessed: '2024-05-20'
+ + date_published: '2024-05-20'
+ + license:
+ + name: Creative Commons Attribution 4.0
- - sources:
- - - name: International Monetary Fund (via World Bank)
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - published_by: World Bank
~ Column total (changed metadata)
- - description: |-
+ + description_short: Total funding for polio eradication. This data is expressed in US dollars, adjusted for inflation.
+ + description_from_producer: |-
- - description_short: Total funding for polio eradication. This data is expressed in US dollars, adjusted for inflation.
+ + - producer: International Monetary Fund (via World Bank)
+ + title: World Development Indicators
+ + description: |-
+ + The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
+ + citation_full: World Bank's World Development Indicators (WDI).
+ + attribution: Multiple sources compiled by World Bank (2024)
+ + url_main: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
+ + url_download: http://databank.worldbank.org/data/download/WDI_csv.zip
+ + date_accessed: '2024-05-20'
+ + date_published: '2024-05-20'
+ + license:
+ + name: Creative Commons Attribution 4.0
- - sources:
- - - name: International Monetary Fund (via World Bank)
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - published_by: World Bank
= Dataset garden/research_development/2024-05-20/patents_wdi_unwpp
= Table patents_articles
~ Dataset garden/worldbank_wdi/2024-05-20/wdi (new version)
- - title: World Development Indicators - World Bank (2023.05.11)
+ + title: World Development Indicators
- - sources:
- - - name: World Bank
- - description: |-
- - The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - date_accessed: '2023-05-29'
- - publication_date: '2023-05-11'
- - licenses:
- - - name: Creative Commons Attribution 4.0
- - url: https://www.worldbank.org/en/about/legal/terms-of-use-for-datasets
- - version: '2023-05-29'
? ^ ^
+ + version: '2024-05-20'
? ^ ^
= Table wdi
~ Dim country
+ + New values: 231 / 14570 (1.59%)
year country
2022 Middle-income countries
2022 Myanmar
2022 Romania
2022 Sint Maarten (Dutch part)
2022 United Kingdom
~ Dim year
+ + New values: 231 / 14570 (1.59%)
country year
Middle-income countries 2022
Myanmar 2022
Romania 2022
Sint Maarten (Dutch part) 2022
United Kingdom 2022
~ Column ag_agr_trac_no (changed metadata, new data)
- - description: |-
+ + description_from_producer: |-
+ + origins:
+ + - producer: Food and Agriculture Organization of the United Nations (via World Bank)
+ + title: World Development Indicators
+ + description: |-
+ + The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
+ + citation_full: World Bank's World Development Indicators (WDI).
+ + attribution: Multiple sources compiled by World Bank (2024)
+ + url_main: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
+ + url_download: http://databank.worldbank.org/data/download/WDI_csv.zip
+ + date_accessed: '2024-05-20'
+ + date_published: '2024-05-20'
+ + license:
+ + name: Creative Commons Attribution 4.0
- - sources:
- - - name: Food and Agriculture Organization of the United Nations (via World Bank)
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - published_by: World Bank
+ + New values: 231 / 14570 (1.59%)
country year ag_agr_trac_no
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Column ag_con_fert_pt_zs (changed metadata, new data, changed data)
- - description: |-
+ + description_from_producer: |-
+ + origins:
+ + - producer: Food and Agriculture Organization of the United Nations (via World Bank)
+ + title: World Development Indicators
+ + description: |-
+ + The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
+ + citation_full: World Bank's World Development Indicators (WDI).
+ + attribution: Multiple sources compiled by World Bank (2024)
+ + url_main: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
+ + url_download: http://databank.worldbank.org/data/download/WDI_csv.zip
+ + date_accessed: '2024-05-20'
+ + date_published: '2024-05-20'
+ + license:
+ + name: Creative Commons Attribution 4.0
- - sources:
- - - name: Food and Agriculture Organization of the United Nations (via World Bank)
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - published_by: World Bank
+ + New values: 231 / 14570 (1.59%)
country year ag_con_fert_pt_zs
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 2022 NaN
Sint Maarten (Dutch part) 2022 NaN
United Kingdom 2022 NaN
~ Changed values: 790 / 14570 (5.42%)
country year ag_con_fert_pt_zs - ag_con_fert_pt_zs +
Low-income countries 1980 127.720383 124.880226
Lower-middle-income countries 2002 105.411446 99.312263
Malaysia 2020 215.594070 146.695343
North Korea 2016 204.710602 240.969574
Upper-middle-income countries 1975 140.318863 143.183517
~ Column ag_con_fert_zs (changed metadata, new data, changed data)
- - description: |-
+ + description_from_producer: |-
+ + origins:
+ + - producer: Food and Agriculture Organization of the United Nations (via World Bank)
+ + title: World Development Indicators
+ + description: |-
+ + The World Development Indicators (WDI) is the primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
+ + citation_full: World Bank's World Development Indicators (WDI).
+ + attribution: Multiple sources compiled by World Bank (2024)
+ + url_main: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
+ + url_download: http://databank.worldbank.org/data/download/WDI_csv.zip
+ + date_accessed: '2024-05-20'
+ + date_published: '2024-05-20'
+ + license:
+ + name: Creative Commons Attribution 4.0
- - sources:
- - - name: Food and Agriculture Organization of the United Nations (via World Bank)
- - url: https://datacatalog.worldbank.org/search/dataset/0037712/World-Development-Indicators
- - source_data_url: http://databank.worldbank.org/data/download/WDI_csv.zip
- - published_by: World Bank
+ + New values: 231 / 14570 (1.59%)
country year ag_con_fert_zs
Middle-income countries 2022 NaN
Myanmar 2022 NaN
Romania 202
...diff too long, truncated...Automatically updated datasets matching weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile are not included Edited: 2024-05-31 08:30:28 UTC |
Marigold
force-pushed
the
wdi-sources-to-origins
branch
4 times, most recently
from
8cfd48e to
a85cbe6
Compare
pabloarosado
approved these changes
May 29, 2024
Marigold
force-pushed
the
wdi-update
branch
2 times, most recently
from
c513405 to
1f601a0
Compare
Marigold
force-pushed
the
wdi-sources-to-origins
branch
3 times, most recently
from
0b7f1c6 to
b1c9519
Compare
Marigold
force-pushed
the
wdi-sources-to-origins
branch
from
b1c9519 to
5ba8247
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Remove misc function
add_origins_to_wdiand update dependencygarden/worldbank_wdi/2023-05-29/wditogarden/worldbank_wdi/2024-05-20/wdiin all steps. I used ChatGPT-generatedwdi.sources.jsonto get a producer (previouslysource.name) for a source from WDI metadata. Initially, I wanted to avoid ChatGPT and just dump the full source name as producer, but it was too messy and longer than 255 characters. If we wanted to polish it even further, we could use ChatGPT to create a full origin from given source name (not sure if it's worth it though).Here is an example of an old chart and a new chart. It's almost identical except for indicator's description, which has been moved to
description_from_producer.