📊 Calculate median migration distance between borders (#3566)

* Start a new staging server for branch 'median-distance-borders'

* 📊 median distance calculation for borders

* ❇️ improve metadata

* ❇️ fix metadata

* ✨ add origin for natural earth data

* 🐛 fix distance calculation and add geopy as dependency

* 🐛 fix migration distance calculation and change nat earth data to existing snapshot

* ✨ fix distance matrix calculation and improve logging with tqdm

---------

Co-authored-by: Tuna Acisu <[email protected]>
Co-authored-by: Marigold <[email protected]>

dag/main.yml

@@ -850,6 +850,13 @@ steps:
   data-private://grapher/owid/latest/ig_countries:
     - data-private://garden/owid/latest/ig_countries
 
+# Migration distances
+  data://garden/migration/2024-11-19/migration_distance:
+    - data://garden/un/2024-07-16/migrant_stock
+    - data://garden/geography/2023-11-28/nat_earth_110
+  data://grapher/migration/2024-11-19/migration_distance:
+    - data://garden/migration/2024-11-19/migration_distance
+
   # Migration between regions, based on UN DESA flows
   data://garden/migration/2024-11-18/migration_between_regions:
     - data://garden/un/2024-07-16/migrant_stock
@@ -867,6 +874,7 @@ steps:
   data://grapher/imf/2024-11-25/world_economic_outlook:
     - data://garden/imf/2024-11-25/world_economic_outlook
 
+
 include:
   - dag/open_numbers.yml
   - dag/faostat.yml

etl/steps/data/garden/migration/2024-11-19/migration_distance.countries.json

@@ -0,0 +1,260 @@
+{
+  "Afghanistan": "Afghanistan",
+  "Albania": "Albania",
+  "Algeria": "Algeria",
+  "Angola": "Angola",
+  "Antarctica": "Antarctica",
+  "Argentina": "Argentina",
+  "Armenia": "Armenia",
+  "Australia": "Australia",
+  "Austria": "Austria",
+  "Azerbaijan": "Azerbaijan",
+  "Bahamas": "Bahamas",
+  "Bangladesh": "Bangladesh",
+  "Belarus": "Belarus",
+  "Belgium": "Belgium",
+  "Belize": "Belize",
+  "Benin": "Benin",
+  "Bhutan": "Bhutan",
+  "Bolivia": "Bolivia",
+  "Botswana": "Botswana",
+  "Brazil": "Brazil",
+  "Brunei": "Brunei",
+  "Bulgaria": "Bulgaria",
+  "Burkina Faso": "Burkina Faso",
+  "Burundi": "Burundi",
+  "Cambodia": "Cambodia",
+  "Cameroon": "Cameroon",
+  "Canada": "Canada",
+  "Chad": "Chad",
+  "Chile": "Chile",
+  "China": "China",
+  "Colombia": "Colombia",
+  "Congo": "Congo",
+  "Costa Rica": "Costa Rica",
+  "Croatia": "Croatia",
+  "Cuba": "Cuba",
+  "Cyprus": "Cyprus",
+  "Czechia": "Czechia",
+  "Denmark": "Denmark",
+  "Djibouti": "Djibouti",
+  "Ecuador": "Ecuador",
+  "Egypt": "Egypt",
+  "El Salvador": "El Salvador",
+  "Eritrea": "Eritrea",
+  "Estonia": "Estonia",
+  "Ethiopia": "Ethiopia",
+  "Fiji": "Fiji",
+  "Finland": "Finland",
+  "France": "France",
+  "Gabon": "Gabon",
+  "Gambia": "Gambia",
+  "Georgia": "Georgia",
+  "Germany": "Germany",
+  "Ghana": "Ghana",
+  "Greece": "Greece",
+  "Greenland": "Greenland",
+  "Guatemala": "Guatemala",
+  "Guinea": "Guinea",
+  "Guinea-Bissau": "Guinea-Bissau",
+  "Guyana": "Guyana",
+  "Haiti": "Haiti",
+  "Honduras": "Honduras",
+  "Hungary": "Hungary",
+  "Iceland": "Iceland",
+  "India": "India",
+  "Indonesia": "Indonesia",
+  "Iran": "Iran",
+  "Iraq": "Iraq",
+  "Ireland": "Ireland",
+  "Israel": "Israel",
+  "Italy": "Italy",
+  "Jamaica": "Jamaica",
+  "Japan": "Japan",
+  "Jordan": "Jordan",
+  "Kazakhstan": "Kazakhstan",
+  "Kenya": "Kenya",
+  "Kosovo": "Kosovo",
+  "Kuwait": "Kuwait",
+  "Kyrgyzstan": "Kyrgyzstan",
+  "Laos": "Laos",
+  "Latvia": "Latvia",
+  "Lebanon": "Lebanon",
+  "Lesotho": "Lesotho",
+  "Liberia": "Liberia",
+  "Libya": "Libya",
+  "Lithuania": "Lithuania",
+  "Luxembourg": "Luxembourg",
+  "Madagascar": "Madagascar",
+  "Malawi": "Malawi",
+  "Malaysia": "Malaysia",
+  "Mali": "Mali",
+  "Mauritania": "Mauritania",
+  "Mexico": "Mexico",
+  "Moldova": "Moldova",
+  "Mongolia": "Mongolia",
+  "Montenegro": "Montenegro",
+  "Morocco": "Morocco",
+  "Mozambique": "Mozambique",
+  "Myanmar": "Myanmar",
+  "Namibia": "Namibia",
+  "Nepal": "Nepal",
+  "Netherlands": "Netherlands",
+  "New Caledonia": "New Caledonia",
+  "New Zealand": "New Zealand",
+  "Nicaragua": "Nicaragua",
+  "Niger": "Niger",
+  "Nigeria": "Nigeria",
+  "North Korea": "North Korea",
+  "North Macedonia": "North Macedonia",
+  "Norway": "Norway",
+  "Oman": "Oman",
+  "Pakistan": "Pakistan",
+  "Palestine": "Palestine",
+  "Panama": "Panama",
+  "Papua New Guinea": "Papua New Guinea",
+  "Paraguay": "Paraguay",
+  "Peru": "Peru",
+  "Philippines": "Philippines",
+  "Poland": "Poland",
+  "Portugal": "Portugal",
+  "Puerto Rico": "Puerto Rico",
+  "Qatar": "Qatar",
+  "Romania": "Romania",
+  "Russia": "Russia",
+  "Rwanda": "Rwanda",
+  "Saudi Arabia": "Saudi Arabia",
+  "Senegal": "Senegal",
+  "Serbia": "Serbia",
+  "Sierra Leone": "Sierra Leone",
+  "Slovakia": "Slovakia",
+  "Slovenia": "Slovenia",
+  "Somalia": "Somalia",
+  "Somaliland": "Somaliland",
+  "South Africa": "South Africa",
+  "South Korea": "South Korea",
+  "Spain": "Spain",
+  "Sri Lanka": "Sri Lanka",
+  "Sudan": "Sudan",
+  "Suriname": "Suriname",
+  "Sweden": "Sweden",
+  "Switzerland": "Switzerland",
+  "Syria": "Syria",
+  "Taiwan": "Taiwan",
+  "Tajikistan": "Tajikistan",
+  "Tanzania": "Tanzania",
+  "Thailand": "Thailand",
+  "Togo": "Togo",
+  "Trinidad and Tobago": "Trinidad and Tobago",
+  "Tunisia": "Tunisia",
+  "Turkey": "Turkey",
+  "Turkmenistan": "Turkmenistan",
+  "Uganda": "Uganda",
+  "Ukraine": "Ukraine",
+  "United Arab Emirates": "United Arab Emirates",
+  "United Kingdom": "United Kingdom",
+  "Uruguay": "Uruguay",
+  "Uzbekistan": "Uzbekistan",
+  "Vanuatu": "Vanuatu",
+  "Venezuela": "Venezuela",
+  "Vietnam": "Vietnam",
+  "Yemen": "Yemen",
+  "Zambia": "Zambia",
+  "Zimbabwe": "Zimbabwe",
+  "Fr. S. Antarctic Lands": "French Southern Territories",
+  "American Samoa": "American Samoa",
+  "Andorra": "Andorra",
+  "Anguilla": "Anguilla",
+  "Antigua and Barbuda": "Antigua and Barbuda",
+  "Aruba": "Aruba",
+  "Bahrain": "Bahrain",
+  "Barbados": "Barbados",
+  "Bermuda": "Bermuda",
+  "British Indian Ocean Territory": "British Indian Ocean Territory",
+  "British Virgin Islands": "British Virgin Islands",
+  "Cayman Islands": "Cayman Islands",
+  "Comoros": "Comoros",
+  "Cook Islands": "Cook Islands",
+  "Dominica": "Dominica",
+  "Faroe Islands": "Faroe Islands",
+  "French Polynesia": "French Polynesia",
+  "Gibraltar": "Gibraltar",
+  "Grenada": "Grenada",
+  "Guam": "Guam",
+  "Guernsey": "Guernsey",
+  "Isle of Man": "Isle of Man",
+  "Jersey": "Jersey",
+  "Kiribati": "Kiribati",
+  "Liechtenstein": "Liechtenstein",
+  "Maldives": "Maldives",
+  "Malta": "Malta",
+  "Marshall Islands": "Marshall Islands",
+  "Mauritius": "Mauritius",
+  "Monaco": "Monaco",
+  "Nauru": "Nauru",
+  "Niue": "Niue",
+  "Northern Mariana Islands": "Northern Mariana Islands",
+  "Palau": "Palau",
+  "Saint Barthelemy": "Saint Barthelemy",
+  "Saint Helena": "Saint Helena",
+  "Saint Kitts and Nevis": "Saint Kitts and Nevis",
+  "Saint Lucia": "Saint Lucia",
+  "Saint Pierre and Miquelon": "Saint Pierre and Miquelon",
+  "Saint Vincent and the Grenadines": "Saint Vincent and the Grenadines",
+  "Samoa": "Samoa",
+  "San Marino": "San Marino",
+  "Sao Tome and Principe": "Sao Tome and Principe",
+  "Seychelles": "Seychelles",
+  "Singapore": "Singapore",
+  "Tonga": "Tonga",
+  "Turks and Caicos Islands": "Turks and Caicos Islands",
+  "Tuvalu": "Tuvalu",
+  "United States Virgin Islands": "United States Virgin Islands",
+  "Vatican": "Vatican",
+  "Wallis and Futuna": "Wallis and Futuna",
+  "Aland Islands": "Aland Islands",
+  "Bosnia and Herzegovina": "Bosnia and Herzegovina",
+  "Cape Verde": "Cape Verde",
+  "Central African Republic": "Central African Republic",
+  "Cote d'Ivoire": "Cote d'Ivoire",
+  "Curacao": "Curacao",
+  "Democratic Republic of Congo": "Democratic Republic of Congo",
+  "Dominican Republic": "Dominican Republic",
+  "East Timor": "East Timor",
+  "Equatorial Guinea": "Equatorial Guinea",
+  "Eswatini": "Eswatini",
+  "Falkland Islands": "Falkland Islands",
+  "Heard Island and McDonald Islands": "Heard Island and McDonald Islands",
+  "Hong Kong": "Hong Kong",
+  "Macao": "Macao",
+  "Micronesia (country)": "Micronesia (country)",
+  "Montserrat": "Montserrat",
+  "Norfolk Island": "Norfolk Island",
+  "Northern Cyprus": "Northern Cyprus",
+  "Pitcairn": "Pitcairn",
+  "Saint Martin (French part)": "Saint Martin (French part)",
+  "Sint Maarten (Dutch part)": "Sint Maarten (Dutch part)",
+  "Solomon Islands": "Solomon Islands",
+  "South Sudan": "South Sudan",
+  "United States": "United States",
+  "Western Sahara": "Western Sahara",
+  "USNB Guantanamo Bay": "Guantanamo Bay",
+  "Akrotiri": "Akrotiri",
+  "Serranilla Bank": "Serranilla Bank",
+  "S. Geo. and the Is.": "South Georgia and the South Sandwich Islands",
+  "Bir Tawil": "Bir Tawil",
+  "Scarborough Reef": "Scarborough Reef",
+  "Spratly Is.": "Spratly Islands",
+  "Coral Sea Is.": "Coral Sea Islands",
+  "U.S. Minor Outlying Is.": "U.S. Minor Outlying Islands",
+  "Siachen Glacier": "Siachen Glacier",
+  "Indian Ocean Ter.": "Indian Ocean Territories",
+  "Southern Patagonian Ice Field": "Southern Patagonian Ice Field",
+  "Cyprus U.N. Buffer Zone": "Cyprus U.N. Buffer Zone",
+  "Dhekelia": "Dhekelia",
+  "Clipperton I.": "Clipperton Island",
+  "Ashmore and Cartier Is.": "Ashmore and Cartier Islands",
+  "Bajo Nuevo Bank": "Bajo Nuevo Bank",
+  "Baikonur": "Baikonur",
+  "Brazilian I.": "Brazilian Island"
+}

etl/steps/data/garden/migration/2024-11-19/migration_distance.meta.yml

@@ -0,0 +1,37 @@
+# NOTE: To learn more about the fields, hover over their names.
+definitions:
+  common:
+    presentation:
+      topic_tags:
+        - Migration
+    processing_level: major
+    description_processing: >-
+      - The migration distance indicator measures the median distance between the borders of the origin and destination country of migrants. It is calculated by first taking the minimum distance between the borders of origin and destination countries for each migrant, and then taking the median of these minimum distances.
+      - The indicator is based on the UN DESA International Migrant Stock data, which provides the number of international migrants by country of origin and destination. The geospatial data for the borders of countries is retrieved from the Natural Earth project (https://www.naturalearthdata.com/downloads/110m-cultural-vectors/), specifically the "Admin 0 - countries and tiny country points" datasets.
+
+
+dataset:
+  update_period_days: 365
+  title: Median migration distance (between borders)
+  description: |-
+    This data set includes the median distance between the borders of the origin and destination country of migrants. It is calculated by first taking the minimum distance between the borders of origin and destination countries for each migrant, and then calculating the median of these minimum distances.
+
+    The number of international migrants and their origin and destination countries are based on the UN DESA International Migrant Stock data. The geospatial data for the borders of countries is retrieved from the Natural Earth project (https://www.naturalearthdata.com/downloads/110m-cultural-vectors/), specifically the admin 0 - countries and tiny country points datasets.
+
+
+
+tables:
+  migration_distance:
+    variables:
+      median_distance:
+        title: Median migration distance for emigrants leaving this country
+        description_short: |-
+          The median distance between the borders of the origin and destination country of migrants. It is calculated by first taking the minimum distance between the borders of origin and destination countries for each migrant, and then calculating the median of these minimum distances.
+        unit: kilometers
+        short_unit: km
+      total_emigrants:
+        title: Total number of migrants leaving this country
+        description_short: The total number of international migrants leaving this country.
+        unit: persons
+        short_unit: ""
+