Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📊 minerals: Add data for missing minerals #3668

Merged
merged 17 commits into from
Dec 3, 2024

Conversation

pabloarosado
Copy link
Contributor

@pabloarosado pabloarosado commented Dec 2, 2024

  • Fix an issue when extracting historical USGS data. This lets us get missing historical data for certain minerals, e.g. lithium and rare earths production and unit values.
  • Create a dataset on global mine production by mineral, with a draft chart (but we may not use it in the end, since tonnes of production represent different things for different minerals, as discussed on Slack).
  • Archive an unnecessary fasttrack dataset.

@pabloarosado pabloarosado self-assigned this Dec 2, 2024
@pabloarosado pabloarosado changed the title 📊 minerals: Add minerals unit value 📊 minerals: Add unit value for missing minerals Dec 2, 2024
@owidbot
Copy link
Contributor

owidbot commented Dec 2, 2024

Quick links (staging server):

Site Dev Site Preview Admin Wizard Docs

Login: ssh owid@staging-site-add-minerals-unit-value

chart-diff: ✅
  • 2/2 reviewed charts
  • Modified: 1/1
  • New: 1/1
  • Rejected: 0
data-diff: ❌ Found differences
= Dataset garden/demography/2023-06-27/world_population_comparison
  = Table world_population_comparison
= Dataset garden/health/latest/global_health_mpox
  = Table global_health_mpox
= Dataset garden/minerals/2024-07-15/minerals
  = Table minerals
    ~ Column production_gemstones_mine_tonnes (changed metadata, changed data)
+       +   Notes found in original USGS historical data:
+       +   - Note on global production: World production data prior to 1944 and for 1949–54 are reported in total gemstone-quality diamonds, plus industrial diamonds. World production data for 1944–48 and 1955 to the most recent year represent the total quantity of natural gemstones quality diamonds. Data are from the MR and the MYB.
+       + 
+       +   - producer: United States Geological Survey
+       +     title: Historical Statistics for Mineral and Material Commodities
+       +     description: |-
+       +       Information on mineral commodities, including production and unit value (the real and nominal price in U.S. dollars of a metric ton of apparent consumption). For many of the commodities, data are reported as far back as 1900.
+       +       The data was compiled primarily from publications of the United States Geological Survey (USGS) and the U.S. Bureau of Mines (USBM). The principal references are the annual Minerals Yearbook (MYB) publication and its predecessor, Mineral Resources of the United States (MR). Other USGS and USBM publications used as references included: Statistical Compendium (SC), Metal Prices in the United States Through 1991 (MP91), Metal Prices in the United States Through 1998 (MP98), Metal Prices in the United States Through 2010, MCS and its predecessor Commodity Data Summaries (CDS), Minerals Facts and Problems (MFP), and various USBM Information Circulars. In some cases, USGS mineral commodity specialists added previously unpublished or recently revised data to the historical statistics series.
+       +     citation_full: |-
+       +       Kelly, T.D., and Matos, G.R., comps., 2014, Historical statistics for mineral and material commodities in the United States (2016 version): U.S. Geological Survey Data Series 140.
+       +     attribution: USGS - Historical Statistics for Mineral and Material Commodities (2023)
+       +     attribution_short: USGS
+       +     url_main: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +     date_accessed: '2024-07-15'
+       +     date_published: '2023'
+       +     license:
+       +       name: Public domain
+       +       url: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +   grapher_config:
+       +     note: Values are reported as tonnes of gemstone-quality diamonds.

        ~ Changed values: 70 / 8380 (0.84%)
          country  year  production_gemstones_mine_tonnes -  production_gemstones_mine_tonnes +
            World  1972                                <NA>                                2.54
            World  1976                                <NA>                                1.86
            World  1977                                <NA>                                1.86
            World  1980                                <NA>                                2.09
            World  1998                                <NA>                                12.1
    ~ Column production_iodine_mine_tonnes (changed metadata, changed data)
+       +   Notes found in original USGS historical data:
+       +   - Note on United States production: Production is given for three time periods (1932–37, 1969–83, and 1988–2005). Data for the first and last time periods were from the salient statistics table and(or) a production table in the MYB. Data from 1969–83 were from the MFP. Production data for other time periods were withheld because the data were proprietary.
+       +   - Note on global production: World production data for 1960–75, are reported in the world mine production table of the CDS. Data for 1976 to the most recent year are reported in the world production table of the MYB. Excludes production in the United States for 2006 to the most recent year.
+       + 
+       +   - producer: United States Geological Survey
+       +     title: Historical Statistics for Mineral and Material Commodities
+       +     description: |-
+       +       Information on mineral commodities, including production and unit value (the real and nominal price in U.S. dollars of a metric ton of apparent consumption). For many of the commodities, data are reported as far back as 1900.
+       +       The data was compiled primarily from publications of the United States Geological Survey (USGS) and the U.S. Bureau of Mines (USBM). The principal references are the annual Minerals Yearbook (MYB) publication and its predecessor, Mineral Resources of the United States (MR). Other USGS and USBM publications used as references included: Statistical Compendium (SC), Metal Prices in the United States Through 1991 (MP91), Metal Prices in the United States Through 1998 (MP98), Metal Prices in the United States Through 2010, MCS and its predecessor Commodity Data Summaries (CDS), Minerals Facts and Problems (MFP), and various USBM Information Circulars. In some cases, USGS mineral commodity specialists added previously unpublished or recently revised data to the historical statistics series.
+       +     citation_full: |-
+       +       Kelly, T.D., and Matos, G.R., comps., 2014, Historical statistics for mineral and material commodities in the United States (2016 version): U.S. Geological Survey Data Series 140.
+       +     attribution: USGS - Historical Statistics for Mineral and Material Commodities (2023)
+       +     attribution_short: USGS
+       +     url_main: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +     date_accessed: '2024-07-15'
+       +     date_published: '2023'
+       +     license:
+       +       name: Public domain
+       +       url: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities

        ~ Changed values: 108 / 8380 (1.29%)
                country  year  production_iodine_mine_tonnes -  production_iodine_mine_tonnes +
                  Chile  1979                      2409.600098                             <NA>
              Indonesia  1979                        25.287001                             <NA>
          United States  1936                             <NA>                            106.0
                  World  1987                             <NA>                          12700.0
                  World  1996                             <NA>                          14100.0
    ~ Column production_lithium_mine_tonnes (changed metadata, changed data)
+       +   Notes found in original USGS historical data:
+       +   - Note on United States production: Production data for lithium refers to lithium contained in material produced or shipped from mines and brine operations in the United States. Production data for 1940–54 include both gross tons of lithium minerals and compound production and Li2O content of these products. Li2O contains 46.46 percent lithium; this information was used to determine lithium content for those years. Because production data for 1940–54 included dilithium sodium phosphate, the average lithium content of domestic production varied from 2.50 percent to 5.20 percent for the period. Most lithium ores average about 2.00 percent and, and dilithium sodium phosphate contains about 10.5 percent lithium.
+       +   - Note on global production: World production data are in metric tons of gross product of lithium minerals and brine. Since 1967, lithium production was reported as ore and ore concentrate from mines and lithium carbonate from brine deposits. Since 2000, lithium production was reported as ore concentrate from mines and lithium carbonate from brine deposits, as lithium content, and as lithium carbonate equivalent. World production data for 1966–67 do not include data from Rhodesia (Zimbabwe) and some other African countries. Zimbabwe was by far the largest producer at the time. After 1954, world production does not include U.S. production. Data were not available for 1900–24.
+       + 
+       +   - producer: United States Geological Survey
+       +     title: Historical Statistics for Mineral and Material Commodities
+       +     description: |-
+       +       Information on mineral commodities, including production and unit value (the real and nominal price in U.S. dollars of a metric ton of apparent consumption). For many of the commodities, data are reported as far back as 1900.
+       +       The data was compiled primarily from publications of the United States Geological Survey (USGS) and the U.S. Bureau of Mines (USBM). The principal references are the annual Minerals Yearbook (MYB) publication and its predecessor, Mineral Resources of the United States (MR). Other USGS and USBM publications used as references included: Statistical Compendium (SC), Metal Prices in the United States Through 1991 (MP91), Metal Prices in the United States Through 1998 (MP98), Metal Prices in the United States Through 2010, MCS and its predecessor Commodity Data Summaries (CDS), Minerals Facts and Problems (MFP), and various USBM Information Circulars. In some cases, USGS mineral commodity specialists added previously unpublished or recently revised data to the historical statistics series.
+       +     citation_full: |-
+       +       Kelly, T.D., and Matos, G.R., comps., 2014, Historical statistics for mineral and material commodities in the United States (2016 version): U.S. Geological Survey Data Series 140.
+       +     attribution: USGS - Historical Statistics for Mineral and Material Commodities (2023)
+       +     attribution_short: USGS
+       +     url_main: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +     date_accessed: '2024-07-15'
+       +     date_published: '2023'
+       +     license:
+       +       name: Public domain
+       +       url: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +   grapher_config:
+       +     note: Values are reported as tonnes of lithium content.

        ~ Changed values: 20 / 8380 (0.24%)
          country  year  production_lithium_mine_tonnes -  production_lithium_mine_tonnes +
            World  2001                              <NA>                             13100
            World  2008                              <NA>                             24700
            World  2010                              <NA>                             24400
            World  2018                              <NA>                             91800
            World  2019                              <NA>                             83800
    ~ Column production_potash_mine_tonnes (changed metadata, changed data)
+       +   Notes found in original USGS historical data:
+       +   - Note on United States production: Potash production data are reported domestic sales data. All data are reported in terms of their K2O equivalents. Except for 1900 and 1905, no sales data are available prior to 1915. Data are reported in the MR and the MYB.
+       +   - Note on global production: Potash world mine production data are reported in terms of K2O equivalents. Potash world production data are not available prior to 1919. Data are reported in the MR and the MYB.
+       + 
+       +   - producer: United States Geological Survey
+       +     title: Historical Statistics for Mineral and Material Commodities
+       +     description: |-
+       +       Information on mineral commodities, including production and unit value (the real and nominal price in U.S. dollars of a metric ton of apparent consumption). For many of the commodities, data are reported as far back as 1900.
+       +       The data was compiled primarily from publications of the United States Geological Survey (USGS) and the U.S. Bureau of Mines (USBM). The principal references are the annual Minerals Yearbook (MYB) publication and its predecessor, Mineral Resources of the United States (MR). Other USGS and USBM publications used as references included: Statistical Compendium (SC), Metal Prices in the United States Through 1991 (MP91), Metal Prices in the United States Through 1998 (MP98), Metal Prices in the United States Through 2010, MCS and its predecessor Commodity Data Summaries (CDS), Minerals Facts and Problems (MFP), and various USBM Information Circulars. In some cases, USGS mineral commodity specialists added previously unpublished or recently revised data to the historical statistics series.
+       +     citation_full: |-
+       +       Kelly, T.D., and Matos, G.R., comps., 2014, Historical statistics for mineral and material commodities in the United States (2016 version): U.S. Geological Survey Data Series 140.
+       +     attribution: USGS - Historical Statistics for Mineral and Material Commodities (2023)
+       +     attribution_short: USGS
+       +     url_main: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +     date_accessed: '2024-07-15'
+       +     date_published: '2023'
+       +     license:
+       +       name: Public domain
+       +       url: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities

        ~ Changed values: 208 / 8380 (2.48%)
                country  year  production_potash_mine_tonnes -  production_potash_mine_tonnes +
          United States  1918                             <NA>                            35000
          United States  1925                             <NA>                            23400
          United States  1946                             <NA>                           842000
          United States  1993                             <NA>                          1480000
                  World  1999                             <NA>                         27200000
    ~ Column production_rare_earths_mine_tonnes (changed metadata, changed data)
+       +   Notes found in original USGS historical data:
+       +   - Note on United States production: Production data were for the amount of contained rare-earth oxides (REO) equivalent content in bastnäsite and monazite mineral concentrates produced in the United States. Data were not available for 1911–14, 1918–24, 1926–47, and 1949 and withheld for 1956 and 1962 in order to avoid disclosing proprietary data.
+       +   - Note on global production: World production data were for REO equivalent content of ores produced.
+       + 
+       +   - producer: United States Geological Survey
+       +     title: Historical Statistics for Mineral and Material Commodities
+       +     description: |-
+       +       Information on mineral commodities, including production and unit value (the real and nominal price in U.S. dollars of a metric ton of apparent consumption). For many of the commodities, data are reported as far back as 1900.
+       +       The data was compiled primarily from publications of the United States Geological Survey (USGS) and the U.S. Bureau of Mines (USBM). The principal references are the annual Minerals Yearbook (MYB) publication and its predecessor, Mineral Resources of the United States (MR). Other USGS and USBM publications used as references included: Statistical Compendium (SC), Metal Prices in the United States Through 1991 (MP91), Metal Prices in the United States Through 1998 (MP98), Metal Prices in the United States Through 2010, MCS and its predecessor Commodity Data Summaries (CDS), Minerals Facts and Problems (MFP), and various USBM Information Circulars. In some cases, USGS mineral commodity specialists added previously unpublished or recently revised data to the historical statistics series.
+       +     citation_full: |-
+       +       Kelly, T.D., and Matos, G.R., comps., 2014, Historical statistics for mineral and material commodities in the United States (2016 version): U.S. Geological Survey Data Series 140.
+       +     attribution: USGS - Historical Statistics for Mineral and Material Commodities (2023)
+       +     attribution_short: USGS
+       +     url_main: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +     date_accessed: '2024-07-15'
+       +     date_published: '2023'
+       +     license:
+       +       name: Public domain
+       +       url: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities

        ~ Changed values: 204 / 8380 (2.43%)
                country  year  production_rare_earths_mine_tonnes -  production_rare_earths_mine_tonnes +
          United States  1952                                  <NA>                                1110.0
          United States  1969                                  <NA>                               12500.0
          United States  1973                                  <NA>                               17500.0
          United States  1981                                  <NA>                               17100.0
                  World  1925                                  <NA>                                  12.0
    ~ Column share_of_global_production_gemstones_mine_tonnes (changed metadata, changed data)
+       +   Notes found in original USGS historical data:
+       +   - Note on global production: World production data prior to 1944 and for 1949–54 are reported in total gemstone-quality diamonds, plus industrial diamonds. World production data for 1944–48 and 1955 to the most recent year represent the total quantity of natural gemstones quality diamonds. Data are from the MR and the MYB.
+       + 
+       +   - producer: United States Geological Survey
+       +     title: Historical Statistics for Mineral and Material Commodities
+       +     description: |-
+       +       Information on mineral commodities, including production and unit value (the real and nominal price in U.S. dollars of a metric ton of apparent consumption). For many of the commodities, data are reported as far back as 1900.
+       +       The data was compiled primarily from publications of the United States Geological Survey (USGS) and the U.S. Bureau of Mines (USBM). The principal references are the annual Minerals Yearbook (MYB) publication and its predecessor, Mineral Resources of the United States (MR). Other USGS and USBM publications used as references included: Statistical Compendium (SC), Metal Prices in the United States Through 1991 (MP91), Metal Prices in the United States Through 1998 (MP98), Metal Prices in the United States Through 2010, MCS and its predecessor Commodity Data Summaries (CDS), Minerals Facts and Problems (MFP), and various USBM Information Circulars. In some cases, USGS mineral commodity specialists added previously unpublished or recently revised data to the historical statistics series.
+       +     citation_full: |-
+       +       Kelly, T.D., and Matos, G.R., comps., 2014, Historical statistics for mineral and material commodities in the United States (2016 version): U.S. Geological Survey Data Series 140.
+       +     attribution: USGS - Historical Statistics for Mineral and Material Commodities (2023)
+       +     attribution_short: USGS
+       +     url_main: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +     date_accessed: '2024-07-15'
+       +     date_published: '2023'
+       +     license:
+       +       name: Public domain
+       +       url: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +   grapher_config:
+       +     note: Values are reported as tonnes of gemstone-quality diamonds.

        ~ Changed values: 70 / 8380 (0.84%)
          country  year  share_of_global_production_gemstones_mine_tonnes -  share_of_global_production_gemstones_mine_tonnes +
            World  1972                                                <NA>                                               100.0
            World  1976                                                <NA>                                               100.0
            World  1977                                                <NA>                                               100.0
            World  1980                                                <NA>                                               100.0
            World  1998                                                <NA>                                               100.0
    ~ Column share_of_global_production_iodine_mine_tonnes (changed metadata, changed data)
+       +   Notes found in original USGS historical data:
+       +   - Note on United States production: Production is given for three time periods (1932–37, 1969–83, and 1988–2005). Data for the first and last time periods were from the salient statistics table and(or) a production table in the MYB. Data from 1969–83 were from the MFP. Production data for other time periods were withheld because the data were proprietary.
+       +   - Note on global production: World production data for 1960–75, are reported in the world mine production table of the CDS. Data for 1976 to the most recent year are reported in the world production table of the MYB. Excludes production in the United States for 2006 to the most recent year.
+       + 
+       +   - producer: United States Geological Survey
+       +     title: Historical Statistics for Mineral and Material Commodities
+       +     description: |-
+       +       Information on mineral commodities, including production and unit value (the real and nominal price in U.S. dollars of a metric ton of apparent consumption). For many of the commodities, data are reported as far back as 1900.
+       +       The data was compiled primarily from publications of the United States Geological Survey (USGS) and the U.S. Bureau of Mines (USBM). The principal references are the annual Minerals Yearbook (MYB) publication and its predecessor, Mineral Resources of the United States (MR). Other USGS and USBM publications used as references included: Statistical Compendium (SC), Metal Prices in the United States Through 1991 (MP91), Metal Prices in the United States Through 1998 (MP98), Metal Prices in the United States Through 2010, MCS and its predecessor Commodity Data Summaries (CDS), Minerals Facts and Problems (MFP), and various USBM Information Circulars. In some cases, USGS mineral commodity specialists added previously unpublished or recently revised data to the historical statistics series.
+       +     citation_full: |-
+       +       Kelly, T.D., and Matos, G.R., comps., 2014, Historical statistics for mineral and material commodities in the United States (2016 version): U.S. Geological Survey Data Series 140.
+       +     attribution: USGS - Historical Statistics for Mineral and Material Commodities (2023)
+       +     attribution_short: USGS
+       +     url_main: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +     date_accessed: '2024-07-15'
+       +     date_published: '2023'
+       +     license:
+       +       name: Public domain
+       +       url: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities

        ~ Changed values: 396 / 8380 (4.73%)
                country  year  share_of_global_production_iodine_mine_tonnes -  share_of_global_production_iodine_mine_tonnes +
                  Chile  2013                                             <NA>                                        67.064934
                  Chile  2019                                             <NA>                                        66.114288
          United States  2007                                             <NA>                                         4.562737
             Uzbekistan  2000                                             <NA>                                         0.010256
             Uzbekistan  2001                                             <NA>                                         0.009662
    ~ Column share_of_global_production_lithium_mine_tonnes (changed metadata, changed data)
+       +   Notes found in original USGS historical data:
+       +   - Note on United States production: Production data for lithium refers to lithium contained in material produced or shipped from mines and brine operations in the United States. Production data for 1940–54 include both gross tons of lithium minerals and compound production and Li2O content of these products. Li2O contains 46.46 percent lithium; this information was used to determine lithium content for those years. Because production data for 1940–54 included dilithium sodium phosphate, the average lithium content of domestic production varied from 2.50 percent to 5.20 percent for the period. Most lithium ores average about 2.00 percent and, and dilithium sodium phosphate contains about 10.5 percent lithium.
+       +   - Note on global production: World production data are in metric tons of gross product of lithium minerals and brine. Since 1967, lithium production was reported as ore and ore concentrate from mines and lithium carbonate from brine deposits. Since 2000, lithium production was reported as ore concentrate from mines and lithium carbonate from brine deposits, as lithium content, and as lithium carbonate equivalent. World production data for 1966–67 do not include data from Rhodesia (Zimbabwe) and some other African countries. Zimbabwe was by far the largest producer at the time. After 1954, world production does not include U.S. production. Data were not available for 1900–24.
+       + 
+       +   - producer: United States Geological Survey
+       +     title: Historical Statistics for Mineral and Material Commodities
+       +     description: |-
+       +       Information on mineral commodities, including production and unit value (the real and nominal price in U.S. dollars of a metric ton of apparent consumption). For many of the commodities, data are reported as far back as 1900.
+       +       The data was compiled primarily from publications of the United States Geological Survey (USGS) and the U.S. Bureau of Mines (USBM). The principal references are the annual Minerals Yearbook (MYB) publication and its predecessor, Mineral Resources of the United States (MR). Other USGS and USBM publications used as references included: Statistical Compendium (SC), Metal Prices in the United States Through 1991 (MP91), Metal Prices in the United States Through 1998 (MP98), Metal Prices in the United States Through 2010, MCS and its predecessor Commodity Data Summaries (CDS), Minerals Facts and Problems (MFP), and various USBM Information Circulars. In some cases, USGS mineral commodity specialists added previously unpublished or recently revised data to the historical statistics series.
+       +     citation_full: |-
+       +       Kelly, T.D., and Matos, G.R., comps., 2014, Historical statistics for mineral and material commodities in the United States (2016 version): U.S. Geological Survey Data Series 140.
+       +     attribution: USGS - Historical Statistics for Mineral and Material Commodities (2023)
+       +     attribution_short: USGS
+       +     url_main: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +     date_accessed: '2024-07-15'
+       +     date_published: '2023'
+       +     license:
+       +       name: Public domain
+       +       url: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +   grapher_config:
+       +     note: Values are reported as tonnes of lithium content.

        ~ Changed values: 20 / 8380 (0.24%)
          country  year  share_of_global_production_lithium_mine_tonnes -  share_of_global_production_lithium_mine_tonnes +
            World  2001                                              <NA>                                             100.0
            World  2008                                              <NA>                                             100.0
            World  2010                                              <NA>                                             100.0
            World  2018                                              <NA>                                             100.0
            World  2019                                              <NA>                                             100.0
    ~ Column share_of_global_production_potash_mine_tonnes (changed metadata, changed data)
+       +   Notes found in original USGS historical data:
+       +   - Note on United States production: Potash production data are reported domestic sales data. All data are reported in terms of their K2O equivalents. Except for 1900 and 1905, no sales data are available prior to 1915. Data are reported in the MR and the MYB.
+       +   - Note on global production: Potash world mine production data are reported in terms of K2O equivalents. Potash world production data are not available prior to 1919. Data are reported in the MR and the MYB.
+       + 
+       +   - producer: United States Geological Survey
+       +     title: Historical Statistics for Mineral and Material Commodities
+       +     description: |-
+       +       Information on mineral commodities, including production and unit value (the real and nominal price in U.S. dollars of a metric ton of apparent consumption). For many of the commodities, data are reported as far back as 1900.
+       +       The data was compiled primarily from publications of the United States Geological Survey (USGS) and the U.S. Bureau of Mines (USBM). The principal references are the annual Minerals Yearbook (MYB) publication and its predecessor, Mineral Resources of the United States (MR). Other USGS and USBM publications used as references included: Statistical Compendium (SC), Metal Prices in the United States Through 1991 (MP91), Metal Prices in the United States Through 1998 (MP98), Metal Prices in the United States Through 2010, MCS and its predecessor Commodity Data Summaries (CDS), Minerals Facts and Problems (MFP), and various USBM Information Circulars. In some cases, USGS mineral commodity specialists added previously unpublished or recently revised data to the historical statistics series.
+       +     citation_full: |-
+       +       Kelly, T.D., and Matos, G.R., comps., 2014, Historical statistics for mineral and material commodities in the United States (2016 version): U.S. Geological Survey Data Series 140.
+       +     attribution: USGS - Historical Statistics for Mineral and Material Commodities (2023)
+       +     attribution_short: USGS
+       +     url_main: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +     date_accessed: '2024-07-15'
+       +     date_published: '2023'
+       +     license:
+       +       name: Public domain
+       +       url: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities

        ~ Changed values: 202 / 8380 (2.41%)
                country  year  share_of_global_production_potash_mine_tonnes -  share_of_global_production_potash_mine_tonnes +
          United States  1937                                             <NA>                                          8.58156
          United States  2017                                             <NA>                                         1.140143
                  World  1925                                             <NA>                                            100.0
                  World  1989                                             <NA>                                            100.0
                  World  1996                                             <NA>                                            100.0
    ~ Column share_of_global_production_rare_earths_mine_tonnes (changed metadata, changed data)
+       +   Notes found in original USGS historical data:
+       +   - Note on United States production: Production data were for the amount of contained rare-earth oxides (REO) equivalent content in bastnäsite and monazite mineral concentrates produced in the United States. Data were not available for 1911–14, 1918–24, 1926–47, and 1949 and withheld for 1956 and 1962 in order to avoid disclosing proprietary data.
+       +   - Note on global production: World production data were for REO equivalent content of ores produced.
+       + 
+       +   - producer: United States Geological Survey
+       +     title: Historical Statistics for Mineral and Material Commodities
+       +     description: |-
+       +       Information on mineral commodities, including production and unit value (the real and nominal price in U.S. dollars of a metric ton of apparent consumption). For many of the commodities, data are reported as far back as 1900.
+       +       The data was compiled primarily from publications of the United States Geological Survey (USGS) and the U.S. Bureau of Mines (USBM). The principal references are the annual Minerals Yearbook (MYB) publication and its predecessor, Mineral Resources of the United States (MR). Other USGS and USBM publications used as references included: Statistical Compendium (SC), Metal Prices in the United States Through 1991 (MP91), Metal Prices in the United States Through 1998 (MP98), Metal Prices in the United States Through 2010, MCS and its predecessor Commodity Data Summaries (CDS), Minerals Facts and Problems (MFP), and various USBM Information Circulars. In some cases, USGS mineral commodity specialists added previously unpublished or recently revised data to the historical statistics series.
+       +     citation_full: |-
+       +       Kelly, T.D., and Matos, G.R., comps., 2014, Historical statistics for mineral and material commodities in the United States (2016 version): U.S. Geological Survey Data Series 140.
+       +     attribution: USGS - Historical Statistics for Mineral and Material Commodities (2023)
+       +     attribution_short: USGS
+       +     url_main: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities
+       +     date_accessed: '2024-07-15'
+       +     date_published: '2023'
+       +     license:
+       +       name: Public domain
+       +       url: https://www.usgs.gov/centers/national-minerals-information-center/historical-statistics-mineral-and-material-commodities

        ~ Changed values: 204 / 8380 (2.43%)
                country  year  share_of_global_production_rare_earths_mine_tonnes -  share_of_global_production_rare_earths_mine_tonnes +
          United States  1952                                                  <NA>                                              60.98901
          United States  1969                                                  <NA>                                             69.060776
          United States  1973                                                  <NA>                                             72.916664
          United States  1981                                                  <NA>                                             55.882355
                  World  1925                                                  <NA>                                                 100.0
    ~ Column unit_value_aluminum_smelter_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Aluminum
+       +   shortUnit: $
    ~ Column unit_value_antimony_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Antimony
+       +   shortUnit: $
    ~ Column unit_value_arsenic_processing_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Arsenic
+       +   shortUnit: $
    ~ Column unit_value_asbestos_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Asbestos
+       +   shortUnit: $
    ~ Column unit_value_barite_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Barite
+       +   shortUnit: $
    ~ Column unit_value_bauxite_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Bauxite
+       +   shortUnit: $
    ~ Column unit_value_beryllium_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Beryllium
+       +   shortUnit: $
    ~ Column unit_value_bismuth_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Bismuth
+       +   shortUnit: $
    ~ Column unit_value_boron_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Boron
+       +   shortUnit: $
    ~ Column unit_value_cadmium_refinery_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Cadmium
+       +   shortUnit: $
    ~ Column unit_value_cement_processing_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Cement
+       +   shortUnit: $
    ~ Column unit_value_cobalt_value_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Cobalt
+       +   shortUnit: $
    ~ Column unit_value_copper_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Copper
+       +   shortUnit: $
    ~ Column unit_value_diamond_mine_and_synthetic__industrial_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Diamond
+       +   shortUnit: $
    ~ Column unit_value_feldspar_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Feldspar
+       +   shortUnit: $
    ~ Column unit_value_fluorspar_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Fluorspar
+       +   shortUnit: $
    ~ Column unit_value_gallium_refinery_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Gallium
+       +   shortUnit: $
    ~ Column unit_value_garnet_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Garnet
+       +   shortUnit: $
+   + Column unit_value_gemstones_mine_constant_1998_usd_per_tonne
    ~ Column unit_value_germanium_refinery_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Germanium
+       +   shortUnit: $
    ~ Column unit_value_gold_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Gold
+       +   shortUnit: $
    ~ Column unit_value_graphite_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Graphite
+       +   shortUnit: $
    ~ Column unit_value_gypsum_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Gypsum
+       +   shortUnit: $
    ~ Column unit_value_helium_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Helium
+       +   shortUnit: $
+   + Column unit_value_iodine_mine_constant_1998_usd_per_tonne
    ~ Column unit_value_iron_ore_mine__crude_ore_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Iron ore
+       +   shortUnit: $
    ~ Column unit_value_iron_smelter__pig_iron_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Iron
+       +   shortUnit: $
    ~ Column unit_value_lead_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Lead
+       +   shortUnit: $
    ~ Column unit_value_lime_processing_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Lime
+       +   shortUnit: $
+   + Column unit_value_lithium_mine_constant_1998_usd_per_tonne
    ~ Column unit_value_magnesium_compounds_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Magnesium compounds
+       +   shortUnit: $
    ~ Column unit_value_magnesium_metal_smelter_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Magnesium metal
+       +   shortUnit: $
    ~ Column unit_value_manganese_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Manganese
+       +   shortUnit: $
    ~ Column unit_value_mercury_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Mercury
+       +   shortUnit: $
    ~ Column unit_value_mica_mine__scrap_and_flake_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Mica (scrap and flake)
+       +   shortUnit: $
    ~ Column unit_value_mica_mine__sheet_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Mica (sheet)
+       +   shortUnit: $
    ~ Column unit_value_molybdenum_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Molybdenum
+       +   shortUnit: $
    ~ Column unit_value_nickel_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Nickel
+       +   shortUnit: $
    ~ Column unit_value_niobium_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Niobium
+       +   shortUnit: $
    ~ Column unit_value_nitrogen_fixed_ammonia_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Nitrogen
+       +   shortUnit: $
    ~ Column unit_value_phosphate_rock_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Phosphate rock
+       +   shortUnit: $
+   + Column unit_value_potash_mine_constant_1998_usd_per_tonne
+   + Column unit_value_rare_earths_mine_constant_1998_usd_per_tonne
+   + Column unit_value_rhenium_mine_constant_1998_usd_per_tonne
    ~ Column unit_value_salt_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Salt
+       +   shortUnit: $
    ~ Column unit_value_sand_and_gravel_mine__construction_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Sand and gravel
+       +   shortUnit: $
    ~ Column unit_value_sand_and_gravel_mine__industrial_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Sand and gravel
+       +   shortUnit: $
    ~ Column unit_value_selenium_refinery_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Selenium
+       +   shortUnit: $
    ~ Column unit_value_silicon_processing_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Silicon
+       +   shortUnit: $
    ~ Column unit_value_silver_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Silver
+       +   shortUnit: $
    ~ Column unit_value_steel_processing__crude_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Steel
+       +   shortUnit: $
    ~ Column unit_value_strontium_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Strontium
+       +   shortUnit: $
    ~ Column unit_value_sulfur_processing_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Sulfur
+       +   shortUnit: $
    ~ Column unit_value_talc_and_pyrophyllite_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Talc and pyrophyllite
+       +   shortUnit: $
    ~ Column unit_value_tantalum_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Tantalum
+       +   shortUnit: $
    ~ Column unit_value_tin_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Tin
+       +   shortUnit: $
    ~ Column unit_value_tungsten_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Tungsten
+       +   shortUnit: $
    ~ Column unit_value_vanadium_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Vanadium
+       +   shortUnit: $
    ~ Column unit_value_zinc_mine_constant_1998_usd_per_tonne (changed metadata)
+       + display:
+       +   name: Zinc
+       +   shortUnit: $
= Dataset garden/unu_wider/2023-11-01/government_revenue_dataset
  = Table government_revenue_dataset
= Dataset garden/usgs/2024-07-15/historical_statistics_for_mineral_and_material_commodities
  = Table historical_statistics_for_mineral_and_material_commodities_flat
+   + Column production_gemstones_mine_tonnes
+   + Column production_iodine_mine_tonnes
+   + Column production_lithium_mine_tonnes
+   + Column production_potash_mine_tonnes
+   + Column production_rare_earths_mine_tonnes
+   + Column production_rhenium_mine_tonnes
+   + Column unit_value_gemstones_mine_constant_1998_usd_per_tonne
+   + Column unit_value_iodine_mine_constant_1998_usd_per_tonne
+   + Column unit_value_lithium_mine_constant_1998_usd_per_tonne
+   + Column unit_value_potash_mine_constant_1998_usd_per_tonne
+   + Column unit_value_rare_earths_mine_constant_1998_usd_per_tonne
+   + Column unit_value_rhenium_mine_constant_1998_usd_per_tonne
  = Table historical_unit_value
    ~ Dim country
+       + New values: 550 / 7065 (7.78%)
           year commodity sub_commodity                        unit country
           1938    Iodine          Mine constant 1998 US$ per tonne   World
           1941 Gemstones          Mine constant 1998 US$ per tonne   World
           1950    Potash          Mine constant 1998 US$ per tonne   World
           1958   Lithium          Mine constant 1998 US$ per tonne   World
           2004    Iodine          Mine constant 1998 US$ per tonne   World
    ~ Dim year
+       + New values: 550 / 7065 (7.78%)
          country commodity sub_commodity                        unit  year
            World    Iodine          Mine constant 1998 US$ per tonne  1938
            World Gemstones          Mine constant 1998 US$ per tonne  1941
            World    Potash          Mine constant 1998 US$ per tonne  1950
            World   Lithium          Mine constant 1998 US$ per tonne  1958
            World    Iodine          Mine constant 1998 US$ per tonne  2004
    ~ Dim commodity
+       + New values: 550 / 7065 (7.78%)
          country  year sub_commodity                        unit commodity
            World  1938          Mine constant 1998 US$ per tonne    Iodine
            World  1941          Mine constant 1998 US$ per tonne Gemstones
            World  1950          Mine constant 1998 US$ per tonne    Potash
            World  1958          Mine constant 1998 US$ per tonne   Lithium
            World  2004          Mine constant 1998 US$ per tonne    Iodine
    ~ Dim sub_commodity
+       + New values: 550 / 7065 (7.78%)
          country  year commodity                        unit sub_commodity
            World  1938    Iodine constant 1998 US$ per tonne          Mine
            World  1941 Gemstones constant 1998 US$ per tonne          Mine
            World  1950    Potash constant 1998 US$ per tonne          Mine
            World  1958   Lithium constant 1998 US$ per tonne          Mine
            World  2004    Iodine constant 1998 US$ per tonne          Mine
    ~ Dim unit
+       + New values: 550 / 7065 (7.78%)
          country  year commodity sub_commodity                        unit
            World  1938    Iodine          Mine constant 1998 US$ per tonne
            World  1941 Gemstones          Mine constant 1998 US$ per tonne
            World  1950    Potash          Mine constant 1998 US$ per tonne
            World  1958   Lithium          Mine constant 1998 US$ per tonne
            World  2004    Iodine          Mine constant 1998 US$ per tonne
    ~ Column notes_unit_value (new data)
+       + New values: 550 / 7065 (7.78%)
          country  year commodity sub_commodity                        unit                                                                                                                                                                   notes_unit_value
            World  1938    Iodine          Mine constant 1998 US$ per tonne  ['The Consumer Price Index conversion factor, with 1998 as the base year, is used to adjust unit value in current U.S. dollars to the unit value in constant 1998 U.S. dollars.']
            World  1941 Gemstones          Mine constant 1998 US$ per tonne ['The Consumer Price Index conversion factor, with 1998 as the base year, was used to adjust unit value in current U.S. dollars to the unit value in constant 1998 U.S. dollars.']
            World  1950    Potash          Mine constant 1998 US$ per tonne  ['The Consumer Price Index conversion factor, with 1998 as the base year, is used to adjust unit value in current U.S. dollars to the unit value in constant 1998 U.S. dollars.']
            World  1958   Lithium          Mine constant 1998 US$ per tonne ['The Consumer Price Index conversion factor, with 1998 as the base year, was used to adjust the unit value in current U.S. dollars to unit value in constant 1998 U.S. dollars.']
            World  2004    Iodine          Mine constant 1998 US$ per tonne  ['The Consumer Price Index conversion factor, with 1998 as the base year, is used to adjust unit value in current U.S. dollars to the unit value in constant 1998 U.S. dollars.']
    ~ Column unit_value (new data)
+       + New values: 550 / 7065 (7.78%)
          country  year commodity sub_commodity                        unit    unit_value
            World  1938    Iodine          Mine constant 1998 US$ per tonne       20700.0
            World  1941 Gemstones          Mine constant 1998 US$ per tonne  3920000000.0
            World  1950    Potash          Mine constant 1998 US$ per tonne         282.0
            World  1958   Lithium          Mine constant 1998 US$ per tonne        9100.0
            World  2004    Iodine          Mine constant 1998 US$ per tonne       11400.0
= Dataset garden/war/2023-09-21/mars
  = Table mars_country
  = Table mars
    ~ Column number_deaths_ongoing_conflicts_high (changed metadata)
-       -   <% set per_capita = "" %>
    ~ Column number_deaths_ongoing_conflicts_high_per_capita (changed metadata)
+       +   <% set per_capita = " per 100,000 people" %> <%- if conflict_type == "all" -%>
-       -   <% set per_capita = " per 100,000 people" %> <% set per_capita = "" %>
-       -   <%- if conflict_type == "all" -%>
    ~ Column number_deaths_ongoing_conflicts_low (changed metadata)
-       -   <% set per_capita = "" %>
    ~ Column number_deaths_ongoing_conflicts_low_per_capita (changed metadata)
+       +   <% set per_capita = " per 100,000 people" %> <%- if conflict_type == "all" -%>
-       -   <% set per_capita = " per 100,000 people" %> <% set per_capita = "" %>
-       -   <%- if conflict_type == "all" -%>
= Dataset garden/who/2024-09-09/flu_test
  = Table flu_test
    ~ Dim country
-       - Removed values: 17 / 72411 (0.02%)
                date    country
          2024-11-25    Bahrain
          2024-10-21 East Timor
          2024-11-04 East Timor
          2024-11-11 East Timor
          2024-11-18       Mali
    ~ Dim date
-       - Removed values: 17 / 72411 (0.02%)
             country       date
             Bahrain 2024-11-25
          East Timor 2024-10-21
          East Timor 2024-11-04
          East Timor 2024-11-11
                Mali 2024-11-18
    ~ Column denomcombined (changed data)
-       - Removed values: 17 / 72411 (0.02%)
             country       date  denomcombined
             Bahrain 2024-11-25            115
          East Timor 2024-10-21             60
          East Timor 2024-11-04             30
          East Timor 2024-11-11             49
                Mali 2024-11-18             49
        ~ Changed values: 19 / 72411 (0.03%)
                country       date  denomcombined -  denomcombined +
             Bangladesh 2024-11-11              381              217
                  Japan 2024-10-28               25               20
                  Japan 2024-11-11               12                5
          United States 2024-10-14            71705            71690
          United States 2024-11-11            76392            61182
    ~ Column pcnt_poscombined (changed data)
-       - Removed values: 17 / 72411 (0.02%)
             country       date  pcnt_poscombined
             Bahrain 2024-11-25         22.608696
          East Timor 2024-10-21          8.333333
          East Timor 2024-11-04              50.0
          East Timor 2024-11-11         51.020409
                Mali 2024-11-18          2.040816
        ~ Changed values: 19 / 72411 (0.03%)
                country       date  pcnt_poscombined -  pcnt_poscombined +
             Bangladesh 2024-11-11             6.56168            4.608295
                  Japan 2024-10-28                60.0                55.0
                  Japan 2024-11-11           91.666664                80.0
          United States 2024-10-14            1.023638            1.022458
          United States 2024-11-11             2.73458            2.538328
= Dataset garden/who/latest/monkeypox
  = Table monkeypox


Legend: +New  ~Modified  -Removed  =Identical  Details
Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet

Automatically updated datasets matching weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included

Edited: 2024-12-03 12:33:02 UTC
Execution time: 45.81 seconds

@pabloarosado pabloarosado changed the title 📊 minerals: Add unit value for missing minerals 📊 minerals: Add data for missing minerals Dec 3, 2024
@pabloarosado pabloarosado marked this pull request as ready for review December 3, 2024 12:34
@pabloarosado pabloarosado merged commit 5742c76 into master Dec 3, 2024
10 checks passed
@pabloarosado pabloarosado deleted the add-minerals-unit-value branch December 3, 2024 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants