Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ chart-diff improvements #2680

Merged
merged 1 commit into from
May 24, 2024
Merged

✨ chart-diff improvements #2680

merged 1 commit into from
May 24, 2024

Conversation

Marigold
Copy link
Collaborator

@Marigold Marigold commented May 22, 2024

Various improvements to chart-diff that I found to be useful when updating WDI dataset.

  • Add pagination to charts in chart-diff with 10 charts per page (WDI has 500+ charts...). We can try removing it when expander lazy loads its content (it's on their roadmap for August 2024)
  • Add option to use STAGING=1 env which assigns your current branch to STAGING. This lets you put STAGING=1 in your .env and always run ETL against the correct staging database
  • Move Refresh button next to Approve / Pending
  • Fix minor bugs in indicators upgrader encountered when updating WDI

@Marigold Marigold force-pushed the chart-diff-improvements branch 2 times, most recently from 20c0a10 to 0bbb164 Compare May 22, 2024 14:34
@owidbot
Copy link
Contributor

owidbot commented May 22, 2024

Quick links (staging server):

Site Admin Wizard

Login: ssh owid@staging-site-chart-diff-improvements

chart-diff: ✅ No new or modified charts.
data-diff: ❌ Found differences
= Dataset garden/emdat/2024-04-11/natural_disasters
  = Table natural_disasters_decadal
    ~ Column affected_per_100k_people (changed metadata)
-       - title: Number of affected persons per 100,000 people
        ?        ^^^^^^^^^^^           --------
+       + title: Affected per 100,000 people
        ?        ^
+       + description_short: |-
+       +   Rate of people per 100,000 requiring immediate assistance during a period of emergency, i.e. requiring basic survival needs such as food, water, shelter, sanitation and immediate medical assistance.
    ~ Column homeless_per_100k_people (changed metadata)
-       - title: Number of homeless persons per 100,000 people
        ?        ^^^^^^^^^^^           --------
+       + title: Homeless per 100,000 people
        ?        ^
+       + description_short: Rate of people per 100,000 whose houses are destroyed or heavily damaged and therefore need shelter after
+       +   an event.
    ~ Column injured_per_100k_people (changed metadata)
-       - title: Number of injured persons per 100,000 people
        ?        ^^^^^^^^^^^          --------
+       + title: Injured per 100,000 people
        ?        ^
+       + description_short: |-
+       +   Rate of people per 100,000 suffering from physical injuries, trauma or an illness requiring immediate medical assistance as a direct result of a disaster.
    ~ Column insured_damages_per_gdp (changed metadata)
-       - title: Insured damages from natural disasters as a share of GDP
        ?                        ----------------  ----  ---
+       + title: Insured damages as a share of GDP
    ~ Column n_events (changed metadata)
-       - title: Number of reported natural disasters
+       + title: Natural disasters
    ~ Column n_events_per_100k_people (changed metadata)
-       - title: Number of events per 100,000 people
+       + title: Natural disasters per 100,000 people
+       + description_short: Rate of reported natural disasters per 100,000 people.
    ~ Column reconstruction_costs (changed metadata)
-       - description_short: Reconstruction costs.
    ~ Column reconstruction_costs_per_gdp (changed metadata)
-       - title: Reconstruction costs from natural disasters as a share of GDP
        ?                             ----------------  ----  ---
+       + title: Reconstruction costs as a share of GDP
    ~ Column total_affected_per_100k_people (changed metadata)
-       - title: Total number of affected persons per 100,000 people
        ?             ----------             --------
+       + title: Total affected per 100,000 people
+       + description_short: Rate of people per 100,000 affected by a natural disaster.
    ~ Column total_damages (changed metadata)
-       - title: Total economic damage from natural disasters
+       + title: Total economic damages
    ~ Column total_damages_per_gdp (changed metadata)
-       - title: Total economic damages from natural disasters as a share of GDP
        ?                               ----------------  ----  ---
+       + title: Total economic damages as a share of GDP
    ~ Column total_dead (changed metadata)
-       - title: Total deaths
        ?        ^^^^^^^
+       + title: Deaths
        ?        ^
-       - description_short: Total number of deaths as a result of a natural disaster.
+       + description_short: Total number of dead and missing people.
    ~ Column total_dead_per_100k_people (changed metadata)
-       - title: Total number of deaths per 100,000 people
        ?        ^^^^^^^^^^^^^^^^^
+       + title: Deaths per 100,000 people
        ?        ^
+       + description_short: Death rate per 100,000 people as a result of a natural disaster.
  = Table natural_disasters_decadal_impact
  = Table natural_disasters_yearly_impact
  = Table natural_disasters_decadal_deaths
  = Table natural_disasters_yearly
    ~ Column affected_per_100k_people (changed metadata)
-       - title: Number of affected persons per 100,000 people
        ?        ^^^^^^^^^^^           --------
+       + title: Affected per 100,000 people
        ?        ^
+       + description_short: |-
+       +   Rate of people per 100,000 requiring immediate assistance during a period of emergency, i.e. requiring basic survival needs such as food, water, shelter, sanitation and immediate medical assistance.
    ~ Column homeless_per_100k_people (changed metadata)
-       - title: Number of homeless persons per 100,000 people
        ?        ^^^^^^^^^^^           --------
+       + title: Homeless per 100,000 people
        ?        ^
+       + description_short: Rate of people per 100,000 whose houses are destroyed or heavily damaged and therefore need shelter after
+       +   an event.
    ~ Column injured_per_100k_people (changed metadata)
-       - title: Number of injured persons per 100,000 people
        ?        ^^^^^^^^^^^          --------
+       + title: Injured per 100,000 people
        ?        ^
+       + description_short: |-
+       +   Rate of people per 100,000 suffering from physical injuries, trauma or an illness requiring immediate medical assistance as a direct result of a disaster.
    ~ Column insured_damages_per_gdp (changed metadata)
-       - title: Insured damages from natural disasters as a share of GDP
        ?                        ----------------  ----  ---
+       + title: Insured damages as a share of GDP
    ~ Column n_events (changed metadata)
-       - title: Number of reported natural disasters
+       + title: Natural disasters
    ~ Column n_events_per_100k_people (changed metadata)
-       - title: Number of events per 100,000 people
+       + title: Natural disasters per 100,000 people
+       + description_short: Rate of reported natural disasters per 100,000 people.
    ~ Column reconstruction_costs (changed metadata)
-       - description_short: Reconstruction costs.
    ~ Column reconstruction_costs_per_gdp (changed metadata)
-       - title: Reconstruction costs from natural disasters as a share of GDP
        ?                             ----------------  ----  ---
+       + title: Reconstruction costs as a share of GDP
    ~ Column total_affected_per_100k_people (changed metadata)
-       - title: Total number of affected persons per 100,000 people
        ?             ----------             --------
+       + title: Total affected per 100,000 people
+       + description_short: Rate of people per 100,000 affected by a natural disaster.
    ~ Column total_damages (changed metadata)
-       - title: Total economic damage from natural disasters
+       + title: Total economic damages
    ~ Column total_damages_per_gdp (changed metadata)
-       - title: Total economic damages from natural disasters as a share of GDP
        ?                               ----------------  ----  ---
+       + title: Total economic damages as a share of GDP
    ~ Column total_dead (changed metadata)
-       - title: Total deaths
        ?        ^^^^^^^
+       + title: Deaths
        ?        ^
-       - description_short: Total number of deaths as a result of a natural disaster.
+       + description_short: Total number of dead and missing people.
    ~ Column total_dead_per_100k_people (changed metadata)
-       - title: Total number of deaths per 100,000 people
        ?        ^^^^^^^^^^^^^^^^^
+       + title: Deaths per 100,000 people
        ?        ^
+       + description_short: Death rate per 100,000 people as a result of a natural disaster.
  = Table natural_disasters_yearly_deaths


Legend: +New  ~Modified  -Removed  =Identical  Details
Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet

Automatically updated datasets matching weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile are not included

Edited: 2024-05-22 14:59:29 UTC
Execution time: 24.59 seconds

@Marigold Marigold force-pushed the chart-diff-improvements branch from 0bbb164 to 0de2531 Compare May 22, 2024 14:49
@Marigold Marigold requested a review from lucasrodes May 22, 2024 14:49
@Marigold Marigold force-pushed the chart-diff-improvements branch 2 times, most recently from 27933b4 to 2acac2e Compare May 22, 2024 14:57
@lucasrodes
Copy link
Member

Thanks for doing this, mojmir! I'll take a look in brief.

Lately I've been adding lists of bugs to this issue #2610; so in case you solved any, could be useful if you crossed them. Let me know if you do it, or I can take a look later otherwise.

@Marigold
Copy link
Collaborator Author

Thanks for reminder Lucas. I couldn't find any of them there (those that I fixed are minor).

Comment on lines +326 to +330

modified_charts_pagination = Pagination(
chart_diffs_modified, items_per_page=CHART_PER_PAGE, pagination_key="pagination_modified"
)
modified_charts_pagination.show_controls()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

beautiful

Copy link
Member

@lucasrodes lucasrodes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @Marigold !

@Marigold Marigold force-pushed the chart-diff-improvements branch from 2acac2e to 6b22650 Compare May 24, 2024 07:42
@Marigold Marigold merged commit 9eb6009 into master May 24, 2024
5 of 9 checks passed
@Marigold Marigold deleted the chart-diff-improvements branch May 24, 2024 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants