🔨 Jinja metadata for UN IGME #1898

spoonerf · 2023-11-02T14:57:51Z

Hey @Marigold this is a bit of a WIP but I'm trying to figure out Jinja templates. I think I am along the right lines but it doesn't seem to be adding the metadata to the variables as I expect. Probably missing something silly but if you could take a look when you get a chance that would be ace!

spoonerf · 2023-11-02T15:32:14Z

Hey @Marigold this is a bit of a WIP but I'm trying to figure out Jinja templates. I think I am along the right lines but it doesn't seem to be adding the metadata to the variables as I expect. Probably missing something silly but if you could take a look when you get a chance that would be ace!

Ah, I think it's because the section at the top e.g. where I have indicator, that should in fact be variable names? I probably need to restructure the data so that there is a column for each indicator...

Marigold · 2023-11-02T16:13:58Z

You were pretty close @spoonerf! Here is the corrected version (took me a while, working with templates can get complex)

definitions:
  common:
    presentation:
      topic_tags:
        - Child & Infant Mortality
  sex: |-
    <% if sex == "Both sexes" %>

    <% elif sex == "Male" %>
    male
    <% elif sex == "Female" %>
    female
    <%- endif -%>
  wealth_quintile: |-
    <% if wealth_quintile == "All wealth quintiles" %>
    all wealth quintiles
    <% elif wealth_quintile == "Poorest quintile" %>
    the poorest 20% of the population
    <% elif wealth_quintile == "Second poorest quintile" %>
    the second poorest 20% of the population
    <% elif wealth_quintile == "Middle wealth quintile" %>
    the middle 20% of the population in terms of wealth
    <% elif wealth_quintile == "Fourth poorest quintile" %>
    the second richest 20% of the population
    <% elif wealth_quintile == "Richest quintile" %>
    the richest 20% of the population
    <%- endif -%>
  unit_of_measure:
    unit: |-
      <% if unit_of_measure == "Deaths per 1,000 live births" %>
      deaths per 1,000 live births
      <% elif unit_of_measure == "Deaths per 1000 children aged 10" %>
      deaths per 1,000 children aged 10
      <% elif unit_of_measure == "Deaths per 1000 children aged 15" %>
      deaths per 1,000 children aged 15
      <% elif unit_of_measure == "Deaths per 1000 children aged 1" %>
      deaths per 1,000 children aged 1
      <% elif unit_of_measure == "Deaths per 1000 children aged 20" %>
      deaths per 1,000 children aged 20
      <% elif unit_of_measure == "Deaths per 1000 children aged 5" %>
      deaths per 1,000 children aged 5
      <% elif unit_of_measure == "Number of stillbirths" %>
      stillbirths
      <% elif unit_of_measure == "Stillbirths per 1,000 total births" %>
      stillbirths per 1,000 total births
      <% elif unit_of_measure == "Number of deaths" %>
      deaths
      <%- endif -%>
  indicator:
    display_name: |-
      <% if indicator == "Neonatal mortality rate" %>
      The estimated number of deaths of << sex >> children aged less than 28 days, per 1,000 live births, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate 1-59 months" %>
      The estimated number of deaths of << sex >> children aged between 1 and 59 months, per 1,000 live births, in <<wealth_quintile>>.
      <% elif indicator == "Infant mortality rate" %>
      The estimated number of deaths of << sex >> children aged under one year, per 1,000 live births, in <<wealth_quintile>>.
      <% elif indicator == "Under-five mortality rate" %>
      The estimated number of deaths of << sex >> children aged under five years, per 1,000 live births, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 10-14" %>
      The estimated number of deaths of << sex >> children aged between 10 and 14, per 1,000 children aged 10, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 10-19" %>
      The estimated number of deaths of << sex >> children aged between 10 and 19, per 1,000 children aged 10, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 15-19" %>
      The estimated number of deaths of << sex >> children aged between 15 and 19, per 1,000 children aged 15, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 15-24" %>
      The estimated number of deaths of << sex >> children aged between 15 and 24, per 1,000 children aged 15, in <<wealth_quintile>>.
      <% elif indicator == "Child Mortality rate age 1-4" %>
      The estimated number of deaths of << sex >> children aged between 1 and 4, per 1,000 children aged 1, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 20-24" %>
      The estimated number of deaths of << sex >> children aged between 20 and 24, per 1,000 children aged 20, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-24" %>
      The estimated number of deaths of << sex >> children aged between 5 and 24, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-9" %>
      The estimated number of deaths of << sex >> children aged between 5 and 9, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Stillbirths" %>
      The estimated number of stillbirths, in <<wealth_quintile>>.
      <% elif indicator == "Stillbirth rate" %>
      The estimated number of stillbirths, per 1,000 total births, in <<wealth_quintile>>.
      <% elif indicator == "Neonatal deaths" %>
      The estimated number of deaths of << sex >> children aged less than 28 days, in <<wealth_quintile>>.
      <% elif indicator == "Deaths 1-59 months" %>
      The estimated number of deaths of << sex >> children aged between 1 and 59 months, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <% elif indicator == "Mortality rate age 5-14" %>
      The estimated number of deaths of << sex >> children aged between 5 and 14, per 1,000 children aged 5, in <<wealth_quintile>>.
      <%- endif -%>
tables:
  igme:
    variables:
      observation_value:
        title: Observation value
        unit: |-
          {definitions.unit_of_measure.unit}
        display:
          name: |-
            {definitions.indicator.display_name}
      lower_bound:
        title: Lower bound
        unit: |-
          {definitions.unit_of_measure.unit}
        display:
          name: |-
            {definitions.indicator.display_name}
      upper_bound:
        title: Upper bound
        unit: |-
          {definitions.unit_of_measure.unit}
        display:
          name: |-
            {definitions.indicator.display_name}

A couple of comments:

In Jinja templates you should use << ... >> to reference dimension values, not <%
Dynamic placeholders must not contain whitespace, i.e. use {definitions.unit_of_measure.unit} and not { definitions.unit_of_measure.unit } (not sure why it doesn't work...)
I had to change definitions.indicator.display.name to definitions.indicator.display_name because YAML doesn't like the dot

The template itself is good after fixing those small issues. I've tried running etl un/2023-08-16/igme --grapher and got proper display names in Admin.

(When inserting to grapher, I got the error Variables used in charts will not be deleted automatically. That typically means variables have changed short names. If you struggle with fixing it, ping me and I'll be glad to help)

spoonerf · 2023-11-02T16:24:19Z

Amazing thanks @Marigold! I'll try and figure out the next steps and almost certainly ping you. I think it's likely the short names will be different, but I think we've got round that before by using chart revisions?

Marigold · 2023-11-02T17:10:00Z

but I think we've got round that before by using chart revisions?

Exactly. By the way it'd be great if you could ping me after this PR is ready before doing chart revisions. I'd like to test syncing of changes from staging servers to production.

spoonerf · 2023-11-03T12:26:45Z

but I think we've got round that before by using chart revisions?

Exactly. By the way it'd be great if you could ping me after this PR is ready before doing chart revisions. I'd like to test syncing of changes from staging servers to production.

Hey @Marigold - I think we're at this stage now, I believe the only issue remaining is the Variables used in charts will not be deleted automatically one.

Marigold · 2023-11-07T08:32:06Z

I looked into the Variables used in charts will not be deleted automatically error. It happens because variables now use different (short) names.

You should first rebase or merge this PR onto the master. I'd also recommend reordering index levels to be consistent with the current version to make chart revision easier

tb = tb.reorder_levels(["country", "year", "unit_of_measure", "indicator", "sex", "wealth_quintile"])
tb_youth = tb_youth.reorder_levels(["country", "year", "unit_of_measure", "indicator", "sex", "wealth_quintile"])

If you make this change and commit back, staging server will rebuild ETL and you should see an updated dataset there.

Once you have that, the best way to fix this error is to map those variables with

STAGING=igme-metadata etl-wizard charts

and then approving them in chart revisions.

I tried to do it myself, but it's not as straightforward as some old variables don't exist in the current dataset (something with middle income group and possibly others).

With the new workflow, you can approve chart revisions on the staging server, then I can sync them to production (after we merge this PR).

spoonerf · 2023-11-07T10:12:17Z

Amazing, thanks @Marigold!

I had messed up some of the variables, making variable matching impossible, but sorted that out now. I have approved the chart revisions on the staging server. So I think we can merge this now? I will wait to hear from you before merging.

* changing units to % * fixing short_unit * fixing conversion

This reverts commit 2c6a7a3.

spoonerf added 4 commits November 2, 2023 09:50

adding under fifteen deaths

66d6a4d

update

5af6681

adding jinja

75f06df

more jinja

0bb7c8d

github-actions bot assigned spoonerf Nov 2, 2023

merge w master

b33c32a

spoonerf requested a review from Marigold November 2, 2023 15:07

spoonerf added 3 commits November 2, 2023 15:23

adding topic back in

d185a9b

jinja adjust

52f72f5

more

6047759

spoonerf removed the request for review from Marigold November 2, 2023 15:57

spoonerf added 10 commits November 3, 2023 11:03

cleaning up the code

8558075

adding u15 mr metadata

b2c6f1b

tidying up

3995603

decimal places

c64b8b1

schema ignore

9ee534a

adding u15 to grapher

1394ec9

adding post-neonatal

cbd3055

adding under 15 to jinja

19ad54a

deduplicating jinja

0488661

fixing long run child mortality

d000554

spoonerf marked this pull request as ready for review November 3, 2023 12:20

spoonerf requested a review from Marigold November 3, 2023 12:25

Merge branch 'master' into igme-metadata

fdb4212

spoonerf added 5 commits November 7, 2023 09:12

reordering short name as per mojmir's suggestion

a9cc92c

adding description short

b487da0

removing desc short

874b2b6

removing unusued variables from u15 table

3eabfb3

fix

5af1f3c

🔨 Converting UN IGME per 1,000 rates to percentages (#1918)

6cfcf4b

* changing units to % * fixing short_unit * fixing conversion

spoonerf merged commit 2c6a7a3 into master Nov 8, 2023
1 check passed

spoonerf deleted the igme-metadata branch November 8, 2023 09:33

spoonerf added a commit that referenced this pull request Nov 8, 2023

Revert "🔨 Jinja metadata for UN IGME (#1898)"

4c7705b

This reverts commit 2c6a7a3.

spoonerf mentioned this pull request Nov 8, 2023

Revert "🔨 Jinja metadata for UN IGME" #1924

Closed

spoonerf restored the igme-metadata branch November 8, 2023 09:42

lucasrodes deleted the igme-metadata branch June 13, 2024 20:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🔨 Jinja metadata for UN IGME #1898

🔨 Jinja metadata for UN IGME #1898

spoonerf commented Nov 2, 2023 •

edited

Loading

spoonerf commented Nov 2, 2023

Marigold commented Nov 2, 2023

spoonerf commented Nov 2, 2023

Marigold commented Nov 2, 2023

spoonerf commented Nov 3, 2023

Marigold commented Nov 7, 2023

spoonerf commented Nov 7, 2023

🔨 Jinja metadata for UN IGME #1898

🔨 Jinja metadata for UN IGME #1898

Conversation

spoonerf commented Nov 2, 2023 • edited Loading

spoonerf commented Nov 2, 2023

Marigold commented Nov 2, 2023

spoonerf commented Nov 2, 2023

Marigold commented Nov 2, 2023

spoonerf commented Nov 3, 2023

Marigold commented Nov 7, 2023

spoonerf commented Nov 7, 2023

spoonerf commented Nov 2, 2023 •

edited

Loading