Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to Rwanda REMA data source #1089

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Update to Rwanda REMA data source #1089

wants to merge 11 commits into from

Conversation

majesticio
Copy link
Contributor

Updates to our Rwanda source, uses a new data URL

@majesticio majesticio requested a review from caparker March 8, 2024 21:52
  Showing parameters at the top of the file and added a method to show
  min/max of values when doing dry runs.
@caparker
Copy link
Collaborator

A few things to do before we can push this one

  1. I updated the way that we are specifying the units. Moving forward lets do it this way for all new/updated adapters.
  2. Is there a way to get the siteid in there as the source_id? For example
"title": "Rusizi", "site_id": "RSZ"
  1. Is there a reason that you are filtering out the zeros from the measurements?
  2. Update the documentation for this adapter based on our new template. I would like to see more details about the units and why we have chosen the units we have. If there is data already in the database then lets look at the range of that data and include it.

I added a method that will summarize the parameters while doing the dry run, the output will look something like this

  {
    "message": "[Dry Run] New measurements found for rwanda-rema: 890",
    "failures": {},
    "count": 890,
    "duration": 0.117,
    "from": "2024-03-14T23:39:15.000Z",
    "to": "2024-03-14T23:39:15.000Z",
    "parameters": {
      "co": {
        "min": 0,
        "max": 766.23,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "no2": {
        "min": 0,
        "max": 38.01,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "o3": {
        "min": 0,
        "max": 57.47,
        "nulls": 0,
        "errors": 0,
        "count": 140
      },
      "pm10": {
        "min": 0,
        "max": 71.46,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "pm25": {
        "min": 0,
        "max": 60.71,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "so2": {
        "min": -218.94,
        "max": 59.97,
        "nulls": 0,
        "errors": 0,
        "count": 150
      }
    },
    "sourceName": "rwanda-rema"
  }
]

That should help with figuring out the units.

@majesticio
Copy link
Contributor Author

That all sounds good. I was seeing a lot of zero values being returned which I interpreted to mean that there was no data available, which was assumption on my part. I will stop filtering for 0 values.

Copy link
Collaborator

@caparker caparker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good and its functional but lets go ahead and add docstrings. Also take a quick look through it and add a comment above any line that you think could throw an error. I want us to start anticipating errors and so this could be a good exercise to get us started.

@majesticio
Copy link
Contributor Author

Requested changes have been made and merge conflicts resolved

Copy link
Collaborator

@caparker caparker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great but lets update it so that we dont loose all the measurements if just one of them is bad. So add a try/catch block in that foreach loop, likely the one that loops through the properties of a feature and then log out the error so we can eventually see it but dont rethrow it.

@majesticio
Copy link
Contributor Author

Added a try/catch block as requested

@majesticio
Copy link
Contributor Author

There is an issue with time stamps being ahead with this adapter. The provided timestamps eg 2024-04-16T17:32:45Z imply that this is UTC time. However, the resulting measurements are slightly ahead of current time. Additionally, looking at the timestamps ignoring UTC timezone, they align completely with the local time in Rwanda.

I have updated the code to compensate for this discrepancy and will add a note in the documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants