Nhsn indicator #2080

aysim319 · 2024-11-18T20:07:09Z

Description

Adding new indicator NHSN

Associated Issue(s)

Addresses NHSN #2079

nmdefries · 2024-11-20T18:12:51Z

Waiting on decisions from tooling team on what smoothing and geo aggregations we should do, and if we should include any other metrics (currently generates total flu and covid hospital admissions. Source data also includes metrics by age group, but we aren't ingesting those).

RE geo aggregation, the previous hhs indicator only pulled state and nation data, and didn't do any geo aggregation. This indicator does the same thing.

@dshemetov @brookslogan @dsweber2 for input.

nmdefries · 2024-11-20T18:16:48Z

Due to production issues, will aim to release this early next week.

aysim319 · 2024-11-20T22:04:26Z

context from david:

oh, also, dunno if you saw the post from logan, they're doing this really dumb thing of putting the preliminary data in a separate source https://data.cdc.gov/Public-Health-Surveillance/Weekly-Hospital-Respiratory-Data-HRD-Metrics-by-Ju/mpgq-jmmr/about_data
we should probably include those as separate versions in the same indicator
oh er, when I said versions, I meant versions (like the release date) rather than different signals. Definitely not the best wording choice.

Currently have it such that they're seperate signals similar to nssp secondary source to differienate them as a place holder

dsweber2 · 2024-11-21T19:02:50Z

Working through this. I tried a couple of smoothings (3 & 4 weeks, both trailing and centered) to see if that would help tame some of the more extreme data issues, but it didn't really help. I had a use-case where hhs_regions would've been handy, so that's probably worth adding eventually (probably more important to get this live first though).

dsweber2

Looks good to me! A few things I'm a bit confused about, and some minor notes, but I would be glad to see what this does on staging.

nhsn/DETAILS.md

nhsn/delphi_nhsn/constants.py

dsweber2 · 2024-11-21T18:08:29Z

nhsn/delphi_nhsn/pull.py

+        Dataframe as described above.
+    """
+    # Pull data from Socrata API
+    df = pull_data(socrata_token, dataset_id="ua7e-t2fy")


It's easier to test transform logic if you separate the Extract and Transform steps. Probably fine in this case though, since we're not really transforming it.

I believe this is holdover from my nssp and nchs dirty code practice. If we aren't doing a full rewrite soon, I would refactor them at some point. For now, they all work (ugly).

I think I was bouncing back and forth, part of it being what if the other data source has different transformations down the line? maybe better to have a method defined? something along those lines. I did refactor the pull_data, since that doesn't seem like it would change

dsweber2 · 2024-11-21T19:06:08Z

nhsn/tests/test_pull.py

@@ -0,0 +1,149 @@
+import glob
+import json


apparently json is unused

Suggested change

import json

dsweber2 · 2024-11-21T19:11:06Z

nhsn/tests/test_run.py

+
+
+class TestRun:
+    # the 14th was a Monday


I'm sure this comment made sense at one point >.<

nhsn/tests/test_run.py

minhkhul

Just need these and I think we can go to staging:

covidcast-indicators/_template_python/INDICATOR_DEV_GUIDE.md

Lines 473 to 483 in feea58d

    
           * Add module name to the `build` job in `.github/workflows/python-ci.yml`. 
        
             This allows github actions to run on this indicator code, which includes unit tests and linting. 
        
           * Add module name to the ["Copy version to indicator directory" step](https://github.com/cmu-delphi/covidcast-indicators/blob/f01185767a9847d8082baf4f1e17be50a39047c2/.github/workflows/create-release.yml#L64) in `.github/workflows/create-release.yml`. 
        
           * Add top-level directory name to [`indicator_list` in `Jenkinsfile`](https://github.com/cmu-delphi/covidcast-indicators/blob/f01185767a9847d8082baf4f1e17be50a39047c2/Jenkinsfile#L13). 
        
             This allows your code to be automatically deployed to staging after your branch is merged to main, and deployed to prod after `covidcast-indicators` is released. 
        
           * Create `ansible/templates/{top_level_directory_name}-params-prod.json.j2` based on your `params.json.template` with some adjustment: 
        
              * "export_dir": "/common/covidcast/receiving/{data-source-name}" 
        
              * "log_filename": "/var/log/indicators/{top_level_directory_name}.log" 
        
           * Define any sensitive variables as "secrets" in the [Ansible `vars.yaml`](https://github.com/cmu-delphi/covidcast-indicators/blob/main/ansible/vars.yaml) and [vault](https://github.com/cmu-delphi/covidcast-indicators/blob/main/ansible/vault.yaml). 
        
           Refer to [this guide](https://docs.google.com/document/d/1Bbuvtoxowt7x2_8USx_JY-yTo-Av3oAFlhyG-vXGG-c/edit#heading=h.8kkoy8sx3t7f) for more vault info. 
        
           * Add configs for Sir Complains-a-Lot ("sirCAL") alerting in sirCAL's [local](https://github.com/cmu-delphi/covidcast-indicators/blob/main/sir_complainsalot/params.json.template) and [Ansible](https://github.com/cmu-delphi/covidcast-indicators/blob/main/ansible/templates/sir_complainsalot-params-prod.json.j2) params templates.

nhsn/params.json.template

Co-authored-by: minhkhul <[email protected]>

nhsn/delphi_nhsn/run.py

nhsn/params.json.template

Co-authored-by: minhkhul <[email protected]>

nmdefries · 2024-12-06T20:42:42Z

Plots for statistical review have been approved

aysim319 · 2024-12-07T00:10:54Z

Waiting for signal name approval

aysim319 · 2024-12-10T15:57:49Z

Waiting for signal name approval

new signal name approved

aysim319 added 3 commits November 14, 2024 18:29

first implimentation

022c1e0

figuring out metric/sensor to use

541b0aa

first take

1c79068

aysim319 assigned nolangormley and minhkhul Nov 18, 2024

working on test

9a9780b

aysim319 requested review from nolangormley and minhkhul November 20, 2024 18:07

aysim319 unassigned nolangormley and minhkhul Nov 20, 2024

nmdefries marked this pull request as ready for review November 20, 2024 18:14

aysim319 and others added 9 commits November 20, 2024 15:38

added preliminary data source

24a17cd

adding indicator for gitaction

dde6685

lint

f5ad4c9

replace with setup.py

aedf0e9

more lint

25c24f1

fixed date range for test

4a4e169

lint

3cfcdf6

Update DETAILS.md

b818ca8

fix output data

eca8fe0

dsweber2 approved these changes Nov 21, 2024

View reviewed changes

aysim319 added 3 commits November 21, 2024 18:30

analysis in progress

711ace3

lint and suggestions

48a8a76

more analysis

8117096

minhkhul requested changes Nov 25, 2024

View reviewed changes

minhkhul reviewed Nov 25, 2024

View reviewed changes

nhsn/params.json.template Outdated Show resolved Hide resolved

minhkhul reviewed Nov 25, 2024

View reviewed changes

nhsn/params.json.template Outdated Show resolved Hide resolved

minhkhul reviewed Nov 25, 2024

View reviewed changes

nhsn/params.json.template Show resolved Hide resolved

aysim319 and others added 10 commits November 25, 2024 13:39

add hhs geo aggregate

411694c

more analysis

f42c2e9

update DETAILS.md

7da9e5a

Update nhsn/params.json.template

1b4c277

Co-authored-by: minhkhul <[email protected]>

Update nhsn/params.json.template

84f34fa

Co-authored-by: minhkhul <[email protected]>

Merge remote-tracking branch 'origin/nhsn_indicator' into nhsn_indicator

1dc6343

cleaning up anaylsis

3b5fbee

rename geo_id column name

e97c3cc

suggested / needed to deploy

26d69f3

adding default locations for deployment

678822a

minhkhul requested changes Nov 27, 2024

View reviewed changes

nhsn/delphi_nhsn/run.py Outdated Show resolved Hide resolved

minhkhul reviewed Nov 27, 2024

View reviewed changes

nhsn/params.json.template Outdated Show resolved Hide resolved

aysim319 and others added 3 commits December 2, 2024 10:13

fix geo aggregation for hhs

840ebb7

Co-authored-by: minhkhul <[email protected]>

Update nhsn/params.json.template

de601d6

Co-authored-by: minhkhul <[email protected]>

lint

928a5c7

minhkhul approved these changes Dec 5, 2024

View reviewed changes

needed to add hhs in to geo for tests

3ba877e

aysim319 requested a review from nmdefries December 6, 2024 20:37

fixed and added more plots

162152e

cleaning up notebook and adding details

5f26ff1

aysim319 added 2 commits December 10, 2024 10:15

new signal name

0000c73

needed to update the type dict also

de9dc95

aysim319 merged commit 5ff844a into main Dec 10, 2024
17 checks passed

minhkhul mentioned this pull request Dec 11, 2024

Release covidcast-indicators 0.3.57 #2086

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nhsn indicator #2080

Nhsn indicator #2080

aysim319 commented Nov 18, 2024

nmdefries commented Nov 20, 2024 •

edited

Loading

nmdefries commented Nov 20, 2024 •

edited

Loading

aysim319 commented Nov 20, 2024 •

edited

Loading

dsweber2 commented Nov 21, 2024

dsweber2 left a comment

dsweber2 Nov 21, 2024

minhkhul Nov 21, 2024

aysim319 Nov 21, 2024

dsweber2 Nov 21, 2024 •

edited

Loading

dsweber2 Nov 21, 2024

minhkhul left a comment

nmdefries commented Dec 6, 2024

aysim319 commented Dec 7, 2024

aysim319 commented Dec 10, 2024

	* Add module name to the `build` job in `.github/workflows/python-ci.yml`.
	This allows github actions to run on this indicator code, which includes unit tests and linting.
	* Add module name to the ["Copy version to indicator directory" step](https://github.com/cmu-delphi/covidcast-indicators/blob/f01185767a9847d8082baf4f1e17be50a39047c2/.github/workflows/create-release.yml#L64) in `.github/workflows/create-release.yml`.
	* Add top-level directory name to [`indicator_list` in `Jenkinsfile`](https://github.com/cmu-delphi/covidcast-indicators/blob/f01185767a9847d8082baf4f1e17be50a39047c2/Jenkinsfile#L13).
	This allows your code to be automatically deployed to staging after your branch is merged to main, and deployed to prod after `covidcast-indicators` is released.
	* Create `ansible/templates/{top_level_directory_name}-params-prod.json.j2` based on your `params.json.template` with some adjustment:
	* "export_dir": "/common/covidcast/receiving/{data-source-name}"
	* "log_filename": "/var/log/indicators/{top_level_directory_name}.log"
	* Define any sensitive variables as "secrets" in the [Ansible `vars.yaml`](https://github.com/cmu-delphi/covidcast-indicators/blob/main/ansible/vars.yaml) and [vault](https://github.com/cmu-delphi/covidcast-indicators/blob/main/ansible/vault.yaml).
	Refer to [this guide](https://docs.google.com/document/d/1Bbuvtoxowt7x2_8USx_JY-yTo-Av3oAFlhyG-vXGG-c/edit#heading=h.8kkoy8sx3t7f) for more vault info.
	* Add configs for Sir Complains-a-Lot ("sirCAL") alerting in sirCAL's [local](https://github.com/cmu-delphi/covidcast-indicators/blob/main/sir_complainsalot/params.json.template) and [Ansible](https://github.com/cmu-delphi/covidcast-indicators/blob/main/ansible/templates/sir_complainsalot-params-prod.json.j2) params templates.

Nhsn indicator #2080

Nhsn indicator #2080

Conversation

aysim319 commented Nov 18, 2024

Description

Associated Issue(s)

nmdefries commented Nov 20, 2024 • edited Loading

nmdefries commented Nov 20, 2024 • edited Loading

aysim319 commented Nov 20, 2024 • edited Loading

dsweber2 commented Nov 21, 2024

dsweber2 left a comment

Choose a reason for hiding this comment

dsweber2 Nov 21, 2024

Choose a reason for hiding this comment

minhkhul Nov 21, 2024

Choose a reason for hiding this comment

aysim319 Nov 21, 2024

Choose a reason for hiding this comment

dsweber2 Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

dsweber2 Nov 21, 2024

Choose a reason for hiding this comment

minhkhul left a comment

Choose a reason for hiding this comment

nmdefries commented Dec 6, 2024

aysim319 commented Dec 7, 2024

aysim319 commented Dec 10, 2024

nmdefries commented Nov 20, 2024 •

edited

Loading

nmdefries commented Nov 20, 2024 •

edited

Loading

aysim319 commented Nov 20, 2024 •

edited

Loading

dsweber2 Nov 21, 2024 •

edited

Loading