Fix initial view #81

hcorson-dosch-usgs · 2022-12-02T16:11:48Z

Summary

This PR addresses the observed problems with the initial view and animation, discussed here in PR #79. Key problems were

The initial view did not seem to reflect starting conditions for wy 22
Levels for sites in HI and FL appeared static throughout much of the year

The problems turned out to be tied to NA values in the data, stemming from 5 issues:

Issue	Fix
In the Javascript, `NA` GWL values being automatically assigned a path and color equivalent to 'very low'	Assign path value of `null` when data value `NA`
The first row of the `gw-conditions-peaks-timeseries-wy22.csv` on S3 (GWL values for 1st day) was nearly entirely NA	Fixed when I had to re-run `1_fetch` in order to dig into the steps in `2_process` and `3_visualize`
Values for HI and FL sites were NA from October-January and April-September	Fixed when I had to re-run `1_fetch` in order to dig into the steps in `2_process` and `3_visualize`
GWL values for some sites were assigned a `daily_quant` and `quant_category` of `NA` because they fell outside of the historical range	Assign the value the max or min quantile as appropriate
Some sites had no groundwater data for the water year, but were still included	When subset to viz period, filter out sites with no data - this dropped 5 sites

Updated view

There are now 2211 total sites for WY22. This is the new initial view:

To run

Pull this fork to your local clone and run npm run serve

Overview of key changes

Changes to JavaScript:

Both when drawing the initial view and during animation, path values are now set to null if the GWL value for that day for that site is NA
- Previously, NA values had been assigned a value equivalent to the "very low" category:
- This fixed the initial view and display of NA values in FL and HI from October- January and then again April-end of September.

Changes to R pipeline:

Not a change, but I had to re-run 1_fetch, 2_process and 3_visualize in order to work in the pipeline. This re-pulled the NWIS data, and resolved the NA values on day 1 and the NA values in HI and FL
Some sites w/ groundwater level data were being assigned a quantile of NA because the level fell outside of the historic range. I added a step in compare_to_historic to assign the value the max or min quantile as appropriate.
Some sites had no data for the water year but were still included. Those are now filtered out when we filter to the viz period.

Additional minor changes

Other minor JavaScript changes

Drop use of var throughout
Use single source to define day sequence
Make sure the first site is added to peaky
Since the groundwater sites are added to the svg entirely with D3 (we are not currently binding data to elements that already exist within the DOM) I simplified the syntax of how the sites are added for readability.

Minor edits to the pipeline:

Add the states shapefile as a dependency for gw_sites_sf_shifted
Use the 'default' AWS profile since the DSP recommends setting up AWS profiles with the default profile name, so I think most of our team has it set up that way
Drop filepath named within function since the filename is already passed as an argument
Update dependencies in 1_fetch since 'gw_sites.rds' doesn't exist

cnell-usgs · 2022-12-07T03:07:55Z

src/components/GWL.vue

+          // this.d3.selectAll(".ticker-date")
+          // .transition()
+          //   .duration(this.day_length) 
+          //   .text(this.dates[start])
+          // .end()
+          // .then(() => this.animateLine(start))


Was this not getting used?

Yeah, the code that adds the date ticker was already commented out, as was the end animation for it, so I commented out this piece since it's not currently being used.

cnell-usgs · 2022-12-07T03:08:21Z

src/components/GWL.vue

-          this.peak_grp = map_svg.selectAll("path.gwl_glyph")
-            .data(data, function(d) { return d ? d.key : this.class; }) // binds data based on class/key
-            .join("path") // match with selection
-            .attr("transform", d => `translate(` + d.site_x + ' ' + d.site_y + `) scale(0.35 0.35)`)
-
-        // draws a path for each site, using the first date
-        this.peak_grp 
+        // set up group to hold paths
+        const peakSvgGroup = map_svg.append("g")
+          .attr("id", "peak-map-grp")
+
+        // Add path for each site, using the first date
+        this.peak_grp = peakSvgGroup.selectAll("gwl")
+          .data(data)
+          .enter()
+          .append("path") // append path for each site
+          .attr("transform", d => `translate(` + d.site_x + ' ' + d.site_y + `) scale(0.35 0.35)`)


cnell-usgs · 2022-12-07T03:09:02Z

src/components/GWL.vue

+              let today = d.gwl[start] == "NA" ? null : self.quant_path_gylph(d.gwl[start]) // set to null if value is NA
+              let yesterday = d.gwl[start-1] == "NA" ? null : self.quant_path_gylph(d.gwl[start-1]) // set to null if value is NA


I did not register the difference of NA vs null

cnell-usgs

Thanks for writing this out so clearly. You caught some known issues plus some 💯 . The animation ran for me and reflects what I would expect with these edits. Ends on this view:

For the last fix noted in the table, "When subset to viz period, filter out sites with no data - this dropped 5 sites" this works for now, but when we start offering multiple time periods we will need to revisit. If the sites get filtered each time frame, we'll need to have corresponding site locations and in some way ensure that the sites on the map include all sites with data. That may mean moving filtering to vue, so it can be dynamic to the user.

Somewhere before this we lost the dotted lines that delineated and labelled all the islands are territories. I'm not sure why that is.

hcorson-dosch-usgs · 2022-12-07T14:38:27Z

For the last fix noted in the table, "When subset to viz period, filter out sites with no data - this dropped 5 sites" this works for now, but when we start offering multiple time periods we will need to revisit. If the sites get filtered each time frame, we'll need to have corresponding site locations and in some way ensure that the sites on the map include all sites with data. That may mean moving filtering to vue, so it can be dynamic to the user.

Ooh, good point. I think I'm going to start an issue re: switching to a monthly view w/ multiple time periods, and will include this consideration there.

Somewhere before this we lost the dotted lines that delineated and labelled all the islands are territories. I'm not sure why that is.

I noticed that, too! I'll take a poke back through and see if I can find where it dropped out

hcorson-dosch-usgs · 2022-12-08T00:11:21Z

Added back in the map labels and section lines, which I think maybe had been manually added to the R-generated svg before? I've now set them up so that they are in a separate svg, which is committed to the repo. Looks like this now:

hcorson-dosch-usgs · 2022-12-08T00:47:41Z

To keep the outer glow from being cut off at the top, I've also added in a transformation when the background map svg is written, the labels svg is overlaid, and the sites are added to the map. Now looks like this:

cnell-usgs · 2022-12-08T04:37:37Z

Added back in the map labels and section lines, which I think maybe had been manually added to the R-generated svg before? I've now set them up so that they are in a separate svg, which is committed to the repo. Looks like this now:

sounds like something I did in a rush 😥

cnell-usgs

Good work, Hayley. You really went above and beyond here.

hcorson-dosch-usgs added 15 commits November 30, 2022 11:44

drop use of var to set variables throughout

24e6fac

use this.days in place of repeated var day_seq

eef7921

fix error where first site not added to peaky array

970d20c

simplify d3 syntax for when site paths are added to map

ffad8a5

account for NAs when setting and transitioning paths

aadf994

automated changes to package-lock.json

4920a2e

Add states shp as dependency for gw_sites_sf_shifted

33e708f

Use default profile, per DSP manual

bd0aa71

Update svg and s3 ind files

53157eb

Account for values outside of historic range

a6daeda

Drop sites w/ entirely NA GWL values during viz period

c9bd3d7

Fix typo in range fix

3ad8157

Update data files on S3

ee9f44b

drop filepath named in function, since provided as argument

70a950c

Update site info file dependency in 1_fetch

38bf198

hcorson-dosch-usgs requested a review from cnell-usgs December 2, 2022 16:11

hcorson-dosch-usgs added 2 commits December 6, 2022 09:30

Use approx rule = 2 instead of separate statement

6270404

drop now unused variables

8545ef3

cnell-usgs reviewed Dec 7, 2022

View reviewed changes

cnell-usgs approved these changes Dec 7, 2022

View reviewed changes

hcorson-dosch-usgs added 3 commits December 7, 2022 18:05

Auto update to package lock

93827bf

Add in map labels as separate imported svg

3975c99

Separate svg with map sections and labels

5eb3b7d

hcorson-dosch-usgs requested a review from cnell-usgs December 8, 2022 00:11

hcorson-dosch-usgs added 2 commits December 7, 2022 18:41

Add svg transformation so that glow not cut off

f3a0894

Adjust transformation

7a39a24

cnell-usgs approved these changes Dec 8, 2022

View reviewed changes

hcorson-dosch-usgs merged commit 9d7005a into DOI-USGS:main Dec 8, 2022

hcorson-dosch-usgs deleted the fix_initial_view branch December 8, 2022 14:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix initial view #81

Fix initial view #81

hcorson-dosch-usgs commented Dec 2, 2022 •

edited

Loading

cnell-usgs Dec 7, 2022

hcorson-dosch-usgs Dec 7, 2022

cnell-usgs Dec 7, 2022

cnell-usgs Dec 7, 2022

cnell-usgs left a comment

hcorson-dosch-usgs commented Dec 7, 2022

hcorson-dosch-usgs commented Dec 8, 2022 •

edited

Loading

hcorson-dosch-usgs commented Dec 8, 2022

cnell-usgs commented Dec 8, 2022

cnell-usgs left a comment

		let today = d.gwl[start] == "NA" ? null : self.quant_path_gylph(d.gwl[start]) // set to null if value is NA
		let yesterday = d.gwl[start-1] == "NA" ? null : self.quant_path_gylph(d.gwl[start-1]) // set to null if value is NA

Fix initial view #81

Fix initial view #81

Conversation

hcorson-dosch-usgs commented Dec 2, 2022 • edited Loading

Summary

Updated view

To run

Overview of key changes

Changes to JavaScript:

Changes to R pipeline:

Additional minor changes

cnell-usgs Dec 7, 2022

Choose a reason for hiding this comment

hcorson-dosch-usgs Dec 7, 2022

Choose a reason for hiding this comment

cnell-usgs Dec 7, 2022

Choose a reason for hiding this comment

cnell-usgs Dec 7, 2022

Choose a reason for hiding this comment

cnell-usgs left a comment

Choose a reason for hiding this comment

hcorson-dosch-usgs commented Dec 7, 2022

hcorson-dosch-usgs commented Dec 8, 2022 • edited Loading

hcorson-dosch-usgs commented Dec 8, 2022

cnell-usgs commented Dec 8, 2022

cnell-usgs left a comment

Choose a reason for hiding this comment

hcorson-dosch-usgs commented Dec 2, 2022 •

edited

Loading

hcorson-dosch-usgs commented Dec 8, 2022 •

edited

Loading