Download Sensor Data failing

[neilh10]

For a node on monitormywatershed.org, When I press the "DOWNLAOD SENSOR DATA" there is no action. In the past this results in a download of all stored data. There is no feedback to say its failed.

eg for the following in Chrome or Firefox
https://monitormywatershed.org/sites/nh_LCC45/

I tried this over 4 hours a number of times today - and no download any time

Monitoring MMW uptime doesn't seem to indicate any failures in DataStream bit.ly/mmw-uptime

Should I expect to be able to diagnose a failure -for user facing Homepage

The data stream does show an earlier failure

[neilh10]

This worked today. It maybe that it is sensitive to the size of the database being downloaded, and might need an indication that work is in progress.

[neilh10]

I'm not getting this on firefox looks like its firefox timing out after 5minutes.
On the following page I've pressed the "DOWNLOAD SENSOR DATA" - https://monitormywatershed.org/sites/TUCA_PO03/
The 5minutes later I get this

It would be nice to know the feedback is useful, something happening with it.

Sorta feels like >NULL

[aufdenkampe]

@neilh10, thanks for bringing this to our attention. The CSV download endpoint is unfortunately one of those tech-debt challenges that we unfortunately have had to put in our Tech Debt Parking Lot due to lack of funding.

The current CSV download endpoint is very inefficient and I'm guessing that your long time series are causing it to fail.

@ptomasula and I will look into it further, and also prioritize the following issues for the next round of maintenance work expected later this winter.

• Create a proper data download API #567
• Add ability to limit csv download to a custom date range. #568

cc: @ScottEnsign, @SRGDamia1, @s-hicks2

[ptomasula]

Thanks @neilh10. I agree with everything @aufdenkampe noted above, but will add that server logs confirm that this is a server timeout issue.

As a short term work around I would suggest trying to download timeseries for individual parameters as opposed to the download all the station data.

Downloading the timeseries for the single parameter still takes a good number of seconds (~30-40 seconds) but did successfully complete for me, and was well below the timeout threshold of 300 seconds. The URL in your screenshot above includes 8 result_ids, which with the 30-40 seconds per timeseries is right around that timeout limit.

[neilh10]

@ptomasula thanks for the tip of being able to get some data out.
Interesting that downloading one column takes less resources than downloading all columns.
My usual process is to look at validating the data by checking if there is stable set of data wireless delivered - from some specific past date.
So that means I download "sequence number", and then the area of interest the depth.

@aufdenkampe thanks for the response and insights. i hear the story, and technologies have discrete increments - and like the metaphor of the chain the weakest link in the chain defines the chain. Software has creative ways of being fixed, often by characterizing and detecting weak links before they become visible to everyone. oh well.!

[aufdenkampe]

@neilh10, I'm glad that you now have a solution for getting what you need while while you wait for the fix.

As you can see by the dates on related "download" issues, we've been aware of this issue from long before we inherited the codebase (see #312 from 2018) but were only able to complete one (#651) of the three tasks we identified to fix the problem (see #650 (comment)). We had to prioritize other more pressing issues.

Fortunately, we've been able to take care of those bigger issues and can now prioritize this one.

[neilh10]

Given the timescales, and I contributed absolute basic features in 2018, maybe its a conversation about the simplest vision of a working system.
If a user can't expect a "boot net replacement" what value are all the other bells and whistles.
Just my 2c
https://productdiscoverygroup.com/learn/the-product-discovery-valley-of-death#:~:text=A%20team%20might%20have%20entered,room%20for%20experimentation%20or%20innovation)

[ScottEnsign]

@neilh10, there is faster and more flexible method of retrieving data from Monitor My Watershed using R and python scripts. @SRGDamia1 shared the python version and R version.

[neilh10]

@ScottEnsign thanks for the suggestion - I'm trying to support real-world landowners, there is a good chance if they ask a question its likely ~ with a grin ~ to be about where do they get a Python to ask for the data.