Create demo that outlines how to import their own data from CSV into the ENTR warehouse and run the example OpenOA notebooks from the data.

[jordanperr]

Definition of Done

Done when we have the workflow documented and tested for how to do a trial run of ENTR on the runtime image using CSV data stakeholders can upload.

Problem/Context

A demonstration of the ENTR modeling guidelines all the way through to seeing analytics will make this "real" for interested stakeholders.

User Story/Stories

As a an interested stakeholder I want a sandbox demonstration of ENTR without running it on my production infrastructure (i.e. without risk) so that I can understand its value; synthesize how ENTR could be deployed in production at my company beyond just the docs; and have artifacts to help me, my peers, and management make a decision on whether to adopt it or not.

Design/Solution Ideas

• need to define the use cases to demo so we can clearly define the data requirements

Discovery Work Outline/Notes

• [ ]

[ejsimley]

Ideas from 4/12/2022

• Use Spark to read in csv files
• User specifies column mapping to ENTR column names
• dbt converts wide tables to ~4 column tables in standard ENTR format

[ejsimley]

Reanalysis downloading notes from 3/28/2022 meeting:

• The user will be responsible for downloading reanalysis data as csv files from PlanetOS using the OpenOA toolkit (or another method if they prefer) when importing their operational data to the warehouse rather than when OpenOA is used when analyzing data
• We can write a script to create a warehouse table from the downloaded csv files
• We can provide documentation and an example showing how to download reanalysis data using the OpenOA PlanetOS toolkit

Reanalysis data importing notes from 4/4/2022

• User will download reanalysis files and drop into specific folder. Or automate by looping through plant table and downloading files for all plants
• dbt workflow to add downloaded csv files to seed table
• Eric to send Lewis example python code to download reanalysis data using OpenOA
• For now, we'll create a separate table for each reanalysis product

[lewisarmistead]

Examples that we would need to support: CP's data from Zenodo
https://zenodo.org/record/5946808
https://zenodo.org/record/5841834

[ejsimley]

To avoid memory issues with dbt seed, we decided to try reading from csv using SQL when materializing warehouse tables