Update readmes

README.md

@@ -1,15 +1,21 @@
 # blobfish
 
-Repo used in experimentation with Resource Description Framework metadata creation, automation, and querying
+## Summary
+This repository is a proof of concept to evaluate the usability of resource description framework (RDF) metadata in documenting extensible data pipelines. The data pipeline documented by this repository is the mirroring, transformation, and subsequent use of NOAA AORC gridded precipitation data for stochaistic storm transposition (SST) modeling.
 
-## AORC
-Resources related to creating an ontology used in documenting a data pipeline for creating hourly composites of AORC precipitation data
+## Pipeline Description
+This pipeline can be broken into 3 stages
 
-## PYRDF
-Resources used to translate python files to TTL files
+### Mirror
+The first stage is taking the AORC data from an FTP server, operated by NOAA, and putting it on s3. This is to allow for easier access and less potential network interruptions. This data is zipped into packages covering one month each and is partitioned by river forecast center (RFC).
 
-## CKAN
-Resources used to query TTL RDF files that have been uploaded to the Dewberry CKAN instance
+### Composite
+The second stage is unzipping the montly data into series of hourly netCDF data, aligning the datasets temporally, and merging the data to a contiguous coverage of the united states, rather than separated by RFC region.
 
-## SST
-Resources demonstrating potential applications of extensions on the DCAT-US vocabulary proposed as a standard for federal data documentation
+### Transposition
+The third stage is using the data now that it has been transformed into a more convenient format. Our use case is utilizing the precipitation data in SST hydrological modeling. This repo does not encompass the data processing, but rather consumes the metadata produced during this process and converts these to a compatible format for use as RDF
+
+## RDF Implementation
+With the exception of the source datasets, the metadata created by this repo is not RDF, but rather plain JSON (not JSON-LD, an RDF format). This is because this repo works in tandem with extensions created on the CKAN api specifically for parsing and serializing the uploaded metadata into an RDF format.
+
+These extensions can be found (here)[https://example.org]

blobfish/README.md

@@ -0,0 +1,6 @@
+## blobfish
+
+Repo used in experimentation with Resource Description Framework metadata creation, automation, and querying
+
+### /aorc
+Holds the scripts utilized in the creation of metadata POST requests which interact with CKAN to create DCAT-compliant dataset instances and catalogs

blobfish/aorc/README.md

@@ -1,40 +1,31 @@
 ## AORC
+Holds the main scripts as well as the utility scripts and classes that are used to upload metadata necessary for the creation of metadata to CKAN
 
-A collection of ontologies and scripts for generating mirrors for use in data pipelines to support the development of geospatial datasets used in risk and resilience studies.
+### /classes
+Holds classes used by individual processes as well as classes shared between processes in the AORC pipeline
 
+### /composite_utils
+Holds utility scripts used by the composite task of the AORC pipeline
 
----
+### /general_utils
+Holds general utility scripts used by multiple processes of the AORC pipeline
 
-#### Ontologies:
+### /mirror_utils
+Holds utility scripts used by the mirror task of the AORC pipeline
 
-[AORC](http://htmlpreview.github.io/?https://github.com/Dewberry/blobfish/blob/aorc/semantics/html/aorc/index.html): Analysis of Record for Calibration precipitation and temperature datasets.
+### /transposition_utils
+Holds utility scripts used by the transposition metadata collection task of the AORC pipeline
 
----
+### composite.py
+Main script for the composite creation task
+Responsible for querying the available mirror datasets from CKAN, aligning the data based on shared temporal coverage, and converting the monthly zipped data to hourly zarr format data, as well as collecting and uploading relevant metadata from this process
 
-#### Scripts:
+### const.py
+Holds constants used in the AORC pipeline, including URIs for data formats, RFC info, data portal URLs, etc.
 
-[AORC](./blobfish/aorc/):
- * [composite](./blobfish/aorc/composite.py) - Script for creating CONUS-level composite gridded datasets composed of mirrored AORC data which utilizes RDF metadata created during the mirroring process
- * [const](./blobfish/aorc/const.py) - Constants used in AORC processing or parsing
- * [parse_composite](./blobfish/aorc/parse_composite.py) - Script for parsing metadata from s3 objects of created composite gridded dataset to add onto existing RDF metadata created during the mirroring process in order to document the relationship between the mirrored datasets, the composited datasets, and the compositing process
+### mirror.py
+Main script for the mirror creation task
+Responsible for verifying the total available data from NOAA, its asynchronous acquistion, and its upload to s3, as well as collecting and uploading relevant metadata not only for the source data but also the mirror datasets created in the process
 
- ##### Setup
-
- To run the scripts in this repo, you should have a .env file in the same directory as this repo which has the following keys:
-
- ```
-AWS_ACCESS_KEY_ID=access_id_here
-AWS_SECRET_ACCESS_KEY=access_key_here
-AWS_DEFAULT_REGION=aws_region_here
-TAG=docker_image_tag
-HASH=docker_image_hash
- ```
-
- The workflow during development was to launch a docker container using the specified tag and hash from the docker hub image https://hub.docker.com/layers/njroberts/blobfish-python/ and use the container to run the scripts in the following sequence in order to both complete the composite process and create RDF TTL files documenting the metadata for the jobs:
-
- ```
-python -m blobfish.aorc.transfer
-python -m blobfish.aorc.parse_transfer
-python -m blobfish.aorc.composite
-python -m blobfish.aorc.parse_composite
- ```
+### transposition_meta.py
+Main script for collecting, parsing, and submitting metadata created during stochaistic storm transposition models to CKAN for serialization as RDF

semantics/README.md

@@ -0,0 +1,12 @@
+## semantics
+
+Holds AORC ontology and SHACL validation materials used to describe and validate RDF metadata created to document the AORC pipeline
+
+### /html
+Holds the HTML copy of the current AORC ontology
+
+### /rdf
+Holds the turtle (.ttl) copy of the current AORC ontology
+
+### /shacl
+Holds the validation rules for the AORC datasets created