Skip to content

Procedure to contribute model output to the portal

Jump to bottom
Rich Signell edited this page Aug 21, 2020 · 50 revisions

1. Move your files to the usgs/Projects folder

Locate the usgs/Projects folder on the server and create a folder for your model output release. Move your model output to be published. See this example for types of files required for reproducibility.

  • input
  • his/avg folder
  • forcings folders
  • initial
  • grid
  • ncml file (see next step)

2. Add datasets to an existing THREDDS Data Server

For each simulation that appears on the portal, you need:

  1. An NcML file that starts with “00_dir” (e.g. “00_dir_roms.ncml”) to aggregate, standardize and describe the dataset
  2. NetCDF output files in a directory

For ROMS datasets, you can use a python script to generate the NcML from a YAML text file as input. Editing the YAML is simpler than editing the NcML directly because you don't have to edit attributes for multiple variables. YAML file includes the following sections

  • id
  • title
  • summary
  • project
  • contacts
  • variables (to display on the portal)

and more to provide the required metadata for the model output. Here's the link to the yaml2ncml page, and an example YAML file used to create a ROMS NcML file. You can use this YAML file as a template to populate your own YAML file.

Note:

  1. Use white space instead of tabs to indent the fields in your YAML file.
  2. To be picked up by the crawler, the name of the resulting NcML file has to start 00_dir and end with .ncml, like 00_dir_roms.ncml.
  3. Make sure the variables you want to show up in the Portal do not have a display attribute set to False.

Running yaml2ncml remotely

You can install yaml2ncml locally following the instructions on the yaml2ncml page. Alternatively, you can do this remotely following the steps here.

yaml2ncml is already installed on geoport and sand as part of the ioos python environment. To make this your default environment when you login to sand and/or geoport, add this line to your .bashrc:

export PATH=/home/usgs/miniconda/bin:$PATH && source activate IOOS

Then try logging out and back in, and you should see something like this:

discarding /home/usgs/miniconda/bin from PATH
prepending /home/usgs/miniconda/envs/IOOS/bin to PATH
(ioos)rsignell@sand:~$

which means that you are using the ioos environment, and you can run the yaml2ncml command.

3. Add your simulation to the catalog of simulations

This is an extra step to follow if for some reason you cannot place your files under the usgs/Projects folder.

Add the catalog that your NcML file is in to the list of thredds catalogs harvested by the catalog database. To do this, click the pen icon to edit, make your changes, and then click the button at the bottom of the page to submit a pull request.

Once your pull request is accepted by the administrator, the python script on geoport will be updated:

ssh gamone.whoi.edu
sudo su - rsignell
cd /opt/docker/harvest/usgs-cmg-portal
git pull

The python script currently runs each hour at 5 minutes past the hour. If everything is successful, there should be new ISO metadata records that appear in /opt/docker/pycsw/force/iso_records, and also these records should be harvested and become available in the pycsw database via the CSW service by 15 minutes past the hour.

Clone this wiki locally