Automate ioos by the numbers generation

[MathewBiddle]

Currently the IOOS by the numbers metrics are collected on a manual basis by running the Jupyter Notebook. The process should be automatically ran, using GH-Actions similar to the GTS metrics, on an annual? quarterly? basis.

[MathewBiddle]

Steps:

• Translate notebook to python script
• determine frequency
• build GitHub Action

[MathewBiddle]

@ocefpaf do you have an example of running a jupyter notebook in a GitHub Action?

[ocefpaf]

@ocefpaf do you have an example of running a jupyter notebook in a GitHub Action?

There are many ways to do that but I prefer a single call to nbconvert. I usually save the notebook without any output and "convert" it to a notebook with the outputs.

[MathewBiddle]

I'd like to follow the single call to nbconvert to run the notebook and be done. Remove the standalone python script. Here's the addition to run the notebook.

      - name: Setup Conda
        uses: s-weigand/setup-conda@v1
        with:
          activate-conda: false
          conda-channels: conda-forge
      - name: Build environment
        shell: bash -l {0}
        run: |
          conda env create -f environment.yml
      - name: Execute Notebook
        run: |
          source activate ioos-btn
          jupyter nbconvert --to notebook --execute IOOS_BTN.ipynb --output=IOOS_BTN.ipynb

[ocefpaf]

I never tried to overwrite the same notebook, need to test it first. The rest looks good. BTW, are you saving some results and publishing? What is the format? You can probably create a table or a simple page and publish as gh-pages to keep the notebook untouched.

[MathewBiddle]

BTW, are you saving some results and publishing? What is the format?

Yeah, I'm saving the results as a csv file, similar to what I've done for the GTS metrics [1] which get collected quarterly with my GH Action [2].

[1] https://github.com/MathewBiddle/ioos_by_the_numbers/tree/main/gts
[2] https://github.com/MathewBiddle/ioos_by_the_numbers/blob/main/.github/workflows/metrics.yml

We're also looking to do something similar for the NGDAC metrics.

So, if you have ideas on how to make this more efficient/prettier, I'm all ears.

We can chat on Thursday about it too.

[ocefpaf]

So, if you have ideas on how to make this more efficient/prettier, I'm all ears.

More efficient? No, what you are doing is the best as far as I know. Prettier? Yes. We should keep the csv, b/c that is more flexible, but save an HTML and table to post as gh-pages. That is a single line with pandas: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_html.html.

[MathewBiddle]

I've been considering moving IOOS_BTN.ipynb [1] to a series of functions, again, for each of the metrics and wrap that all in a standalone python script. That way we could call to something like:

import ioos_metrics

df_btn = ioos_metrics.btn() # by the numbers as a df
df_atn_gts = ioos_metrics.atn_gts() # atn gts metrics as a df

ngdac_glider_days = ioos_metrics.ngdac.glider_days() # give back the number of glider days. Maybe expand to accept start/end

The Jupyter Notebook it becoming a little unwieldly at this point IMO.

[1] - https://github.com/MathewBiddle/ioos_by_the_numbers/blob/main/IOOS_BTN.ipynb

[MathewBiddle]

duplicate?

• Create GH Action to generate new IOOS by the Numbers statistics #98