Nightly or weekly end-to-end tester and benchmarker

[tbenthompson]

https://stackoverflow.com/questions/63014786/how-to-schedule-a-github-actions-nightly-build-but-run-it-only-when-there-where

see also:
#15
#75

[tbenthompson]

start out with just the 2 core machines, and then upgrade to a larger machine if it seems like a useful thing to do.

[tbenthompson]

what i'd like to run:

• all the example notebooks!

how:

run them through the ipython interface so that we can "%run notebook.ipynb"

from IPython import get_ipython
get_ipython().run_line_magic("run", "research/design/ztest.ipynb");
print(locals())
print(globals())

then we can access variables from the notebook and check their values.

then we can also inject some benchmarking code use line_profiler? or something else? and see how long each step takes to run. look into external examples of uploading this data somewhere so that we can look at benchmark evolution over time.

[tbenthompson]

ipython -m pytest tests/

will run tests through ipython so that we have access to the IPython shell.

another issue:

• turn off plotting or direct it to a file?

[tbenthompson]

• modal cron job?

[tbenthompson]

Running this on GitHub-hosted actions runners on a daily or weekly basis would be wonderful. It seems like GPU-enabled actions runners are in beta?

github/roadmap#505
https://support.github.com/ticket/personal/0/1887823

[tbenthompson]

This might be more of a 2023 task but I was learning about stuff and figured it would be good to write down my notes.

inspiration: https://pv.github.io/numpy-bench/

learn about ASV: https://asv.readthedocs.io/en/stable/using.html
this is partially blocked by needing a static site to share the results: #14
also, we need to figure out to run on the same machine:
one approach is to run on github actions and not worry about using the same machine each time. just compare main vs PR branch as some extra information on any PR. this has the downside of not tracking changes over time. airspeed-velocity/asv#796
figure out what numpy/scipy do: https://pv.github.io/numpy-bench/
managed hosted server? stick a box in someone's house?
Gary worked on some of these issues so we should ask him what he did so far.