Set up a benchmarking suite (+ slow end-to-end tests)

[tbenthompson]

This might be more of a 2023 task but I was learning about stuff and figured it would be good to write down my notes.

inspiration: https://pv.github.io/numpy-bench/

• learn about ASV: https://asv.readthedocs.io/en/stable/using.html
• this is partially blocked by needing a static site to share the results: Set up both internal and external static sites for hosting tutorials/docs/other info. #14
• also, we need to figure out to run on the same machine:
  • one approach is to run on github actions and not worry about using the same machine each time. just compare main vs PR branch as some extra information on any PR. this has the downside of not tracking changes over time. Existing sample/recipe for integrating asv with github airspeed-velocity/asv#796
  • figure out what numpy/scipy do: https://pv.github.io/numpy-bench/
  • managed hosted server? stick a box in someone's house?

Gary worked on some of these issues so we should ask him what he did so far.

[tbenthompson]

I think it could make a lot of sense to integrate this with end-to-end testing: #75

[tbenthompson]

Running this on GitHub-hosted actions runners on a daily or weekly basis would be wonderful. It seems like GPU-enabled actions runners are in beta?

github/roadmap#505
https://support.github.com/ticket/personal/0/1887823

[tbenthompson]

closed in favor of #75