Batching, caching, checkpointing, probing.

[tbenthompson]

These are four capabilities that would be good to advance further over time. I don't think they make sense to work on much in isolation from a specific problem at the moment, but every time we do a big problem, we'll hopefully get better at these things. There's more to do on Lewis, so some of this will improve as a result.

We can split these out into their own issues if needed.

Batching: See here. Todo:

• Add a test for multidimensional batching and for N % batch_size == 0.
• Improve the batch API.
• Use this for another problem and use that experience to clean up the API.

Caching: Often, a model will require constructing some results as a prerequisite to running simulations. Berry needs the lookup table. Lewis needs the interpolation tables. It's nice to automatically cache these objects and then load them from disk later if they are needed again. Currently, a rough approximation of this behavior is done here for Lewis. Todo:

• generalized interface and API. Store the data in the tile database?
• that ensures validity by having partial insight or control over the table construction process too. This means the caching layer can make fully sure that the inputs are the same and thus the cached result can safely be loaded.
• perhaps even store the construction code itself so that we can confirm the code hasn't changed? not sure about this one.

Checkpointing: Also a very rough capability. Currently just lives in the adagrid/tuning code where we save the state at each iteration. Upgrades:

• "exponential checkpointing" - checkpoint every iteration for 10 iterations back in time, then every 10th iteration for 100 iterations back, then every 100th for 1000 iterations back... this minimizes disk space requirements while allowing us to restart the computation at various different points in time.
• checkpointing is partially superceded by the tile database since that provides a running checkpoint.
• Cleaner interface, perhaps controlled by AdaState.
• longer term: maybe partial checkpointing logic so that we don't need to save the full state at each iteration, just the updated portion of the state.

Probing: See #96 and #63

[tbenthompson]

With the duckdb backend, we should ignore further checkpointing. The data is already on disk and there's nothing more to do unless we wanted to automate export to S3 or something. But it seems better to just move jobs that need checkpointing into our cloud setup with a different database backend. So the remainder of the checkpointing effort here is subsumed by setting up a proper db server backend!

[tbenthompson]

Most of the remaining work is in #173.