Object Store Caching layer

[ion-elgreco]

Description

Use Case
We currently in the worst case read them both in full twice, we can however cache the first round of reading, which will reduce the second round of getting files from an object store

Related Issue(s)
See discussion here #2760

[scovich]

General I/O caching support is engine's responsibility, not kernel's. Kernel just needs to not get in the way of an engine to implement such caching.

To that end, it would be more helpful to identify specific ways kernel blocks or impedes the engine from doing the kinds of caching it would like to do.

For example, the JsonHandler and ParquetHandler traits provided by the engine should make ideal hook points for caching the results of file reads. The engine currently performs file writes on its own, without kernel involvement, and so it would be up to engine to introduce appropriate caching there as it sees fit.

On the other hand, I know @roeap has been considering higher level caching approaches in delta-rs that would capture the result of log replay rather than going back to the underlying individual files at all. As we identify such optimization opportunities, kernel APIs may need to adapt if the engine currently lacks appropriate hook points.

[scovich]

Heh. I missed that this was a delta-rs issue, not delta-kernel-rs!

That said, if this work does identify gaps in kernel APIs, please do file enhancement issues against delta-kernel-rs.