Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a utility to get a local directory that can be used as a persistent cache #1053

Open
llucax opened this issue Aug 27, 2024 · 0 comments
Open
Labels
part:core Affects the SDK core components (data structures, etc.) priority:❓ We need to figure out how soon this should be addressed type:enhancement New feature or enhancement visitble to users
Milestone

Comments

@llucax
Copy link
Contributor

llucax commented Aug 27, 2024

What's needed?

We need to be able to access some directory that can be used to store cache data that can persist application restarts or re-deployment.

Proposed solution

Add an utility function like def get_cache_dir(unique_cache_key: str) -> pathlib.Path. Where this utility function should live is still to be determine.

For now the SDK should have some initialization parameter called cache_dir_root or similar, and/or being able to get this configuration from some SDK_CACHE_DIR_ROOT (we could also have some fallback default using the XDG standard, like $XDG_CACHE_HOME).

Use cases

A typical use case is to cache data that actors or applications need for startup. For example, to do some predictions we need to collect data for some time and after enough data is collected, we can start predicting. If the application is restarted, this data collection period kicks in again, which can leave the app effectively not doing its job for a non trivial amount of time (warm-up period).

By using a cache, we can save the current status and pick up right were we left it when restarting, so we can skip this warm-up period.

Alternatives and workarounds

No response

Additional context

For cloud apps we might eventually want to use something else than a raw filesystem, like a object store (like AWS S3 compatible storage) for scalability reasons. If we move in that direction, this approach will not be enough, as apps/actors will need to act on a per-file basis, explicitly saving and retrieving from the object store.

@llucax llucax added priority:❓ We need to figure out how soon this should be addressed type:enhancement New feature or enhancement visitble to users part:core Affects the SDK core components (data structures, etc.) labels Aug 27, 2024
@llucax llucax added this to the Untriaged milestone Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
part:core Affects the SDK core components (data structures, etc.) priority:❓ We need to figure out how soon this should be addressed type:enhancement New feature or enhancement visitble to users
Projects
Status: To do
Development

No branches or pull requests

1 participant