Out of memory error when using queryset iterator

[lucasmoeskops]

What happened?

A search engine indexer module uses a queryset iterator to iterate over all the records to index. This sends a sql query to the database to fetch all data, but also to iterate over it and not to actually fetch all of this. Cachalot intercepts the request and seems to ignore the fact that it is going to iterate, instead fetching all the data from the table. This caused on out of memory error on the server.

What should've happened instead?

Cachalot should understand that the query was sent by an iterator and just let it query the database the way it did OR have support for this iteration process and get the right data from the cache somehow.

Steps to reproduce

OS -> Ubuntu
Django version -> 4.2.9
Database -> Postgres
django-cachalot -> 2.6.2

[danlamanna]

I've also run into this because cachalot wants to store the entire result set in local memory (and then in redis). I think it might make sense to be more opinionated around iterators. Given .iterator is pretty common in larger django projects, it's perhaps not ideal to have to wrap every block in with cachalot_disabled()....

@Andrew-Chen-Wang Would a PR making a modification similar to this be considered?

diff --git a/cachalot/monkey_patch.py b/cachalot/monkey_patch.py
index 37b2932..ebb7e59 100644
--- a/cachalot/monkey_patch.py
+++ b/cachalot/monkey_patch.py
@@ -1,4 +1,6 @@
+import logging
 import re
+import types
 from collections.abc import Iterable
 from functools import wraps
 from time import time
@@ -19,6 +21,7 @@ from .utils import (
     UncachableQuery, is_cachable, filter_cachable,
 )
 
+logger = logging.getLogger(__name__)
 
 WRITE_COMPILERS = (SQLInsertCompiler, SQLUpdateCompiler, SQLDeleteCompiler)
 
@@ -62,6 +65,13 @@ def _get_result_or_execute_query(execute_query_func, cache,
                 pass
 
     result = execute_query_func()
+
+    if result.__class__ == types.GeneratorType:
+        logger.info(
+            "Skipping caching of a generator due to potentially large result set."
+        )
+        return result
+
     if result.__class__ not in ITERABLES and isinstance(result, Iterable):
         result = list(result) 

[Andrew-Chen-Wang]

creating a new setting and a short FAQ of a result of this would be great