Do not automatically cache models in temp dirs

[helena-intel]

Currently, CACHE_DIR for exported models is set to a temporary directory unique to the model export. This is not useful, the cache will never be used. It also can cause issues with large exported models on GPU (model loading crashes).

This PR disables automatic model caching if the model save directory is a temporary directory. That doesn't just affect exported models, automatic model caching is now also disabled if users manually save models in /tmp (or Windows equivalent) - but I do not think that that is an issue. Users can always enable model caching manually by setting CACHE_DIR.

I also aligned the cache dir for diffusion models with other models, so now all cache dirs are in a specific _cache directory (previously for stable diffusion the cache dir was the model dir).

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[helena-intel]

The tests failed at:
FAILED tests/openvino/test_modeling.py::OVModelForCausalLMIntegrationTest::test_compare_with_and_without_past_key_values - AssertionError: False is not true : With pkv latency: 336.185 ms, without pkv latency: 361.155 ms, speedup: 1.074
This test fails quite often and I disabled it in this PR. I tested this earlier, when this test first started failing and we reduced the SPEEDUP threshold, and back then I did see speedup when testing manually on Ice Lake Xeon.

[echarlaix]

thanks for the fix @helena-intel

[echarlaix]

what do you think about :

Suggested change

	if "CACHE_DIR" not in self.ov_config.keys() and not str(self.model_save_dir).startswith(gettempdir()):
	if "CACHE_DIR" not in self.ov_config.keys() and not isinstance(self.model_save_dir, TemporaryDirectory):

[helena-intel]

When I tested this, this evaluated to False, model_save_dir was a PosixPath, not a TemporaryDirectory.

[echarlaix]

could you share an example so that I can try to reproduce it ?

[helena-intel]

from tempfile import TemporaryDirectory
from optimum.intel.openvino import OVModelForSequenceClassification

model = OVModelForSequenceClassification.from_pretrained("hf-internal-testing/tiny-random-distilbert", export=True)
print(type(model.model_save_dir))
print(isinstance(model.model_save_dir, TemporaryDirectory))

(you can also add these prints to modeling_base.py)

[echarlaix]

I would prefer we keep the test and set SPEEDUP_CACHE to 1 (as inference when leveraging the pkv should not be slower than without )

[helena-intel]

I checked my logs for failed runs and this was honestly the first one:

But I will put it back if you want.

[echarlaix]

I see, doesn't it mean that there is a serious issue for model inference with use_cache=True ?

[helena-intel]

On the Github Actions runner (Skylake with 2 cores) that is probably. But I don't expect that that's common for inference in production. On more recent Xeon I don't see this issue, use_cache is faster than without.

[echarlaix]

we don't want to modify self.ov_config by adding a cache path specific to the encoder / decoder component so I think we should create a copy that we can modify instead

[helena-intel]

Thanks, I will fix that!