How to clear data cache after report execution?

[NicolasPerritaz]

Hello,

I've tried to enable data cache to speed up XML data access (workaround for issue #762):

taskRun.getAppContext().put(DataEngine.QUERY_EXECUTION_SESSION_ID, cacheName);
taskRun.getAppContext().put(DataEngine.MEMORY_DATA_SET_CACHE, "10000");

However, I noticed that the cache is never cleared after each report execution, which can cause issues, because in some cases I get obsolete data in my report.

I've tried to use a different cache name for each execution, but as the old caches are never cleared I ended up with an OutOfMemory error.

I've tried to force a clear cache before each execution, before running the run task, using:

CacheMapManager.clearCache(Stream.of(cacheName).collect(Collectors.toSet()));

but this causes the report generation to freeze on this line, apparently because of a deadlock. Is there a proper way to clear the cache?

[hvbtup]

I cannot help, as I'm not using XML data sets.

There are other memory parameters as well that probably should be considered, e.g. DataEngine.MEMORY_BUFFER_SIZE (see https://www.eclipse.org/forums/index.php/t/261934/). The parameters are described in the source file DataEngine.java, but I don't understand how they work.

Anyway, there is a related "issue" for JDBC data sets, which I think is worth noting here:
For JDBC data, caching is enabled by default and the cached data is stored in the file system, and I don't think that I explicitly set any of the parameters.

The way I handle these files to prevent running out of disk space in the long run:

I am not using the web viewer, Instead I have integrated BIRT into a java program. Several of these programs run at the same time, controlled by a central controller program (written in Python).
The controller takes care of starting and stopping the Java processes.
Each Java (=BIRT) program instance uses a unique temporary directory (which the controller creates and sets TEMP and TMP environment variables and the java.io.tmpdir system property for the Java process. A java process handles only a limited amount of reports before the controller forces it to shutdown. Then the controller removes the temporary directory.

[hvbtup]

Hmm... I see there is a method clearCache in the source of DataEngine.java.
Maybe it is possible to call this method for the dataset(s) of the report at the end of each report (e.g. by adding a hidden label at the end of the report and call the method in its onCreate event (depending on how you are using the report).

[NicolasPerritaz]

Yes I was looking for another way to clear the cache using the DataEngine, but I didn't find a way to access the DataEngine object in a Event Handler from the reportContext.

[hvbtup]

Regarding "the cache is never cleared after each report execution, which can cause issues, because in some cases I get obsolete data in my report":

I know this from scripted data sets.

The caching mechanism AFAIK works like this:
If the same dataset (in the same report) is executed again with the same dataset parameters, then the cached results will be used.

Thus, if you want to reuse a dataset, make sure that if the results are depending on a variable X, then X must be a dataset parameter.
In other words: Make sure that all dependencies are specified explicitly.

[NicolasPerritaz]

I think it would have the same effect as creating a new cache session for each report execution, with the same issue: the cache grows until filling up all available memory or finally getting cleared by the CacheIDPurgeTimeTask which is scheduled by CacheIDFetcher to run every hour:

private static long idleTime = 3600 * 1000;

this.activeCacheIDs = new java.util.concurrent.ConcurrentHashMap<String, Long>( );
Timer timer = new Timer( true );
TimerTask task = new CacheIDPurgeTimeTask( );
timer.schedule( task, 0, idleTime );

Perhaps a solution would be to parameterize the idle time to allow clearing the cache more frequently depending on the use case.