During a team meeting I've mentioned that some metadata is still in Redis instead of Postgres. Looks like I was wrong, and this was (the last thing) fixed in Feb.

But there are still some things that are not in Redis but should be. This includes the list of matched files.

In the query_ursadb function, we first select files into ursadb iterator (by using the into iterator {query} statement in the query), and then in the run_yara_batch function we "pop" files from the iterator and run yara on them.

Instead, we should run a normal query, save all prefiltered files into the database, and then read unprocessed files from the database instead of from ursadb.

This should be a separate table (not Match) with just job Id and file path. It should work a bit like a task queue and after processing files should be removed.

In short, the roadmap:

• remove into iterator {query}; from query in ursad
• then the result is a list of files instead of an iterator. They must be saved into a new table in the database
• prepare a migration that creates this table
• finally, rework run_yara_batch such that it gets files to process from the database instead of by using ursadb pop
• also remove dead code (like ursadb.pop) later.