Go through DMCI workdir file-accumulation

[johtoblan]

No description provided.

[johtoblan]

Might be caused by postgis being down, should there be an automatic iteration of files in workdir older than X or should they just be removed?

[mortenwh]

We need to assure that they are ingested, then we can delete them

[mortenwh]

Test:

• Push valid xml to dmci
• Let file dist succeed but pycsw and solr dists fail
• In this case, the file should be removed from workdir but added to rejected dir
• Let all dists succeed
• Then, the file should not be in workdir and not in rejected
• Let all dists fail
• The file should be in rejected and not in workdir

If all tests succeed, the reason for the build up of data in the work dir must be that the container has crashed in the middle of an ingestion process. In that case, dmci should pick up the files in workdir and re-run the process. Currently, dmci only acts on API actions (i.e., a file is pushed to the API). DMCI currently does not check the work dir.

app.py line 140 to 165 should be a function: return tools.insert_update(...)

Then a tool should be added that regularly checks the workdir and runs insert_update. If all works as expected, this should empty the workdir.

[mortenwh]

The workdir still often contains files, and seems to not be properly cleaned up. Maybe related to some issues in solr? See #240

[mortenwh]

Is this related to #237 and #240? @shamlymajeed - could you follow up this?