Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Immediate ingest #109

Closed
stranak opened this issue Jan 26, 2014 · 1 comment
Closed

Immediate ingest #109

stranak opened this issue Jan 26, 2014 · 1 comment

Comments

@stranak
Copy link

stranak commented Jan 26, 2014

It is possible (and we should) not to rely on the scheduler to run daemon that ingests submissions periodically, but instead trigger the ingestion directly by the action of submission.

It is obviously better solution, since it makes new submissions immediately visible and accessible. It is also how CERN library, Zenodo and other Invenio users do it. With a bit of luck the config will be in their repository: https://github.com/zenodo

@emanueldima
Copy link
Contributor

I have been working on this. The ingest process is not slow at all. I tested it twice, with 2 files, and I get the correct result page after about 5 seconds after I click the "Deposit" button. This is the behavior of the testing server, I can't test what is going on in the production environment.

From the code paths, I can speculate that delays happen when multiple uploads are submitted at the same time. All the queued bibXXX tasks/actions are queued (using the database, see schTASK) and they are executed sequentially. Apparently, bad things happen when tasks are run concurrently -- it seems that there are no fine grained locks for avoiding race conditions.

So, if several uploads are submitted at once, the first will get ingested immediately and the last will wait for all the others. Or, if the upload is done while any indexing/ranking/sorting service is running, it will have to wait.

The only improvement I see now is to increase the priority of the bibupload tasks, to be executed before indexing/sorting/ranking tasks. Other than that, the UI must change to tell the user that the record is being processed.

llehtine pushed a commit that referenced this issue Apr 3, 2014
fixed #109: faster ingest, proper message while waiting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants