parallelize upload

[nherment]

can we have a --parallel <integer> option to run the import in parallel (children processes) ?

Trello Card

[mcollina]

Is the import CPU-bound at them moment, or is it I/O-bound? Is the CPU running at 100% when importing?
In that case, we do not need children processes, we just need to do more than one seneca call at the same time, as it should be I/O-bound at the moment.

The best way to handle it is to have multiple insert running in parallel, i.e. we forward up to N calls to seneca at the same time. I think it's feasible.

Do you have an ETA for this? This week is incredibly tough for me.

[nherment]

The import is IO bound and I suspect it is IO protocol bound :) --> I suspect the latency to be killing us, hence the child processes 'workaround'.
We probably make too many requests but we can't really batch either, yet. At some point I think the seneca DB layers should provide batching.

no rush for this.