This repository has been archived by the owner on Jul 19, 2018. It is now read-only.
HcfMiddleware sometimes gives fewer links than requested.
#58
Comments
starrify
added a commit
to starrify/scrapylib
that referenced
this issue
Mar 26, 2015
starrify
added a commit
to starrify/scrapylib
that referenced
this issue
Mar 26, 2015
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Currently a call to
hubstorage.frontier.Frontier.readreturns a generator which contains 100 batches, each of which contains 100 links. That makes 100x100=10000 links.When
HS_MAX_LINKSis not exceeding 10000, things look good; But when it's larger than 10000, e.g. specifyingHS_MAX_LINKS = 20000, only 10000 links are generated by theHcfMiddleware.The text was updated successfully, but these errors were encountered: