Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retry logic to Job Tag Update function #168

Open
ftadao opened this issue Dec 20, 2022 · 0 comments
Open

Add retry logic to Job Tag Update function #168

ftadao opened this issue Dec 20, 2022 · 0 comments

Comments

@ftadao
Copy link

ftadao commented Dec 20, 2022

Description

An Internal Server Error pops up whenever a large number of tag updates run parallel or sequentially.

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/project-1.0-py3.10.egg/XX/utils/workflow/__init__.py", line 930, in run
    start_stage, active_stage, ran_stages = self.setup_continuation(
  File "/usr/local/lib/python3.10/site-packages/project-1.0-py3.10.egg/XX/utils/workflow/__init__.py", line 667, in setup_continuation
    self._discard_jobs(start_stage, ran_stages)
  File "/usr/local/lib/python3.10/site-packages/project-1.0-py3.10.egg/XX/utils/workflow/__init__.py", line 705, in _discard_jobs
    self.get_job(jobinfo["key"]).update_tags(
  File "/usr/local/lib/python3.10/site-packages/scrapinghub/client/[jobs.py](http://jobs.py/)", line 503, in update_tags
    self._client._connection._post('jobs_update', 'json', params)
  File "/usr/local/lib/python3.10/site-packages/scrapinghub/[legacy.py](http://legacy.py/)", line 120, in _post
    return self._request(url, params, headers, format, raw, files)
  File "/usr/local/lib/python3.10/site-packages/scrapinghub/client/[exceptions.py](http://exceptions.py/)", line 98, in wrapped
    raise ServerError(http_error=exc)
scrapinghub.client.exceptions.ServerError: Internal server error

This is not a problem if you are doing updates for a couple of jobs, but if you want to mass update this error will pop up eventually.

Adding adaptable retry logic to the update_tags function through that ServerError exception would make it easier to debug and implement large-scale workflows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant