Update catalog gets stuck at 96%

[seirl]

Is it maybe linked to the HTTP 523 errors I see in the logs?

Unhandled exception in thread started by <bound method UpdateCatalogWorker.retrievePartialListOfAddons of <waitdlg.UpdateCatalogWorker object at 0x7f9e2ddfdc18>>
Traceback (most recent call last):
  File "/opt/lcurse/modules/waitdlg.py", line 294, in retrievePartialListOfAddons
    response = self.opener.open("http://www.curse.com/addons/wow?page=%d" % (page))
  File "/usr/lib/python3.4/urllib/request.py", line 470, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 580, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 502, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 442, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 685, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 470, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 580, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 508, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 442, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 588, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 523: Origin Unreachable

[seirl]

Since I can't provide you with a version information (there is none):

% md5sum /usr/bin/lcurse :(
149202d995e804756d9d99ab954d6fab /usr/bin/lcurse

[ephraim]

git revision is always the best version regarding a git repository.
So, which revision do you use? Hopefully the latest!

[seirl]

I'm using the PPA, actually. I can try with the repository if the ppa is not up to date.

[ephraim]

Which PPA :). I don't know anything about an PPA.

[seirl]

Oh okay, I'll install the git version then.

[seirl]

I can confirm it does that on the latest commit in master (d8e02ee).

[comprofix]

I am getting the same but gets stuck at 99%

master d8e02ee fix updateCatalog to reflect recent changes of curse page- second part

markup_type=markup_type))
Unhandled exception in thread started by >
Traceback (most recent call last):
  File "/usr/lib/python3.5/urllib/request.py", line 1240, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 911, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 854, in send
    self.connect()
  File "/usr/lib/python3.5/http/client.py", line 826, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib/python3.5/socket.py", line 693, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.5/socket.py", line 732, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/moe/projects/lcurse/modules/waitdlg.py", line 294, in retrievePartialListOfAddons
    response = self.opener.open("http://www.curse.com/addons/wow?page=%d" % (page))
  File "/usr/lib/python3.5/urllib/request.py", line 465, in open
    response = self._open(req, data)
  File "/usr/lib/python3.5/urllib/request.py", line 483, in _open
    '_open', req)
  File "/usr/lib/python3.5/urllib/request.py", line 443, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 1268, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.5/urllib/request.py", line 1242, in do_open
    raise URLError(err)
urllib.error.URLError: 

[seirl]

I debuggued this problem and made a patch but then everything was still crashing everywhere when clicking on stuff and downloading add-ons. I advise you to download manually your addons while this project is broken.

[ephraim]

In both exceptions, it looks like the connection to curse.com got disconnected:
from first exception quote: urllib.error.HTTPError: HTTP Error 523: Origin Unreachable
from second exception quote: socket.gaierror: [Errno -2] Name or service not known

So I can't really reproduce it, as update catalog works here.

Does it get stuck all the time?

[comprofix]

No not happening all the time. For now it is downloading the catalogue successfully.

retrieved list of addons: 4083

[seirl]

Even if we lose the connection, not having anything that catches an exception to retry is not a sane behavior.

[MathyV]

Try with my pull request #18 if you experience problems again. I can reproduce it easily (perhaps because of my location?) and it is solved for me now.

[snje1987]

I also get this problem with the latest version. it's caused by the opener.open(str(url)) do not have a timeout. My internet connect is not so stable, sometimes this will cause a infinitely wait.

I tried to add a timeout there, but in the function OpenWithRetry, after maxcount times of retry, it will raise a Exception, which was not catched in function retrievePartialListOfAddons, finally terminate the working thread.