Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimizations for api_explorer_get_top_proxies (#51) #60

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Optimizations for api_explorer_get_top_proxies (#51) #60

wants to merge 2 commits into from

Conversation

Zapata
Copy link
Member

@Zapata Zapata commented Aug 3, 2019

  • add start and limit parameters to /top_proxy
  • add /proxy_count
  • parallelize ES accounts retrieval

@Zapata
Optimizations for api_explorer_get_top_proxies (bitshares#51)
b65557d 
  - add start and limit parameters to /top_proxy
 - add /proxy_count
 - parallelize ES accounts retrieval
@Zapata
Copy link
Member Author

Zapata commented Aug 3, 2019

To fix #51

@Zapata Zapata mentioned this pull request Aug 3, 2019
Open
@oxarbitrage oxarbitrage self-requested a review August 23, 2019 14:08
@oxarbitrage
Copy link
Member

When i try to use any of the involved calls in localhost i am getting the following error in the console after about a minute:

(wrappers_env) alfredo@alfredo-System-Product-Name:~/PycharmProjects/pull60/bitshares-explorer-api$ flask run --host=0.0.0.0
 * Tip: There are .env or .flaskenv files present. Do "pip install python-dotenv" to use them.
 * Serving Flask app "app.py"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
INFO:werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
WARNING:elasticsearch:GET https://elasticsearch.bitshares-kibana.info:443/objects-account/_search?scroll=5m&size=1000 [status:N/A request:0.088s]
Traceback (most recent call last):
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 217, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 379, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 772, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 659, in read
    v = self._sslobj.read(len)
SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1941)
WARNING:elasticsearch:GET https://elasticsearch.bitshares-kibana.info:443/objects-account/_search?scroll=5m&size=1000 [status:N/A request:0.277s]
Traceback (most recent call last):
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 217, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 379, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 772, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 659, in read
    v = self._sslobj.read(len)
SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1941)
29 accounts to load via websocket
ERROR:bitshares-explorer-api:Exception on /top_proxies [GET]
Traceback (most recent call last):
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/connexion/decorators/decorator.py", line 48, in wrapper
    response = function(request)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/connexion/decorators/uri_parsing.py", line 143, in wrapper
    response = function(request)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/connexion/decorators/validation.py", line 347, in wrapper
    return function(request)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/connexion/decorators/parameter.py", line 126, in wrapper
    return function(**kwargs)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/api/explorer.py", line 414, in get_top_proxies
    proxies = _get_all_proxies()
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask_caching/__init__.py", line 795, in decorated_function
    rv = f(*args, **kwargs)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/api/explorer.py", line 392, in _get_all_proxies
    holders = _get_holders()
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/flask_caching/__init__.py", line 795, in decorated_function
    rv = f(*args, **kwargs)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/api/explorer.py", line 457, in _get_holders
    accounts = _load_missing_accounts_via_ws(account_ids, accounts)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/api/explorer.py", line 443, in _load_missing_accounts_via_ws
    missing_accounts = _get_accounts_by_chunks_via_ws(accounts_ids_to_load)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/api/explorer.py", line 434, in _get_accounts_by_chunks_via_ws
    accounts = bitshares_ws_client.request('database', 'get_accounts', [ account_ids[i:i+chunk_size] ])
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/services/bitshares_websocket_client.py", line 24, in request
    return self._safe_request(api, method_name, params)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/services/bitshares_websocket_client.py", line 43, in _safe_request
    self.ws.send(request_string)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/websocket/_core.py", line 253, in send
    return self.send_frame(frame)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/websocket/_core.py", line 278, in send_frame
    l = self._send(data)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/websocket/_core.py", line 448, in _send
    return send(self.sock, data)
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/websocket/_socket.py", line 157, in send
    return _send()
  File "/home/alfredo/PycharmProjects/pull60/bitshares-explorer-api/wrappers_env/local/lib/python2.7/site-packages/websocket/_socket.py", line 139, in _send
    return sock.send(data)
  File "/usr/lib/python2.7/ssl.py", line 725, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe
INFO:werkzeug:127.0.0.1 - - [23/Aug/2019 19:09:02] "GET /top_proxies?limit=10 HTTP/1.1" 500 -

Other ES calls work fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@oxarbitrage oxarbitrage Awaiting requested review from oxarbitrage

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Zapata @oxarbitrage