Fully dynamic updates of run["cores"] and run["workers"] (RFC) #2010

vdbergh · 2024-05-14T16:53:57Z

We have everywhere replaced

task["active"] = False

by

set_inactive(task_id, run)

which does the required bookkeeping under the task_lock.

The intention is that the task_lock will only be held for a very short time. So in request_task we only grab it at the moment when we actually add a task to the run (currently it is held during the whole invocation of request_task).

If it works then this PR can serve as a model for other types of bookkeeping like the number of committed games.

Other change:

We start the cache timer in __init__.py at application start up instead of during the first invocation of buffer().

server/fishtest/rundb.py

vdbergh · 2024-05-16T13:49:44Z

I rebased this and took the opportunity to strengthen the comment about the semaphore. I hope it is ok.

vdbergh · 2024-05-16T13:52:42Z

The fact that the semaphore should be strictly less than the number of threads can in principle be verified at start up.

I am revisiting the idea of parsing the config file (development.ini or production.ini) at start up, but not in rundb.py (that was bad) but in __init__.py.

ppigazzini · 2024-05-16T15:33:02Z

DEV running with the PR

vdbergh · 2024-05-16T15:36:40Z

@ppigazzini

When I quit pserve with ^C I need to press ^C twice and then the second time I get

Exception ignored in: <module 'threading' from '/home/fishtest/.pyenv/versions/3.10.8/lib/python3.10/threading.py'>
Traceback (most recent call last):
  File "/home/fishtest/.pyenv/versions/3.10.8/lib/python3.10/threading.py", line 1567, in _shutdown
    lock.acquire()
  File "/home/fishtest/fishtest/server/fishtest/rundb.py", line 356, in exit_run
    sys.exit(0)
SystemExit: 0

This does not happen with master. Do you perhaps know what might be the cause of this?

vdbergh · 2024-05-16T15:54:58Z

I think the error message points to an unreleased lock but I don't really see how that's possible. I will think more.

ppigazzini · 2024-05-16T15:55:28Z

No idea but I confirm that there is an issue, sudo systemctl restart fishtest@{6543..6545} doesn't work:

May 16 17:50:59 dfts-0 systemd[1]: Stopping Fishtest Server port 6543...
May 16 17:50:59 dfts-0 pserve[3350]: flush
May 16 17:50:59 dfts-0 pserve[3350]: .done
May 16 17:52:29 dfts-0 systemd[1]: [email protected]: State 'stop-sigterm' timed out. Killing.
May 16 17:52:29 dfts-0 systemd[1]: [email protected]: Killing process 3350 (pserve) with signal SIGKILL.
May 16 17:52:29 dfts-0 systemd[1]: [email protected]: Killing process 3381 (pserve) with signal SIGKILL.
May 16 17:52:29 dfts-0 systemd[1]: [email protected]: Main process exited, code=killed, status=9/KILL
May 16 17:52:29 dfts-0 systemd[1]: [email protected]: Failed with result 'timeout'.
May 16 17:52:29 dfts-0 systemd[1]: Stopped Fishtest Server port 6543.
May 16 17:53:29 dfts-0 systemd[1]: Started Fishtest Server port 6543

vdbergh · 2024-05-16T15:56:24Z

Ok thanks. That's something that needs to be solved.

ppigazzini · 2024-05-16T15:56:48Z

The schedule scripts don't stop as well:

  835 root      20       35700   1152    932 S        0.0   0:14.67  `- /usr/sbin/cron -f
14467 root      20       61116   3088   2660 S        0.1                `- /usr/sbin/CRON -f
14468 usr00     20        4636    860    792 S        0.0                    `- /bin/sh -c /usr/bin/nice -n 10 /usr/bin/cpulimit -l 50 -f -m -- ${VENV}/bin/python3 ${UPATH}/delta_update_users.py
14469 usr00     30  10   82528    860    772 S   0.3  0.0   0:18.69              `- /usr/bin/cpulimit -l 50 -f -m -- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_upda+
14470 usr00     30  10 1022124 200036  40308 S        3.9   0:29.75                  `- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_update_users.py
16289 root      20       61116   3088   2660 S        0.1                `- /usr/sbin/CRON -f
16290 usr00     20        4636    812    748 S        0.0                    `- /bin/sh -c /usr/bin/nice -n 10 /usr/bin/cpulimit -l 50 -f -m -- ${VENV}/bin/python3 ${UPATH}/delta_update_users.py
16291 usr00     30  10   82528    856    768 S   0.3  0.0   0:15.68              `- /usr/bin/cpulimit -l 50 -f -m -- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_upda+
16292 usr00     30  10 1025196 203236  40700 S        4.0   0:31.82                  `- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_update_users.py
19005 root      20       61116   3088   2660 S        0.1                `- /usr/sbin/CRON -f
19006 usr00     20        4636    784    720 S        0.0                    `- /bin/sh -c /usr/bin/nice -n 10 /usr/bin/cpulimit -l 50 -f -m -- ${VENV}/bin/python3 ${UPATH}/delta_update_users.py
19007 usr00     30  10   82528    856    772 S   0.3  0.0   0:12.16              `- /usr/bin/cpulimit -l 50 -f -m -- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_upda+
19008 usr00     30  10 1015964 194008  40676 S        3.8   0:26.30                  `- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_update_users.py
22617 root      20       61116   3088   2660 S        0.1                `- /usr/sbin/CRON -f
22618 usr00     20        4636    812    748 S        0.0                    `- /bin/sh -c /usr/bin/nice -n 10 /usr/bin/cpulimit -l 50 -f -m -- ${VENV}/bin/python3 ${UPATH}/delta_update_users.py
22619 usr00     30  10   82528    896    808 S   0.3  0.0   0:09.11              `- /usr/bin/cpulimit -l 50 -f -m -- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_upda+
22620 usr00     30  10 1018008 195696  40124 S        3.9   0:27.26                  `- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_update_users.py
27139 root      20       61116   3088   2660 S        0.1                `- /usr/sbin/CRON -f
27140 usr00     20        4636    860    796 S        0.0                    `- /bin/sh -c /usr/bin/nice -n 10 /usr/bin/cpulimit -l 50 -f -m -- ${VENV}/bin/python3 ${UPATH}/delta_update_users.py
27141 usr00     30  10   82528    856    768 S   0.3  0.0   0:05.75              `- /usr/bin/cpulimit -l 50 -f -m -- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_upda+
27142 usr00     30  10 1022124 200072  40336 S        3.9   0:26.16                  `- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_update_users.py
32562 root      20       61116   3088   2660 S        0.1                `- /usr/sbin/CRON -f
32563 usr00     20        4636    872    808 S        0.0                    `- /bin/sh -c /usr/bin/nice -n 10 /usr/bin/cpulimit -l 50 -f -m -- ${VENV}/bin/python3 ${UPATH}/delta_update_users.py
32564 usr00     30  10   82528    932    844 S   0.3  0.0   0:02.32              `- /usr/bin/cpulimit -l 50 -f -m -- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_upda+
32565 usr00     30  10 1023148 201140  40312 S        4.0   0:25.89                  `- /home/usr00/fishtest/server/env/bin/python3 /home/usr00/fishtest/server/utils/delta_update_users.py

vdbergh · 2024-05-16T17:10:02Z

The new stack trace feature will be handy for debugging this.

vdbergh · 2024-05-16T17:56:49Z

@ppigazzini

Unfortunately fishtest just hangs at sys.exit(), seemingly because of a lock (so the stack trace is not useful).

Can you perhaps do ^C on dev (or get the error message from journalctl)? Since you are running a more recent python, perhaps the error message is more illuminating. I see that the shutdown code has changed quite a bit.

ppigazzini · 2024-05-16T19:14:31Z

$ env/bin/pserve production.ini http_port=6543
Starting server in PID 15641.
^Cflush
done
^Cflush
done
Exception ignored in: <module 'threading' from '/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py'>
Traceback (most recent call last):
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1622, in _shutdown
    lock.acquire()
  File "/home/usr00/fishtest/server/fishtest/rundb.py", line 356, in exit_run
    sys.exit(0)
SystemExit: 0

vondele · 2024-05-16T19:14:52Z

It seems like hanging in sys.exit can be because some threads are not finished? That could be visible in the stack traces, i.e. once it is hanging after a ctrl+c, send the usr1 signal, and look at where all threads are. With some luck, the hanging thread can be seen.

vdbergh · 2024-05-16T19:19:37Z

Thanks!

I found at least the culprit. I am calling buffer(run) in the constructor which apparently causes the hang at sys.exit().
Probably indeed something with background threads.

I assume that in the constructor it is ok to write directly to the db since the cache has not been initialized yet.

ppigazzini · 2024-05-16T19:19:41Z

backtrace

Click to view

===================  <_MainThread(MainThread, stopped 140327722854208)>  ======================
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1622, in _shutdown
    lock.acquire()
  File "/home/usr00/fishtest/server/fishtest/__init__.py", line 25, in thread_stack_dump
    traceback.print_stack(sys._current_frames()[th.ident])
===================  <Thread(pymongo_server_monitor_thread, started daemon 140327159478016)>  ======================
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/periodic_executor.py", line 141, in _run
    if not self._target():
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/monitor.py", line 62, in target
    monitor._run()  # type:ignore[attr-defined]
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/monitor.py", line 192, in _run
    self._server_description = self._check_server()
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/monitor.py", line 235, in _check_server
    return self._check_once()
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/monitor.py", line 282, in _check_once
    response, round_trip_time = self._check_with_socket(conn)
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/monitor.py", line 304, in _check_with_socket
    response = Hello(conn._next_reply(), awaitable=True)
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/pool.py", line 918, in _next_reply
    reply = self.receive_message(None)
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/pool.py", line 1037, in receive_message
    return receive_message(self, request_id, self.max_message_size)
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/network.py", line 317, in receive_message
    length, _, response_to, op_code = _UNPACK_HEADER(_receive_data_on_socket(conn, 16, deadline))
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/network.py", line 394, in _receive_data_on_socket
    wait_for_read(conn, deadline)
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/network.py", line 375, in wait_for_read
    readable = conn.socket_checker.select(sock, read=True, timeout=timeout)
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/socket_checker.py", line 66, in select
    res = self._poller.poll(timeout_)
===================  <Thread(pymongo_kill_cursors_thread, started daemon 140327151085312)>  ======================
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/periodic_executor.py", line 156, in _run
    time.sleep(self._min_interval)
===================  <Thread(pymongo_server_rtt_thread, started daemon 140327141644032)>  ======================
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/home/usr00/fishtest/server/env/lib/python3.12/site-packages/pymongo/periodic_executor.py", line 156, in _run
    time.sleep(self._min_interval)
===================  <Timer(Thread-118, started 140326739048192)>  ======================
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1429, in run
    self.finished.wait(self.interval)
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 655, in wait
    signaled = self._cond.wait(timeout)
  File "/home/usr00/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 359, in wait
    gotit = waiter.acquire(True, timeout)

The reason for this PR is that before introducing more dynamic updates we would like to verify that the dynamic updates of run["workers"] and run["cores"] introduced in official-stockfish#2010 are bug free. In order to facilitate the use of periodic timers in Fishtest we have introduced a new scheduler in utils.py which may be interesting in its own right. Currently the timers defined are the cache flush timer and the timer introduced in this PR.

The reason for this PR is that before introducing more dynamic updates we would like to verify that the dynamic updates of run["workers"] and run["cores"] introduced in #2010 are bug free. In order to facilitate the use of periodic timers in Fishtest we have introduced a new scheduler in utils.py which may be interesting in its own right. Currently the timers defined are the cache flush timer and the timer introduced in this PR.

vdbergh force-pushed the workers_cores branch 4 times, most recently from 28472e5 to ac75b37 Compare May 15, 2024 03:25

peregrineshahin reviewed May 15, 2024

View reviewed changes

server/fishtest/rundb.py Outdated Show resolved Hide resolved

vdbergh force-pushed the workers_cores branch 2 times, most recently from 6043a22 to 970e7fe Compare May 15, 2024 12:57

vdbergh mentioned this pull request May 15, 2024

Document load under fleet #2003

Open

vdbergh force-pushed the workers_cores branch from 970e7fe to 5899be9 Compare May 15, 2024 14:38

vdbergh force-pushed the workers_cores branch from 5899be9 to 315ce21 Compare May 16, 2024 13:55

ppigazzini added enhancement server server side changes scaling handle >> 1M worker cores labels May 16, 2024

vdbergh marked this pull request as draft May 16, 2024 17:50

vdbergh force-pushed the workers_cores branch from 315ce21 to 0930646 Compare May 17, 2024 05:03

vdbergh marked this pull request as ready for review May 17, 2024 05:04

vdbergh mentioned this pull request May 29, 2024

hanging server #2041

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fully dynamic updates of run["cores"] and run["workers"] (RFC) #2010

Fully dynamic updates of run["cores"] and run["workers"] (RFC) #2010

vdbergh commented May 14, 2024 •

edited

Loading

vdbergh commented May 16, 2024

vdbergh commented May 16, 2024

ppigazzini commented May 16, 2024

vdbergh commented May 16, 2024 •

edited

Loading

vdbergh commented May 16, 2024

ppigazzini commented May 16, 2024

vdbergh commented May 16, 2024

ppigazzini commented May 16, 2024

vdbergh commented May 16, 2024

vdbergh commented May 16, 2024

ppigazzini commented May 16, 2024

vondele commented May 16, 2024

vdbergh commented May 16, 2024

ppigazzini commented May 16, 2024 •

edited

Loading

Fully dynamic updates of run["cores"] and run["workers"] (RFC) #2010

Fully dynamic updates of run["cores"] and run["workers"] (RFC) #2010

Conversation

vdbergh commented May 14, 2024 • edited Loading

vdbergh commented May 16, 2024

vdbergh commented May 16, 2024

ppigazzini commented May 16, 2024

vdbergh commented May 16, 2024 • edited Loading

vdbergh commented May 16, 2024

ppigazzini commented May 16, 2024

vdbergh commented May 16, 2024

ppigazzini commented May 16, 2024

vdbergh commented May 16, 2024

vdbergh commented May 16, 2024

ppigazzini commented May 16, 2024

vondele commented May 16, 2024

vdbergh commented May 16, 2024

ppigazzini commented May 16, 2024 • edited Loading

vdbergh commented May 14, 2024 •

edited

Loading

vdbergh commented May 16, 2024 •

edited

Loading

ppigazzini commented May 16, 2024 •

edited

Loading