Fatal error python

[AYMENJD]

Hi yesterday i compile nogil python version and i found an unexpected behavior.

On startup my application prints

gc_get_refs(gc): -1
gc_get_refs(gc): -1
gc_get_refs(gc): -1
gc_get_refs(gc): -1
gc_get_refs(gc): -1
Process started (my app output)
gc_get_refs(gc): -1
gc_get_refs(gc): -1

and then after some minutes my app crashed with error:

Fatal Python error: _Py_queue_object: _Py_queue_object called with unowned object
Python runtime state: initialized

[colesbury]

Hi @AYMENJD, thanks for the bug report. What operating system are your using? Can you share the application that crashed?

[AYMENJD]

Hey, thanks for reply.

What operating system are your using?

Ubuntu 18.04 LTS 16 ram 8 cpu.

Can you share the application that crashed?

Unfortunately no.

Its telegram bot but high loaded with 400 update per second.
On startup it:

• load some C functions and shared .so library.
• creates ThreadPool with 400 threads.

And every update will be served in dedicated thread using pool.submit().

The update will be shared between 60+ loaded plugins. These plugins will access to a global variables that contains an important objects like: telegram/database connection object.

And as i can see its hard to reproduce the fatal error. Only gc_get_refs(gc) -1 are always printed on startup.

[colesbury]

Thanks for the information. The gc_get_refs(gc): -1 message signifies a serious problem. I've updated the code to make that a fatal error and to print additional information for debugging. Would you please try pulling, rebuilding, and re-running your application and positing the new error message?

If I understand correctly, the td library is accessed through ctypes, right?

Does the application use other Python C API extensions? What do plugins mean in this context? (Other Python packages or are these plugins something to do with the td Telegram library?)

[AYMENJD]

Thanks for the information. The gc_get_refs(gc): -1 message signifies a serious problem. I've updated the code to make that a fatal error and to print additional information for debugging. Would you please try pulling, rebuilding, and re-running your application and positing the new error message?

My bot is on production mode so will this effect the bot ?

If I understand correctly, the td library is accessed through ctypes, right?

Yes, i use this wrapper: tdjson.py

Does the application use other Python C API extensions? What do plugins mean in this context? (Other Python packages or are these plugins something to do with the td Telegram library?)

In my code i don't use any C API extension but my plugins use other pip packages and may these packages use that.

[AYMENJD]

Also before 3 hours the program also crashed with error: *** stack smashing detected ***: <unknown> terminated

[colesbury]

My bot is on production mode so will this effect the bot ?

It will crash and print debug information instead of printing gc_get_refs(gc): -1. So if your app printed that on start-up, it will crash (with debug info) on start-up instead.

[AYMENJD]

Modules/gcmodule.c:803: visit_reachable: Assertion "gc_refs >= 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x60f4d1d4410
object refcount : 4
object local    : 12
object shared   : 2
object tid      : 0x7fa57f1f0080
object type     : 0x55f7343c33c0
object type name: type
object repr     : <class 'pycld2.error'>
object gc_refs  : -1

referrer: obj=0x60f4f33ca90 (type=dict) gc_get_refs=832852883772 ob_ref_local=8 ob_ref_shared=0 ob_tid=0x7fa57f1f0080
referrer: obj=0x60f4f33dd50 (type=dict) gc_get_refs=832852884382 ob_ref_local=6 ob_ref_shared=2 ob_tid=0x7fa57f1f0080
referrer: obj=0x60f4f33de90 (type=getset_descriptor) gc_get_refs=832852884422 ob_ref_local=6 ob_ref_shared=2 ob_tid=0x7fa57f1f0080
referrer: obj=0x60f4f33dee0 (type=tuple) gc_get_refs=832852884432 ob_ref_local=4 ob_ref_shared=0 ob_tid=0x7fa57f1f0080
referrer: obj=0x60f4caebe90 (type=module) gc_get_refs=1 ob_ref_local=12 ob_ref_shared=2 ob_tid=0x7fa57f1f0080

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Thread 0x00007fa5749a8700 (most recent call first):
  File "/usr/local/lib/ttpynogil/lib/python3.9/threading.py", line 318 in wait
  File "/usr/local/lib/ttpynogil/lib/python3.9/queue.py", line 180 in get
  File "/root/wd/telegram/worker.py", line 42 in _run_thread
  File "/usr/local/lib/ttpynogil/lib/python3.9/threading.py", line 886 in run
  File "/usr/local/lib/ttpynogil/lib/python3.9/threading.py", line 935 in _bootstrap_inner
  File "/usr/local/lib/ttpynogil/lib/python3.9/threading.py", line 906 in _bootstrap

Thread 0x00007fa5751a9700 (most recent call first):
  File "/root/wd/telegram/tdjson.py", line 110 in receive
  File "/root/wd/telegram/client.py", line 926 in _listen_to_td
  File "/usr/local/lib/ttpynogil/lib/python3.9/threading.py", line 886 in run
  File "/usr/local/lib/ttpynogil/lib/python3.9/threading.py", line 935 in _bootstrap_inner
  File "/usr/local/lib/ttpynogil/lib/python3.9/threading.py", line 906 in _bootstrap

Current thread 0x00007fa57f1f0080 (most recent call first):
  File "<frozen importlib._bootstrap_external>", line 648 in _compile_bytecode
  File "<frozen importlib._bootstrap_external>", line 979 in get_code
  File "<frozen importlib._bootstrap_external>", line 847 in exec_module
  File "<frozen importlib._bootstrap>", line 681 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 988 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "/usr/local/lib/ttpynogil/lib/python3.9/site-packages/urllib3/util/connection.py", line 8 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 851 in exec_module
  File "<frozen importlib._bootstrap>", line 681 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 988 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "/usr/local/lib/ttpynogil/lib/python3.9/site-packages/urllib3/util/__init__.py", line 4 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 851 in exec_module
  File "<frozen importlib._bootstrap>", line 681 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 988 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 974 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "/usr/local/lib/ttpynogil/lib/python3.9/site-packages/urllib3/connection.py", line 15 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 851 in exec_module
  File "<frozen importlib._bootstrap>", line 681 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 988 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "/usr/local/lib/ttpynogil/lib/python3.9/site-packages/urllib3/connectionpool.py", line 12 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 851 in exec_module
  File "<frozen importlib._bootstrap>", line 681 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 988 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "/usr/local/lib/ttpynogil/lib/python3.9/site-packages/urllib3/__init__.py", line 13 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 851 in exec_module
  File "<frozen importlib._bootstrap>", line 681 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 988 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "/usr/local/lib/ttpynogil/lib/python3.9/site-packages/requests/__init__.py", line 43 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 851 in exec_module
  File "<frozen importlib._bootstrap>", line 681 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 988 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "/usr/local/lib/ttpynogil/lib/python3.9/site-packages/wit/wit.py", line 11 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 851 in exec_module
  File "<frozen importlib._bootstrap>", line 681 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 988 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "/usr/local/lib/ttpynogil/lib/python3.9/site-packages/wit/__init__.py", line 11 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 851 in exec_module
  File "<frozen importlib._bootstrap>", line 681 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 988 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "/root/wd/plugins/speech2text.py", line 20 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 851 in exec_module
  File "<frozen importlib._bootstrap>", line 681 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 988 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1009 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1032 in _gcd_import
  File "/usr/local/lib/ttpynogil/lib/python3.9/importlib/__init__.py", line 127 in import_module
  File "/root/wd/bot.py", line 122 in load_plugins
  File "/root/wd/bot.py", line 166 in run
  File "/root/wd/__main__.py", line 29 in <module>
  File "/usr/local/lib/ttpynogil/lib/python3.9/runpy.py", line 87 in _run_code
  File "/usr/local/lib/ttpynogil/lib/python3.9/runpy.py", line 197 in _run_module_as_main
Aborted (core dumped)

It prints this.

[colesbury]

Looks like pycld2 crashes in upstream CPython 3.9 too with assertions enabled (debug build), so that's the first problem -- a bug in pycld2. I've filed an issue on the pycld2 github: aboSamoor/pycld2#50.

Until it's fixed upstream, you can try using a patched version of pycld2:

python3 -m pip install --force-reinstall git+https://github.com/colesbury/pycld2.git@a0227798265698aeff0fb457fa35a2e4e0f543a5

[AYMENJD]

Until it's fixed upstream, you can try using a patched version of pycld2:

Thanks!! the gc_get_refs(gc): -1 Fixed.

[AYMENJD]

I have a question.
Why if i do cp /usr/local/lib/pynogil/bin/python3 /usr/bin/pynogil the copied binary doesn't work on pip install or import package.

for example on pynogil -m pip install redis:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/dist-packages/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/__init__.py", line 4, in <module>
    from pip._internal.utils import _log
  File "/usr/local/lib/python3.9/dist-packages/pip/_internal/utils/_log.py", line 8, in <module>
    import logging
  File "/usr/lib/python3.9/logging/__init__.py", line 42, in <module>
    import threading
  File "/usr/lib/python3.9/threading.py", line 36, in <module>
    _set_sentinel = _thread._set_sentinel
AttributeError: module '_thread' has no attribute '_set_sentinel'

but if i do /usr/local/lib/pynogil/bin/python3 -m pip install redis it works fine.

[colesbury]

Why if i do cp /usr/local/lib/pynogil/bin/python3 /usr/bin/pynogil the copied binary doesn't work

You can't copy the python binary into a different python installation. You can't mix the "nogil" Python with the CPython standard library. The "nogil" Python has changes to the standard library and it won't work with the standard library from other python installations.

[AYMENJD]

Why if i do cp /usr/local/lib/pynogil/bin/python3 /usr/bin/pynogil the copied binary doesn't work

You can't copy the python binary into a different python installation. You can't mix the "nogil" Python with the CPython standard library. The "nogil" Python has changes to the standard library and it won't work with the standard library from other python installations.

Got it TnX for ur time.

[AYMENJD]

@AYMENJD I think what you want is a symlink.

OwO, it works! much TnX.