Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Race on GC enabled state under free threading #129533

Closed
hawkinsp opened this issue Jan 31, 2025 · 2 comments
Closed

Race on GC enabled state under free threading #129533

hawkinsp opened this issue Jan 31, 2025 · 2 comments
Assignees
@corona10
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading type-bug An unexpected behavior, bug, or error

Comments

@hawkinsp
Copy link
Contributor

hawkinsp commented Jan 31, 2025 •
edited by bedevere-app bot
Loading

Bug report

Bug description:

The following code, when run under Python 3.13 commit 0468ea1 built with TSAN, produces a data race error:

import concurrent.futures
import functools
import gc
import threading

num_threads = 10

def closure(b):
  b.wait()
  for _ in range(100):
      gc.disable()
      gc.enable()

with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
  for i in range(100):
    b = threading.Barrier(num_threads)
    for _ in range(num_threads):
      executor.submit(functools.partial(closure, b))

TSAN report:

WARNING: ThreadSanitizer: data race (pid=3441855)
  Write of size 4 at 0x5576617e4e34 by thread T10:
    #0 PyGC_Disable /usr/local/google/home/phawkins/p/cpython/Python/gc_free_threading.c:1508:22 (python3.13+0x444e39) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #1 gc_disable_impl /usr/local/google/home/phawkins/p/cpython/Modules/gcmodule.c:51:5 (python3.13+0x4d8cbf) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #2 gc_disable /usr/local/google/home/phawkins/p/cpython/Modules/clinic/gcmodule.c.h:45:12 (python3.13+0x4d8cbf)
    #3 cfunction_vectorcall_NOARGS /usr/local/google/home/phawkins/p/cpython/Objects/methodobject.c:484:24 (python3.13+0x289ed9) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #4 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ead4a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #5 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1ead4a)
    #6 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e271b) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #7 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #8 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #9 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #10 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x5723a2) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #11 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x5723a2)
    #12 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #13 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #14 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #15 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4902) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #16 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #17 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #18 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #19 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef38f) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #20 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef38f)
    #21 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #22 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #23 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #24 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564a82) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #25 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdd87) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)

  Previous write of size 4 at 0x5576617e4e34 by thread T9:
    #0 PyGC_Enable /usr/local/google/home/phawkins/p/cpython/Python/gc_free_threading.c:1499:22 (python3.13+0x444dc9) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #1 gc_enable_impl /usr/local/google/home/phawkins/p/cpython/Modules/gcmodule.c:37:5 (python3.13+0x4d8c9f) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #2 gc_enable /usr/local/google/home/phawkins/p/cpython/Modules/clinic/gcmodule.c.h:27:12 (python3.13+0x4d8c9f)
    #3 cfunction_vectorcall_NOARGS /usr/local/google/home/phawkins/p/cpython/Objects/methodobject.c:484:24 (python3.13+0x289ed9) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #4 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ead4a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #5 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1ead4a)
    #6 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e271b) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #7 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #8 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #9 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #10 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x5723a2) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #11 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x5723a2)
    #12 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #13 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #14 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #15 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4902) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #16 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #17 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #18 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #19 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef38f) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #20 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef38f)
    #21 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #22 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #23 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)

This is my attempt to reproduce a race I saw in the JAX tsan CI, which is similar but not identical:

WARNING: ThreadSanitizer: data race (pid=145998)
  Read of size 4 at 0x55cb2fa81e34 by thread T64 (mutexes: read M0):
    #0 gc_should_collect /__w/jax/jax/cpython/Python/gc_free_threading.c:1062:59 (python3.13+0x44732b) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #1 record_allocation /__w/jax/jax/cpython/Python/gc_free_threading.c:1086:13 (python3.13+0x44732b)
    #2 gc_alloc /__w/jax/jax/cpython/Python/gc_free_threading.c:1708:5 (python3.13+0x44732b)
    #3 _PyObject_GC_New /__w/jax/jax/cpython/Python/gc_free_threading.c:1720:20 (python3.13+0x447101) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #4 tuple_iter /__w/jax/jax/cpython/Objects/tupleobject.c:1110:10 (python3.13+0x2e0700) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #5 PyObject_GetIter /__w/jax/jax/cpython/Objects/abstract.c:2860:25 (python3.13+0x1c1f12) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
...

  Previous write of size 4 at 0x55cb2fa81e34 by thread T70 (mutexes: read M0):
    #0 PyGC_Disable /__w/jax/jax/cpython/Python/gc_free_threading.c:1508:22 (python3.13+0x444e39) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #1 __Pyx_PyType_Ready _traversal.c (_traversal.cpython-313t-x86_64-linux-gnu.so+0x7b16) (BuildId: 0e187ad70a2faa86eeb3f1292897a0491466b409)
    #2 exec_builtin_or_dynamic /__w/jax/jax/cpython/Python/import.c:815:12 (python3.13+0x4658d0) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #3 _imp_exec_dynamic_impl /__w/jax/jax/cpython/Python/import.c:4756:12 (python3.13+0x4658d0)
    #4 _imp_exec_dynamic /__w/jax/jax/cpython/Python/clinic/import.c.h:513:21 (python3.13+0x4658d0)
    #5 cfunction_vectorcall_O /__w/jax/jax/cpython/Objects/methodobject.c:512:24 (python3.13+0x28a135) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #6 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
...

i.e., it looks like there might be a race between gc_should_collect and PyGC_Disable as well.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs
@hawkinsp hawkinsp added the type-bug An unexpected behavior, bug, or error label Jan 31, 2025
hawkinsp added a commit to hawkinsp/jax that referenced this issue Jan 31, 2025
@hawkinsp
Add additional tsan suppressions for races.
ac2cffd 
These were found in https://github.com/jax-ml/jax/actions/runs/13072427817/job/36476819907 and reported in python/cpython#129533 and python/cpython#128714.
@hawkinsp hawkinsp mentioned this issue Jan 31, 2025
Merged
hawkinsp added a commit to hawkinsp/jax that referenced this issue Jan 31, 2025
@hawkinsp
Add additional tsan suppressions for races.
d4c6657 
These were found in https://github.com/jax-ml/jax/actions/runs/13072427817/job/36476819907 and reported in python/cpython#129533 and python/cpython#128714.
@picnixz picnixz added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Jan 31, 2025
@corona10 corona10 self-assigned this Feb 2, 2025
@corona10
Copy link
Member

corona10 commented Feb 2, 2025

I am working on

corona10 added a commit to corona10/cpython that referenced this issue Feb 2, 2025
@corona10
pythongh-129533: Update PyGC_Enable/Disable/IsEnabled to use CAS oper…
1f335ca 
…ation
corona10 added a commit to corona10/cpython that referenced this issue Feb 2, 2025
@corona10
pythongh-129533: Update PyGC_Enable/Disable/IsEnabled to use CAS oper…
62cccf8 
…ation
@bedevere-app bedevere-app bot mentioned this issue Feb 2, 2025
Merged
corona10 added a commit that referenced this issue Feb 6, 2025
@corona10
gh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomic operat…
b184abf 
…ion (gh-129563)
corona10 added a commit to corona10/cpython that referenced this issue Feb 6, 2025
@corona10
pythongh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomic …
aea5951 
…operation (pythongh-129563)

(cherry picked from commit b184abf)
@bedevere-app bedevere-app bot mentioned this issue Feb 6, 2025
Merged
corona10 added a commit that referenced this issue Feb 6, 2025
@corona10
[3.13] gh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomic…
f7af8bc 
… operat… (gh-129756)

gh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomic operation  (gh-129563)

(cherry picked from commit b184abf)
@corona10
Copy link
Member

corona10 commented Feb 7, 2025

@hawkinsp It's also fixed for 3.13, You can check the fix at the next release of 3.13 :)

@corona10 corona10 closed this as completed Feb 7, 2025
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this issue Feb 7, 2025
@corona10 @srinivasreddy
pythongh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomic …
9e29b4f 
…operation (pythongh-129563)
cmaloney pushed a commit to cmaloney/cpython that referenced this issue Feb 8, 2025
@corona10 @cmaloney
pythongh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomic …
8df34e9 
…operation (pythongh-129563)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@corona10 corona10

Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hawkinsp @corona10 @picnixz @Eclips4