-
Notifications
You must be signed in to change notification settings - Fork 96
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Fullscanoperation thread to choose only alive node #9284
Comments
I also noticed this issue. |
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 26, 2024
this commit has the following changes 1 introduce acommon targed_node_lock mechanism that can be used in nemesis and Scan operations 2 common run_nemesis wrapper can provide a node that is not under disruptive_nemesis together with providing a node with no nemesis. This will allow non-disruptive operations to pick the same node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
2 tasks
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 26, 2024
this commit has the following changes 1 introduce acommon targed_node_lock mechanism that can be used in nemesis and Scan operations 2 common run_nemesis wrapper can provide a node that is not under disruptive_nemesis together with providing a node with no nemesis. This will allow non-disruptive operations to pick the same node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 26, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 common run_nemesis wrapper can provide a node that is not under disruptive_nemesis together with providing a node with no nemesis. This will allow non-disruptive operations to pick the same node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 26, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 common run_nemesis wrapper can provide a node that is not under disruptive_nemesis together with providing a node with no nemesis. This will allow non-disruptive operations to pick the same node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 26, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 common run_nemesis wrapper can provide a node that is not under disruptive_nemesis together with providing a node with no nemesis. This will allow non-disruptive operations to pick the same node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 26, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 common run_nemesis wrapper can provide a node that is not under disruptive_nemesis together with providing a node with no nemesis. This will allow non-disruptive operations to pick the same node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 26, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 27, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 27, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 27, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 27, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Nov 28, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 1, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 1, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 1, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 1, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 1, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 2, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 2, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 2, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 2, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 2, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 3, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 3, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 3, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 3, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 3, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 3, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 3, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 9, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 9, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 9, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 10, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 10, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 10, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 10, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 10, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 10, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 10, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 10, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 11, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
temichus
added a commit
to temichus/scylla-cluster-tests
that referenced
this issue
Dec 11, 2024
this commit has the following changes 1 introduce common targed_node_lock mechanism that can be used in nemesis and Scan operations 2 FullScan operation now run only on free of nemeses node 3 change all node.running_nemesis settings to use common methods set/unset_running_nemesis from common targed_node_lock file (except unit tests) 4 change disrupt_rolling_restart_cluster nemesis to lock all nodes in the cluster befo performing restart fixes: scylladb#9284
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Packages
Scylla version:
2024.2.0-20241118.614d56348f46
with build-ide67376d9ddfea081a3bab398f4581ecdde59911d
Kernel Version:
5.15.0-1072-aws
Issue description
Full scan operation chose node which was then used by rolling restart nemesis and cause the error message:
Need to fix FullScan thread to choose only alive nodes
-or-
Fix rolling restart nemesis to mark restarting node as busy for other operations
Impact
Reported Error event mark job as failed.
How frequently does it reproduce?
Describe the frequency with how this issue can be reproduced.
Installation details
Cluster size: 6 nodes (i4i.4xlarge)
Scylla Nodes used in this run:
OS / Image:
ami-06d63888ff4cf3d3f
(aws: undefined_region)Test:
longevity-50gb-3days-test
Test id:
c5d16022-93b6-44b1-9bab-22571a3eade5
Test name:
enterprise-2024.2/tier1/longevity-50gb-3days-test
Test method:
longevity_test.LongevityTest.test_custom_time
Test config file(s):
Logs and commands
$ hydra investigate show-monitor c5d16022-93b6-44b1-9bab-22571a3eade5
$ hydra investigate show-logs c5d16022-93b6-44b1-9bab-22571a3eade5
Logs:
Jenkins job URL
Argus
The text was updated successfully, but these errors were encountered: