-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VReplication: disable foreign_key_checks for bulk data cleanup #15261
Conversation
Signed-off-by: Matt Lord <[email protected]>
Signed-off-by: Matt Lord <[email protected]>
Review ChecklistHello reviewers! 👋 Please follow this checklist when reviewing this Pull Request. General
Tests
Documentation
New flags
If a workflow is added or modified:
Backward compatibility
|
@@ -34,6 +34,8 @@ import ( | |||
binlogdatapb "vitess.io/vitess/go/vt/proto/binlogdata" | |||
) | |||
|
|||
const testWorkflowFlavor = workflowFlavorVtctl |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be changed to workflowFlavorRandom
and also tested locally with workflowFlavorVtctld
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested locally with workflowFlavorVtctld
and it passes:
❯ git diff | cat
diff --git a/go/test/endtoend/vreplication/fk_test.go b/go/test/endtoend/vreplication/fk_test.go
index b7a88aee65..b15856501a 100644
--- a/go/test/endtoend/vreplication/fk_test.go
+++ b/go/test/endtoend/vreplication/fk_test.go
@@ -34,7 +34,7 @@ import (
binlogdatapb "vitess.io/vitess/go/vt/proto/binlogdata"
)
-const testWorkflowFlavor = workflowFlavorVtctl
+const testWorkflowFlavor = workflowFlavorVtctld
// TestFKWorkflow runs a MoveTables workflow with atomic copy for a db with foreign key constraints.
// It inserts initial data, then simulates load. We insert both child rows with foreign keys and those without,
❯ make build
Thu Feb 15 15:12:32 EST 2024: Building source tree
❯ go test -v -timeout 15m -run ^TestFKWorkflow$ vitess.io/vitess/go/test/endtoend/vreplication
VTDATAROOT is /opt/vtdataroot/vreple2e_687716
=== RUN TestFKWorkflow
E0215 15:13:19.528356 41743 vtorc_process.go:103] configuration - {
"Debug": true,
"ListenAddress": ":0",
"MySQLTopologyUser": "orc_client_user",
"MySQLTopologyPassword": "orc_client_user_password",
"MySQLReplicaUser": "vt_repl",
"MySQLReplicaPassword": "",
"RecoveryPeriodBlockSeconds": 1
}
=== RUN TestFKWorkflow/insertInitialFKData
=== RUN TestFKWorkflow/vtctlclient_vdiff_fktarget.fk
=== RUN TestFKWorkflow/vtctlclient_vdiff_fktarget.fk#01
--- PASS: TestFKWorkflow (52.09s)
--- PASS: TestFKWorkflow/insertInitialFKData (0.01s)
--- PASS: TestFKWorkflow/vtctlclient_vdiff_fktarget.fk (5.06s)
--- PASS: TestFKWorkflow/vtctlclient_vdiff_fktarget.fk#01 (5.06s)
PASS
ok vitess.io/vitess/go/test/endtoend/vreplication 53.235s
Will change it to random now and push that change.
Signed-off-by: Matt Lord <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice compact solution.
Signed-off-by: Matt Lord <[email protected]>
Signed-off-by: Matt Lord <[email protected]>
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #15261 +/- ##
=======================================
Coverage 67.46% 67.47%
=======================================
Files 1560 1560
Lines 193186 193205 +19
=======================================
+ Hits 130333 130359 +26
+ Misses 62853 62846 -7 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Matt Lord <[email protected]>
dae1ca5
to
c95720e
Compare
@@ -54,7 +54,7 @@ func TestVDiffStats(t *testing.T) { | |||
defer testStats.controllers[id].TableDiffPhaseTimings.Record(phase, time.Now()) | |||
time.Sleep(sleepTime) | |||
} | |||
want := int64(1.2 * float64(sleepTime)) // Allow 20% overhead for recording timing | |||
want := 10 * sleepTime // Allow 10x overhead for recording timing on flaky test hosts |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This addresses unrelated test flakes seen.
… cleanup (#15261) (#15265) Signed-off-by: Matt Lord <[email protected]> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Matt Lord <[email protected]>
Description
VReplication performs bulk data loads and cleanup (source side on
complete
and target side oncancel
) and for the load portion it disabledforeign_key_checks
(as do other bulk data load tools such asmysqldump
), however, it did NOT do this for the bulk data cleanup. This PR also disablesforeign_key_checks
for the cleanup step to ensure that we can successfully cleanup the data when the tables being dropped/renamed (rename only an option oncomplete
for the original source tables) have dependent constraints.This doesn't necessarily have to be backported to v19, but I thought we should because foreign key support exists there and this fixes an important bug related to that support AND v19 is still in RC.
Related Issue(s)
Checklist