add vtgate flag that explicitly allows vstream copy #125
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Priya Bibra <[email protected]>
Signed-off-by: Priya Bibra <[email protected]>
Signed-off-by: Priya Bibra <[email protected]>
Signed-off-by: Priya Bibra <[email protected]>
Signed-off-by: Priya Bibra <[email protected]>
Signed-off-by: Priya Bibra <[email protected]>
pbibra
commented
Sep 11, 2023
| @@ -540,6 +545,12 @@ func (vs *vstream) streamFromTablet(ctx context.Context, sgtid *binlogdatapb.Sha | |||
| log.Infof("Starting to vstream from %s", tablet.Alias.String()) | |||
| // Safe to access sgtid.Gtid here (because it can't change until streaming begins). | |||
| var vstreamCreatedOnce sync.Once | |||
|
|
|||
| if !vs.vsm.allowVstreamCopy && (sgtid.Gtid == "" || len(sgtid.TablePKs) > 0) { | |||
There was a problem hiding this comment.
verifying whether this condition is sufficient for identifying a vcopy
There was a problem hiding this comment.
Confirmed here: vitessio/vitess#6277
And here: https://slack-pde.slack.com/archives/C01P84R7L02/p1694468217811639
timvaillancourt
approved these changes
Sep 12, 2023
timvaillancourt
pushed a commit
that referenced
this pull request
May 27, 2024
4 tasks
timvaillancourt
added a commit
that referenced
this pull request
May 28, 2024
* add vtgate flag that explicitly allows vstream copy (#125) * fix fs.BoolVar Signed-off-by: Tim Vaillancourt <[email protected]> * VSCopy: Resume the copy phase consistently from given GTID and lastpk (#11103) * VSCopy: Demonstrate to fail a test case on which the vstream API is supposed to resume the copy phase consistently Signed-off-by: yoheimuta <[email protected]> * VSCopy: Resume the copy phase consistently from given GTID and lastpk Signed-off-by: yoheimuta <[email protected]> * Build out the unit test some more Signed-off-by: Matt Lord <[email protected]> * Update tests for new behavior Signed-off-by: Matt Lord <[email protected]> * Improve comments Signed-off-by: Matt Lord <[email protected]> * Limit uvstreamer changes and update test Signed-off-by: Matt Lord <[email protected]> * Revert uvstreamer test changes Signed-off-by: Matt Lord <[email protected]> * Revert all uvstream changes Signed-off-by: Matt Lord <[email protected]> * VCopy: Revert the last three commits Signed-off-by: yoheimuta <[email protected]> * VCopy: Add a new vstream type that allows picking up where we left off Signed-off-by: yoheimuta <[email protected]> * VCopy: Revert the unit test change Signed-off-by: yoheimuta <[email protected]> * VCopy: Fix the end-to-end CI test Signed-off-by: yoheimuta <[email protected]> * Update logic for setting up uvstreamer based on input vgtid/tablepks. Add more catchup events to test Signed-off-by: Rohit Nayak <[email protected]> * Refactor logic to decide if event is to be sent. Enhance unit and e2e tests. Signed-off-by: Rohit Nayak <[email protected]> * Don't send events for tables which we can identify as ones we haven't started copy for Signed-off-by: Rohit Nayak <[email protected]> * Minor changes after self-review Signed-off-by: Rohit Nayak <[email protected]> * Add vstream copy resume to release notes Signed-off-by: Matt Lord <[email protected]> * Address review comments Signed-off-by: Matt Lord <[email protected]> Signed-off-by: yoheimuta <[email protected]> Signed-off-by: Matt Lord <[email protected]> Signed-off-by: Rohit Nayak <[email protected]> Co-authored-by: Matt Lord <[email protected]> Co-authored-by: Rohit Nayak <[email protected]> * VSCopy: Send COPY_COMPLETED events when the copy operation is done (#11740) * VSCopy: Demonstrate to fail a test case on which the vstream API sends new events showing copy completed Signed-off-by: yoheimuta <[email protected]> * VSCopy: Send new events when the copy operation is done Signed-off-by: yoheimuta <[email protected]> * VSCopy: Fix typo Signed-off-by: yoheimuta <[email protected]> * Initialize new map for the 'vstream * from' vtgate sql interface. Make vtadmin web protos Signed-off-by: Rohit Nayak <[email protected]> * VSCopy: Make TestVStreamCopyBasic fail fast to avoid the end2end timeout out Signed-off-by: yoheimuta <[email protected]> * VSCopy: stop sharing the 't1' table among multiple test cases running concurrently Signed-off-by: yoheimuta <[email protected]> * VSCopy: refactor the function signature to be clearer Signed-off-by: yoheimuta <[email protected]> * VSCopy: refactor the VEvents sorter to be simpler Signed-off-by: yoheimuta <[email protected]> * VSCopy: refactor to stop the sorter from including a fully copied event Signed-off-by: yoheimuta <[email protected]> Signed-off-by: yoheimuta <[email protected]> Signed-off-by: Rohit Nayak <[email protected]> Co-authored-by: Rohit Nayak <[email protected]> * VSCopy: Enable to copy from all shards in either a specified keyspace or all keyspaces (#11909) * VSCopy: Demonstrate to fail a test case on which the vstream API request doesn't include keyspace and shard Signed-off-by: yoheimuta <[email protected]> * VSCopy: Copy from all shards in all keyspaces by specifying only an empty gtid Signed-off-by: yoheimuta <[email protected]> * tests: Make TestRowCount stable regardless of the number of keyspaces Signed-off-by: yoheimuta <[email protected]> * tests: Cleanup TestCreateAndDropDatabase correctly to stop TestVStreamCopyWithoutKeyspaceShard from failing when running tests together Signed-off-by: yoheimuta <[email protected]> * tests: Tweak to fix a comment Signed-off-by: yoheimuta <[email protected]> * VSCopy: fix the unit tests when the input vgtid with an empty gtid lacks either keyspace or shard Signed-off-by: yoheimuta <[email protected]> * VSCopy: Keyspace wildcard selection lines up with the table wildcard selection Signed-off-by: yoheimuta <[email protected]> * VSCopy: Tests the VCopy with multiple keyspaces and resharding Signed-off-by: yoheimuta <[email protected]> * VSCopy: Make TestVStreamCopyMultiKeyspaceReshard clearer to check if the streaming two keyspaces works even after reshard Signed-off-by: yoheimuta <[email protected]> * VSCopy: Return an invalid argument error if shards are unspecified and gtid is neither 'current' nor empty Signed-off-by: yoheimuta <[email protected]> * VSCopy: Add a test description about its purpose and target Signed-off-by: yoheimuta <[email protected]> * VSCopy: Remove duplicate literals in the test Signed-off-by: yoheimuta <[email protected]> * VSCopy: Retain defaultReplicas variable in the test Signed-off-by: yoheimuta <[email protected]> * VSCopy: Explain why we are setting Match to 'customer.*' in the test Signed-off-by: yoheimuta <[email protected]> * VSCopy: Remove an unused VStreamFlag for the test Signed-off-by: yoheimuta <[email protected]> * VSCopy: Use sentence capitalization in the test Signed-off-by: yoheimuta <[email protected]> * VSCopy: Verify that we didn't lose any events or get duplicates of the keyspace being reshareded in the test Signed-off-by: yoheimuta <[email protected]> * VSCopy: Return a value instead of a pointer because there is no need to modify the value Signed-off-by: yoheimuta <[email protected]> * VSCopy: Add a comment describing what TestVStreamCopyFromAllKeyspacesAndAllShards is doing and why Signed-off-by: yoheimuta <[email protected]> * VSCopy: Add a comment describing why we expect these specific numbers of events from VStream API Signed-off-by: yoheimuta <[email protected]> * VSCopy: Tweak the test case name Signed-off-by: yoheimuta <[email protected]> * VSCopy: Make a utility function to sort COPY_COMPLETED events in the test Signed-off-by: yoheimuta <[email protected]> * VSCopy: Replace the matcher with a simpler one in the test Signed-off-by: yoheimuta <[email protected]> * VSCopy: Move the print debug call to the FailNow section in the test Signed-off-by: yoheimuta <[email protected]> * VSCopy: Use require.NoError in new tests Signed-off-by: yoheimuta <[email protected]> * VSCopy: Use require instead of t.Fatalf in the test Signed-off-by: yoheimuta <[email protected]> * VSCopy: Apply the reviewer's suggestion to make the error message easier to read Signed-off-by: yoheimuta <[email protected]> * VSCopy: Add a comment noting what we're actually testing Signed-off-by: yoheimuta <[email protected]> * VSCopy: Correct the test comment and elaborate the special-case Signed-off-by: yoheimuta <[email protected]> * VSCopy: Tweak an error message and a comment Signed-off-by: yoheimuta <[email protected]> * VSCopy: Adjust to a change in the signature of a test function that was introduced in the main repository Signed-off-by: yoheimuta <[email protected]> --------- Signed-off-by: yoheimuta <[email protected]> * attempt unit test fix Signed-off-by: Tim Vaillancourt <[email protected]> * update test error expected Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Tim Vaillancourt <[email protected]> Signed-off-by: yoheimuta <[email protected]> Signed-off-by: Matt Lord <[email protected]> Signed-off-by: Rohit Nayak <[email protected]> Co-authored-by: pbibra <[email protected]> Co-authored-by: yohei yoshimuta <[email protected]> Co-authored-by: Matt Lord <[email protected]> Co-authored-by: Rohit Nayak <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We are planning on support vstream copy testing in dev. This flag ensures we don't accidentally allow this in production until RDONLYs are properly supported for bootstrapping. This is only temporary and will be removed from our fork once we're ready in all environments.