Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VStream: When doing a full copy, first send table schemas as DDL statement events #17150

Draft
wants to merge 10,000 commits into
base: main
Choose a base branch
from
Draft

VStream: When doing a full copy, first send table schemas as DDL statement events #17150

wants to merge 10,000 commits into from

Conversation

mattlord
Copy link
Contributor

@mattlord mattlord commented Nov 5, 2024

Description

WiP PoC for #17143

Related Issue(s)

Checklist

  • "Backport to:" labels have been added if this change should be back-ported to release branches
  • If this change is to be back-ported to previous releases, a justification is included in the PR description
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on CI?
  • Documentation was added or is not required

frouioui and others added 30 commits July 30, 2024 18:20
@frouioui
Improve the queries upgrade/downgrade CI workflow by using same test …
1676ebd 
…code version as binary (vitessio#16494)

Signed-off-by: Florent Poinsard <[email protected]>
@harshit-gangal
Modify distributed transaction commit flow (vitessio#16468)
258be53 
Signed-off-by: Harshit Gangal <[email protected]>
@shlomi-noach
shcemadiff: support INSTANT DDL for changing column visibility (v…
5d2a604 
…itessio#16503)

Signed-off-by: Shlomi Noach <[email protected]>
@mcrauwel @shlomi-noach
clarify collations are also supported for db_charset (vitessio#16423)
d0927e7 
Signed-off-by: Matthias Crauwels <[email protected]>
Signed-off-by: Shlomi Noach <[email protected]>
Co-authored-by: Shlomi Noach <[email protected]>
@shlomi-noach
schemadiff: Online DDL support, declarative based (vitessio#16462)
0933269 
Signed-off-by: Shlomi Noach <[email protected]>
@shlomi-noach
Throttler: CheckThrottlerResponseCode to replace HTTP status codes (v…
bbb7a3f 
…itessio#16491)

Signed-off-by: Shlomi Noach <[email protected]>
@shlomi-noach
Throttler: SelfMetric interface, simplify adding new throttler metr…
c834df8 
…ics (vitessio#16469)

Signed-off-by: Shlomi Noach <[email protected]>
@shlomi-noach
Deprecate UpdateThrottlerConfig's --check-as-check-self and `--ch…
da99102 
…eck-as-check-shard` flags (vitessio#16507)

Signed-off-by: Shlomi Noach <[email protected]>
@rohit-nayak-ps
Multi-tenant Movetables: add e2e test for vdiff (vitessio#16309)
69e6fce 
Signed-off-by: Rohit Nayak <[email protected]>
@mattlord
VTAdmin: Upgrade websockets js package (vitessio#16504)
598faf3
@systay
bugfix: Allow cross-keyspace joins (vitessio#16520)
31de9fd 
Signed-off-by: Andres Taylor <[email protected]>
@vitess-bot @frouioui
Upgrade the Golang Dependencies (vitessio#16514)
f083e22 
Signed-off-by: GitHub <[email protected]>
Co-authored-by: frouioui <[email protected]>
@harshit-gangal
fix: show tables to use any keyspace on system schema (vitessio#16521)
1021543 
Signed-off-by: Harshit Gangal <[email protected]>
@harshit-gangal
fix: sequence table next value acl permission to writer role (vitessi…
8ff4b9a 
…o#16509)

Signed-off-by: Harshit Gangal <[email protected]>
@systay
simplify merging logic (vitessio#16525)
660a097 
Signed-off-by: Andres Taylor <[email protected]>
@harshit-gangal
vindex function: error when keyspace not selected (vitessio#16534)
93d12eb 
Signed-off-by: Harshit Gangal <[email protected]>
@harshit-gangal
vttablet api distributed transaction changes (vitessio#16506)
2714453 
Signed-off-by: Harshit Gangal <[email protected]>
@beingnoble03
evalengine: Implement PERIOD_ADD (vitessio#16492)
8940eb5
@GuptaManan100
Fix: Offset planning in hash joins (vitessio#16540)
5629525 
Signed-off-by: Manan Gupta <[email protected]>
@GrahamCampbell
Remove unused formatRelativeTime import (vitessio#16549)
b5c7cca 
Signed-off-by: Graham Campbell <[email protected]>
@mattlord
VReplication: Properly ignore errors from trying to drop tables that …
c160f46 
…don't exist (vitessio#16505)

Signed-off-by: Matt Lord <[email protected]>
@arthurschreiber
Fix RemoveTablet during TabletExternallyReparented causing connec…
342a4bd 
…tion issues (vitessio#16371)

Signed-off-by: Arthur Schreiber <[email protected]>
@shlomi-noach
Throttler/vreplication: fix app name used by VPlayer (vitessio#16578)
4a91e63 
Signed-off-by: Shlomi Noach <[email protected]>
@shlomi-noach
Tablet throttler: remove cached metric associated with removed tablet (…
76434ad 
…vitessio#16555)

Signed-off-by: Shlomi Noach <[email protected]>
@GuptaManan100
Atomic Transaction bug fix with PRS disruption (vitessio#16576)
68c19b4 
Signed-off-by: Manan Gupta <[email protected]>
@beingnoble03
evalengine: Implement PERIOD_DIFF (vitessio#16557)
32d1e60 
Signed-off-by: Noble Mittal <[email protected]>
@harshit-gangal
allow innodb_lock_wait_timeout as system variable (vitessio#16574)
942f97f 
Signed-off-by: Harshit Gangal <[email protected]>
@mattlord
VReplication: Improve table plan builder errors (vitessio#16588)
d3dd2fa 
Signed-off-by: Matt Lord <[email protected]>
@mattlord
VReplication: Return lock error everywhere that LockName fails (vites…
90b6bc5 
…sio#16560)

Signed-off-by: Matt Lord <[email protected]>
@GuptaManan100
Add command to see the unresolved transactions (vitessio#16589)
6f345a9 
Signed-off-by: Manan Gupta <[email protected]>
shlomi-noach and others added 16 commits October 27, 2024 09:39
@shlomi-noach
Online DDL: better error messages in cut-over phase (vitessio#17052)
d415d47 
Signed-off-by: Shlomi Noach <[email protected]>
@GuptaManan100
Improve errant GTID detection in ERS to handle more cases. (vitessio#…
c35754e 
…16926)

Signed-off-by: Manan Gupta <[email protected]>
@harshit-gangal @systay
Delegate Column Availability Checks to MySQL for Single-Route Queries (…
d15cd76 
…vitessio#17077)

Signed-off-by: Harshit Gangal <[email protected]>
Signed-off-by: Andres Taylor <[email protected]>
Co-authored-by: Andres Taylor <[email protected]>
@mattlord @shlomi-noach
Improve Schema Engine's TablesWithSize80 query (vitessio#17066)
ae2c3a6 
Signed-off-by: Matt Lord <[email protected]>
Signed-off-by: Shlomi Noach <[email protected]>
Co-authored-by: Shlomi Noach <[email protected]>
@vitess-bot @rohit-nayak-ps
[main] Copy v21.0.0 release notes (vitessio#17097)
5b3074b 
Signed-off-by: Rohit Nayak <[email protected]>
Co-authored-by: Rohit Nayak <[email protected]>
@harshit-gangal
java package updates for grpc and protobuf and release plugins (vites…
a7f1eb0 
…sio#17100)

Signed-off-by: Harshit Gangal <[email protected]>
@frouioui @mattlord
Fix unreachable errors when taking a backup (vitessio#17062)
7713f9a 
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Co-authored-by: Matt Lord <[email protected]>
@frouioui
Add release-21.0 to the Golang Upgrade workflow (vitessio#16916)
b6fe439 
Signed-off-by: Florent Poinsard <[email protected]>
@GuptaManan100
Fix flaky test TestShardSync (vitessio#17095)
f76b359 
Signed-off-by: Manan Gupta <[email protected]>
@frouioui
Fix panic in vttablet when closing topo server twice (vitessio#17094)
c1f1dde
@mattlord
CI: Correct etcd and ncurses package names for Ubuntu 24.04 (vitessio…
d20460b 
…#16966)

Signed-off-by: Matt Lord <[email protected]>
@beingnoble03
refac: Remove duplicate forAllShards methods from `vt/vtctl/workflo…
2191360 
…w` (vitessio#17025)

Signed-off-by: Noble Mittal <[email protected]>
@mattlord
VReplication: Fixes for generated column handling (vitessio#17107)
fbddae3 
Signed-off-by: Matt Lord <[email protected]>
@GuptaManan100
Fix flakiness in TestDisruptions for two pc testing (vitessio#17106)
ced2010 
Signed-off-by: Manan Gupta <[email protected]>
@dbussink @timvaillancourt
Move to native sqlite3 queries (vitessio#17124)
012e601 
Signed-off-by: Dirkjan Bussink <[email protected]>
Co-authored-by: Tim Vaillancourt <[email protected]>
@shlomi-noach
schemadiff: consistent key ordering in table diff (vitessio#17141)
f907205 
Signed-off-by: Shlomi Noach <[email protected]>
@mattlord mattlord added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: VReplication labels Nov 5, 2024
@vitess-bot vitess-bot
Copy link
Contributor

vitess-bot bot commented Nov 5, 2024

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

  • Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

  • Apply the release notes (needs details) label if users need to know about this change.
  • New features should be documented.
  • There should be some code comments as to why things are implemented the way they are.
  • There should be a comment at the top of each new or modified test to explain what the test does.

New flags

  • Is this flag really necessary?
  • Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
  • vtctl command output order should be stable and awk-able.

@vitess-bot vitess-bot bot added NeedsBackportReason If backport labels have been applied to a PR, a justification is required NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels Nov 5, 2024
@mattlord mattlord removed NeedsIssue A linked issue is missing for this Pull Request NeedsBackportReason If backport labels have been applied to a PR, a justification is required labels Nov 5, 2024
@mattlord mattlord force-pushed the vstream_copy_schemas branch from b1490c7 to 9fdd09d Compare November 5, 2024 16:34
@github-actions github-actions bot added this to the v22.0.0 milestone Nov 5, 2024
@mattlord mattlord force-pushed the vstream_copy_schemas branch from 9fdd09d to 63e8dfe Compare November 5, 2024 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GuptaManan100 GuptaManan100 Awaiting requested review from GuptaManan100 GuptaManan100 will be requested when the pull request is marked ready for review GuptaManan100 is a code owner

@frouioui frouioui Awaiting requested review from frouioui frouioui will be requested when the pull request is marked ready for review frouioui is a code owner

@rohit-nayak-ps rohit-nayak-ps Awaiting requested review from rohit-nayak-ps rohit-nayak-ps will be requested when the pull request is marked ready for review rohit-nayak-ps is a code owner

@deepthi deepthi Awaiting requested review from deepthi deepthi will be requested when the pull request is marked ready for review deepthi is a code owner

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal harshit-gangal will be requested when the pull request is marked ready for review harshit-gangal is a code owner

@systay systay Awaiting requested review from systay systay will be requested when the pull request is marked ready for review systay is a code owner

@shlomi-noach shlomi-noach Awaiting requested review from shlomi-noach shlomi-noach will be requested when the pull request is marked ready for review shlomi-noach is a code owner

@timvaillancourt timvaillancourt Awaiting requested review from timvaillancourt timvaillancourt will be requested when the pull request is marked ready for review timvaillancourt is a code owner

@derekperkins derekperkins Awaiting requested review from derekperkins derekperkins will be requested when the pull request is marked ready for review derekperkins is a code owner

@vmg vmg Awaiting requested review from vmg vmg will be requested when the pull request is marked ready for review vmg is a code owner

@ajm188 ajm188 Awaiting requested review from ajm188 ajm188 will be requested when the pull request is marked ready for review ajm188 is a code owner

@notfelineit notfelineit Awaiting requested review from notfelineit notfelineit will be requested when the pull request is marked ready for review notfelineit is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
Component: VReplication NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says Type: Enhancement Logical improvement (somewhere between a bug and feature)
Projects
None yet
Milestone
v22.0.0
Development

Successfully merging this pull request may close these issues.

Feature Request: richer tablet/column type info in VStream copy phase output