add support for vtgate traffic mirroring

[maxenglander]

Description

This PR is split into two:

1. Topo and workflow protos and commands (this PR).
2. Query serving changes: add support for vtgate traffic mirroring (queryserving) #15992

Add support for cross-keyspace traffic mirroring.

• Mirror rules are added with either ApplyMirrorRules or MoveTables MirrorTraffic.
• Mirror rules mirror a percentage of traffic from a qualified table with an optional tablet type to a qualified table in another keyspace:

  {
    "rules": [
      {
        "from_table": "<keyspace>.<table>[@<tablet-type>]",
        "to_table": "<keyspace>.<table>",
        "percent": 5.0
      }
    ]
  }

• When mirror rules are correctly defined, then VTGate will produce a Mirror engine primitive during the query planning stage, which will contain the original query plan, as well as a "mirror target" plan.
• When VTGate executes a Mirror primitive, it will execute the original plan and mirror target in parallel. If the mirror target plan takes longer than the original plan, it is cancelled.

Limitations

There are a lot of limitations, restrictions, and "missing features" in the initial implementation. Some of the restrictions can be relaxed if demand for new use cases arise, and missing features can be added in future PRs:

• Each qualified table may have at most 1 mirror rule.
• Only SELECT and UNION statements can be mirrored at this time.
• Mirror rules may not be self-referential (same keyspace in from_table and to_table), circular (ks1 => ks2 => ks1), or chained (ks1 => ks2 => ks3).
• MirrorTraffic can only be used on simple, forward-direction MoveTables workflows. MirrorTraffic is not supported, for partial, multi-tenant, reverse, or non-Move Tables workflows at this time.
• MirrorTraffic must be called after MoveTables Create, and before MoveTables SwitchTraffic.
• At the moment, there are no new stats added to VTGate to observe the performance of mirrored queries. VTTablet-level stats may be used for now.

Benchmarks

When mirror rules are defined, memory allocations increase and performance decreases.

Plan builder

During the planning phase, when producing a mirror plan, memory allocations increase while performance decreases compared to when valid mirrors are not defined. This is because, when a mirror plan is produced, the original statement is cloned, and buildRoutePlan is run twice: once for the original statement, and once for the cloned statement.

Note: the times below measure how long it takes to run the entire sub-suite of mirror cases defined in mirror_cases.json.

$ go test ./go/vt/vtgate/planbuilder/ -run "^$" -bench BenchmarkBaselineVsMirrored -v
goos: darwin
goarch: arm64
pkg: vitess.io/vitess/go/vt/vtgate/planbuilder
BenchmarkBaselineVsMirrored
BenchmarkBaselineVsMirrored/Baseline
BenchmarkBaselineVsMirrored/Baseline-8              6372            167741 ns/op           85207 B/op       2145 allocs/op
BenchmarkBaselineVsMirrored/Mirrored
BenchmarkBaselineVsMirrored/Mirrored-8              3262            357991 ns/op          184815 B/op       4612 allocs/op
PASS
ok      vitess.io/vitess/go/vt/vtgate/planbuilder       4.707s

Executor

Likewise, during the query execution phase, memory allocations increase and performance decreases with higher percentages of traffic are mirrored.

$ go test ./go/vt/vtgate/ -bench BenchmarkSelectMirror -run "^$"
W0515 09:01:06.972080   80113 log.go:39] Failed to read in config : Config File "vtconfig" Not Found in "[/Users/maxenglander/dev/vitessio/vitess/go/vt/vtgate]". This is optional, and can be ignored if you are not using config files. For a detailed explanation, see https://github.com/vitessio/vitess/blob/main/doc/viper/viper.md#config-files.
goos: darwin
goarch: arm64
pkg: vitess.io/vitess/go/vt/vtgate
BenchmarkSelectMirror/mirror_0.00%-8               58549             21210 ns/op            7981 B/op        179 allocs/op
BenchmarkSelectMirror/mirror_1.00%-8               57349             22204 ns/op            8068 B/op        181 allocs/op
BenchmarkSelectMirror/mirror_5.00%-8               51040             24883 ns/op            8450 B/op        190 allocs/op
BenchmarkSelectMirror/mirror_10.00%-8              44256             28203 ns/op            8887 B/op        200 allocs/op
BenchmarkSelectMirror/mirror_25.00%-8              31537             37518 ns/op           10233 B/op        233 allocs/op
BenchmarkSelectMirror/mirror_50.00%-8              22518             53053 ns/op           12459 B/op        286 allocs/op
BenchmarkSelectMirror/mirror_100.00%-8             13972             86004 ns/op           17011 B/op        396 allocs/op
PASS
ok      vitess.io/vitess/go/vt/vtgate   15.673s

E2E

$ go test ./go/vt/vtgate/endtoend -bench="BenchmarkMirror" -run="^$"   
goos: darwin
goarch: arm64
pkg: vitess.io/vitess/go/vt/vtgate/endtoend
BenchmarkMirror/point_select,_{_ks_=>_ks1_}.tbl1/mirror_0%-8                7641            159035 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks1_}.tbl1/mirror_1%-8                7116            160606 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks1_}.tbl1/mirror_5%-8                6870            163657 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks1_}.tbl1/mirror_10%-8               6888            167062 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks1_}.tbl1/mirror_25%-8               6301            183383 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks1_}.tbl1/mirror_50%-8               4603            251924 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks1_}.tbl1/mirror_100%-8              3840            304464 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks2_}.tbl1,_{_ks_=>_ks3_}.tbl2/mirror_0%-8                    7581            158086 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks2_}.tbl1,_{_ks_=>_ks3_}.tbl2/mirror_1%-8                    7126            158672 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks2_}.tbl1,_{_ks_=>_ks3_}.tbl2/mirror_5%-8                    7105            163114 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks2_}.tbl1,_{_ks_=>_ks3_}.tbl2/mirror_10%-8                   6618            167025 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks2_}.tbl1,_{_ks_=>_ks3_}.tbl2/mirror_25%-8                   5848            197117 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks2_}.tbl1,_{_ks_=>_ks3_}.tbl2/mirror_50%-8                   5520            223948 ns/op
BenchmarkMirror/point_select,_{_ks_=>_ks2_}.tbl1,_{_ks_=>_ks3_}.tbl2/mirror_100%-8                  3894            304426 ns/op
PASS
ok      vitess.io/vitess/go/vt/vtgate/endtoend  34.057s

E2E, synthetic

TODO: spin up a realistic Vitess cluster with two keyspaces, and compare baseline performance to performance when mirror rules are configured.

Related Issue(s)

Addresses #13772.

Checklist

• "Backport to:" labels have been added if this change should be back-ported to release branches
• If this change is to be back-ported to previous releases, a justification is included in the PR description
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on CI?
• Documentation was added: document mirror rules website#1744

Deployment Notes

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[codecov]

Codecov Report

Attention: Patch coverage is 60.90535% with 95 lines in your changes missing coverage. Please review.

Project coverage is 68.70%. Comparing base (bb76046) to head (91a75e4).

Files	Patch %	Lines
...lient/command/vreplication/common/mirrortraffic.go	24.39%	31 Missing ⚠️
go/vt/vtctl/grpcvtctldserver/server.go	5.00%	19 Missing ⚠️
go/cmd/vtctldclient/command/mirror_rules.go	16.66%	10 Missing ⚠️
go/vt/vtctl/grpcvtctldclient/client_gen.go	0.00%	8 Missing ⚠️
go/vt/vtctl/workflow/server.go	88.33%	7 Missing ⚠️
go/vt/vtcombo/tablet_map.go	0.00%	6 Missing ⚠️
go/vt/vtctl/localvtctldclient/client_gen.go	0.00%	4 Missing ⚠️
go/vt/topo/vschema.go	85.71%	3 Missing ⚠️
go/vt/vtctl/workflow/traffic_switcher.go	88.46%	3 Missing ⚠️
go/cmd/vtcombo/cli/main.go	0.00%	2 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15945      +/-   ##
==========================================
- Coverage   68.71%   68.70%   -0.02%     
==========================================
  Files        1544     1547       +3     
  Lines      198011   198247     +236     
==========================================
+ Hits       136064   136200     +136     
- Misses      61947    62047     +100     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[harshit-gangal]

This is too big a PR to review.
Can we break this into 2 parts?

• Vreplication changes
• Query Serving changes

[maxenglander]

@harshit-gangal can do!

[rohit-nayak-ps]

Nice, excited to see this!

[deepthi]

Does switching traffic delete mirror rules? Interesting, but also seems correct. 👍

[maxenglander]

yes, i think you suggested that during one of the Vitess team meetings i attended, unless i misunderstood 😅

[deepthi]

Is it possible for people to provide MirrorRules through ApplyVSchema?

[maxenglander]

hm, it should not be possible any more than it is possible to change routing rules through ApplyVSchema.

[deepthi]

LGTM. This is going to be good :)