Merge branch 'main' into maxeng-mirror-routingrules

Signed-off-by: Max Englander <[email protected]>

.github/workflows/static_checks_etc.yml

@@ -91,6 +91,16 @@ jobs:
               - 'bootstrap.sh'
               - 'misc/git/hooks/asthelpers'
               - '.github/workflows/static_checks_etc.yml'
+            astfmt:
+              - 'go/tools/astfmtgen/**'
+              - 'go/vt/sqlparser/**'
+              - 'Makefile'
+              - 'build.env'
+              - 'go.sum'
+              - 'go.mod'
+              - 'tools/**'
+              - 'bootstrap.sh'
+              - '.github/workflows/static_checks_etc.yml'
             end_to_end:
               - 'docker/**'
               - 'test.go'
@@ -168,6 +178,11 @@ jobs:
         if: steps.skip-workflow.outputs.skip-workflow == 'false' && (steps.changes.outputs.visitor == 'true' || steps.changes.outputs.go_files == 'true')
         run: |
           misc/git/hooks/asthelpers || exit 1
+          
+      - name: check_ast_format_fast
+        if: steps.skip-workflow.outputs.skip-workflow == 'false' && (steps.changes.outputs.astfmt == 'true' || steps.changes.outputs.go_files == 'true')
+        run: |
+          ./tools/check_astfmtgen.sh || exit 1
 
       - name: run ensure_bootstrap_version
         if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'

.gitignore

@@ -56,6 +56,8 @@ __debug_bin
 /php/composer.phar
 /php/vendor
 
+report*.xml
+
 # vitess.io preview site
 /preview-vitess.io/
 

.golangci.yml

@@ -3,6 +3,13 @@ run:
   timeout: 10m
 
 linters-settings:
+  depguard:
+    rules:
+      use_modern_packages:
+        list-mode: lax
+        deny:
+          - pkg: "math/rand$"
+            desc: Please use math/rand/v2
   errcheck:
     exclude: ./misc/errcheck_excludes.txt
   goimports:
@@ -17,6 +24,7 @@ linters:
   disable-all: true
   enable:
     # Defaults
+    - depguard
     - errcheck
     - govet
     - ineffassign

changelog/19.0/19.0.0/release_notes.md

@@ -53,6 +53,10 @@ Vitess will however, continue to support importing from MySQL 5.7 into Vitess ev
 
 #### <a id="deprecation-vitess-lite-mysqld"/>Docker Image vitess/lite
 
+> [!CAUTION]  
+> If you are using incremental backups, you must remain on the `vitess/lite` image, as the official MySQL image does not have `mysqlbinlog` installed.
+> See https://github.com/vitessio/vitess/issues/16281 for more information.
+
 The `mysqld` binary is now deprecated in the `vitess/lite` Docker image and will be removed in a future release.
 This means that the MySQL/Percona version specific image tags for the `vitess/lite` image are deprecated.
 

changelog/20.0/20.0.0/changelog.md

@@ -39,7 +39,8 @@
  * Flaky test TestOnlineDDLVDiff: add additional check for vreplication workflow to exist [#15695](https://github.com/vitessio/vitess/pull/15695)
  * `schemadiff`: `DROP COLUMN` not eligible for `INSTANT` algorithm if covered by an index [#15714](https://github.com/vitessio/vitess/pull/15714)
  * `schemadiff` INSTANT DDL: impossible changes on tables with `FULLTEXT` index [#15725](https://github.com/vitessio/vitess/pull/15725)
- * SchemaEngine: Ensure GetTableForPos returns table schema for "current" position by default [#15912](https://github.com/vitessio/vitess/pull/15912) 
+ * SchemaEngine: Ensure GetTableForPos returns table schema for "current" position by default [#15912](https://github.com/vitessio/vitess/pull/15912)
+ * [release-20.0-rc] Online DDL shadow table: rename referenced table name in self referencing FK (#16205) [#16208](https://github.com/vitessio/vitess/pull/16208)
 #### Query Serving
  * Make connection killing resilient to MySQL hangs [#14500](https://github.com/vitessio/vitess/pull/14500)
  * TxThrottler: dont throttle unless lag [#14789](https://github.com/vitessio/vitess/pull/14789)
@@ -79,7 +80,9 @@
  * Fix aliasing in queries by keeping required projections  [#15943](https://github.com/vitessio/vitess/pull/15943)
  * connpool: Allow time out during shutdown [#15979](https://github.com/vitessio/vitess/pull/15979)
  * `schemadiff`: assume default collation for textual column when collation is undefined [#16000](https://github.com/vitessio/vitess/pull/16000)
- * fix: remove keyspace when merging subqueries [#16019](https://github.com/vitessio/vitess/pull/16019) 
+ * fix: remove keyspace when merging subqueries [#16019](https://github.com/vitessio/vitess/pull/16019)
+ * [release-20.0-rc] fix: rows affected count for multi table update for non-literal column value (#16181) [#16182](https://github.com/vitessio/vitess/pull/16182)
+ * [release-20.0-rc] Handle Nullability for Columns from Outer Tables (#16174) [#16186](https://github.com/vitessio/vitess/pull/16186)
 #### TabletManager
  * mysqlctl: Improve handling of the lock file [#15404](https://github.com/vitessio/vitess/pull/15404)
  * Fix possible race in MySQL startup and vttablet in parallel [#15538](https://github.com/vitessio/vitess/pull/15538)
@@ -105,9 +108,10 @@
  * VReplication: Take replication lag into account in VStreamManager healthcheck result processing [#15761](https://github.com/vitessio/vitess/pull/15761)
  * VReplication: Improve workflow cancel/delete [#15977](https://github.com/vitessio/vitess/pull/15977)
  * [release-20.0-rc] vtctldclient: Apply (Shard | Keyspace| Table) Routing Rules commands don't work (#16096) [#16125](https://github.com/vitessio/vitess/pull/16125)
- * [release-20.0] vtctldclient: Apply (Shard | Keyspace| Table) Routing Rules commands don't work (#16096) [#16126](https://github.com/vitessio/vitess/pull/16126)
  * [release-20.0-rc] VReplication: Improve workflow cancel/delete (#15977) [#16130](https://github.com/vitessio/vitess/pull/16130)
- * [release-20.0] VReplication: Improve workflow cancel/delete (#15977) [#16131](https://github.com/vitessio/vitess/pull/16131) 
+ * [release-20.0-rc] VDiff CLI: Fix VDiff `show` bug (#16177) [#16199](https://github.com/vitessio/vitess/pull/16199)
+ * [release-20.0-rc] VReplication: handle escaped identifiers in vschema when initializing sequence tables (#16169) [#16218](https://github.com/vitessio/vitess/pull/16218)
+ * [release-20.0-rc] VReplication Workflow: set state correctly when restarting workflow streams in the copy phase (#16217) [#16223](https://github.com/vitessio/vitess/pull/16223)
 #### VTAdmin
  * [VTAdmin API] Fix schema cache flag, add documentation [#15704](https://github.com/vitessio/vitess/pull/15704)
  * [VTAdmin] Remove vtctld web link, improve local example (#15607) [#15824](https://github.com/vitessio/vitess/pull/15824) 
@@ -118,7 +122,7 @@
  * Add timeout to all the contexts used for RPC calls in vtorc [#15991](https://github.com/vitessio/vitess/pull/15991) 
 #### vtexplain
  * vtexplain: Fix setting up the column information [#15275](https://github.com/vitessio/vitess/pull/15275)
- * vtexplain: Ensure memory topo is set up for throttler [#15279](https://github.com/vitessio/vitess/pull/15279) 
+ * vtexplain: Ensure memory topo is set up for throttler [#15279](https://github.com/vitessio/vitess/pull/15279)
 #### vttestserver
  * Revert unwanted logging change to `vttestserver` [#15148](https://github.com/vitessio/vitess/pull/15148)
  * use proper mysql version in the `vttestserver` images [#15235](https://github.com/vitessio/vitess/pull/15235) 
@@ -144,14 +148,14 @@
  * Validate go versions in Static Code Checks CI [#15932](https://github.com/vitessio/vitess/pull/15932)
  * Add CODEOWNERS for tablet throttler and schemadiff [#16036](https://github.com/vitessio/vitess/pull/16036)
  * [release-20.0-rc] Add DCO workflow (#16052) [#16057](https://github.com/vitessio/vitess/pull/16057)
- * [release-20.0] Add DCO workflow (#16052) [#16058](https://github.com/vitessio/vitess/pull/16058)
  * [release-20.0-rc] Remove DCO workaround (#16087) [#16092](https://github.com/vitessio/vitess/pull/16092)
- * [release-20.0] Remove DCO workaround (#16087) [#16093](https://github.com/vitessio/vitess/pull/16093) 
+ * Revert "[release-20.0-rc] Bump to `v20.0.0-SNAPSHOT` after the `v20.00-RC1` release (#16142)" [#16144](https://github.com/vitessio/vitess/pull/16144) 
+#### Docker
+ * Docker/vtadmin: Update node version [#16145](https://github.com/vitessio/vitess/pull/16145)
 #### General
  * [main] Upgrade the Golang version to `go1.22.1` [#15405](https://github.com/vitessio/vitess/pull/15405)
  * Upgrade go version to go1.22.2 [#15642](https://github.com/vitessio/vitess/pull/15642)
  * [main] Upgrade the Golang version to `go1.22.3` [#15865](https://github.com/vitessio/vitess/pull/15865)
- * [release-20.0] Upgrade the Golang version to `go1.22.4` [#16060](https://github.com/vitessio/vitess/pull/16060)
  * [release-20.0-rc] [release-20.0] Upgrade the Golang version to `go1.22.4` (#16060) [#16064](https://github.com/vitessio/vitess/pull/16064) 
 #### Online DDL
  * `onlineddl_scheduler` test: fix flakiness in artifact cleanup test [#15396](https://github.com/vitessio/vitess/pull/15396)
@@ -174,7 +178,6 @@
 #### VTAdmin
  * Bump vite from 4.5.2 to 4.5.3 in /web/vtadmin [#15634](https://github.com/vitessio/vitess/pull/15634)
  * [release-20.0-rc] Update braces package (#16115) [#16119](https://github.com/vitessio/vitess/pull/16119)
- * [release-20.0] Update braces package (#16115) [#16120](https://github.com/vitessio/vitess/pull/16120) 
 #### web UI
  * Remove highcharts dependency pt. 1 [#15970](https://github.com/vitessio/vitess/pull/15970)
 ### Documentation 
@@ -183,9 +186,9 @@
 #### Documentation
  * Fix docs for unmanaged tablets [#15437](https://github.com/vitessio/vitess/pull/15437)
  * [release-20.0-rc] Changelog 20.0: Fix broken links (#16048) [#16075](https://github.com/vitessio/vitess/pull/16075)
- * [release-20.0] Changelog 20.0: Fix broken links (#16048) [#16076](https://github.com/vitessio/vitess/pull/16076) 
 #### General
- * Add Shopify to `ADOPTERS.md` [#15853](https://github.com/vitessio/vitess/pull/15853) 
+ * Add Shopify to `ADOPTERS.md` [#15853](https://github.com/vitessio/vitess/pull/15853)
+ * [release-20.0-rc] release notes: update dml related release notes (#16241) [#16246](https://github.com/vitessio/vitess/pull/16246)
 #### Governance
  * amend contributing guide to ban trivial contributions [#15618](https://github.com/vitessio/vitess/pull/15618)
  * remove koz from active maintainer list [#15733](https://github.com/vitessio/vitess/pull/15733) 
@@ -274,7 +277,8 @@
 #### TabletManager
  * Introducing `ExecuteMultiFetchAsDba` gRPC and `vtctldclient ExecuteMultiFetchAsDBA` command [#15506](https://github.com/vitessio/vitess/pull/15506) 
 #### Throttler
- * VReplication: Add throttler stats [#15221](https://github.com/vitessio/vitess/pull/15221) 
+ * VReplication: Add throttler stats [#15221](https://github.com/vitessio/vitess/pull/15221)
+ * Throttler multi-metrics: proto changes [#16040](https://github.com/vitessio/vitess/pull/16040) 
 #### Topology
  * Topo: Add version support to GetTopologyPath [#15933](https://github.com/vitessio/vitess/pull/15933) 
 #### VReplication
@@ -341,7 +345,8 @@
  * delete TestActionAndTimeout [#15322](https://github.com/vitessio/vitess/pull/15322)
  * Deprecate old reparent metrics and replace with new ones [#16031](https://github.com/vitessio/vitess/pull/16031) 
 #### Docker
- * Revert the removal of the MySQL binaries in the `vitess/lite` image [#16042](https://github.com/vitessio/vitess/pull/16042) 
+ * Revert the removal of the MySQL binaries in the `vitess/lite` image [#16042](https://github.com/vitessio/vitess/pull/16042)
+ * [release-20.0-rc] Remove unnecessary Docker build workflows (#16196) [#16201](https://github.com/vitessio/vitess/pull/16201)
 #### Examples
  * Update env.sh so that is does not error when running on Mac [#15835](https://github.com/vitessio/vitess/pull/15835)
  * Local Examples: Add --binary-as-hex=false flag to mysql alias [#15996](https://github.com/vitessio/vitess/pull/15996) 
@@ -430,7 +435,6 @@
 #### VTTablet
  * Improve performance for `BaseShowTablesWithSizes` query. [#15713](https://github.com/vitessio/vitess/pull/15713)
  * Do not load table stats when booting `vttablet`. [#15715](https://github.com/vitessio/vitess/pull/15715)
- * [release-20.0] Do not load table stats when booting `vttablet`. (#15715) [#16101](https://github.com/vitessio/vitess/pull/16101)
 ### Regression 
 #### Query Serving
  * Fix routing rule query rewrite [#15253](https://github.com/vitessio/vitess/pull/15253)
@@ -439,7 +443,7 @@
  * fix: derived table join column expression to be part of add join predicate on rewrite [#15956](https://github.com/vitessio/vitess/pull/15956)
  * fix: insert on duplicate update to add list argument in the bind variables map [#15961](https://github.com/vitessio/vitess/pull/15961)
  * [release-20.0-rc] fix: order by subquery planning (#16049) [#16133](https://github.com/vitessio/vitess/pull/16133)
- * [release-20.0] fix: order by subquery planning (#16049) [#16134](https://github.com/vitessio/vitess/pull/16134) 
+ * [release-20.0-rc] feat: add a LIMIT 1 on EXISTS subqueries to limit network overhead (#16153) [#16192](https://github.com/vitessio/vitess/pull/16192)
 #### Throttler
  * Enable 'heartbeat_on_demand_duration' in local/examples [#15204](https://github.com/vitessio/vitess/pull/15204) 
 #### vttestserver
@@ -460,14 +464,20 @@
  * Copy `v17.0.7` release notes on `main` [#15890](https://github.com/vitessio/vitess/pull/15890)
  * [release-20.0-rc] Code Freeze for `v20.0.0-RC1` [#16046](https://github.com/vitessio/vitess/pull/16046)
  * Bump to `v21.0.0-SNAPSHOT` after the `v20.0.0-RC1` release [#16047](https://github.com/vitessio/vitess/pull/16047)
+ * [release-20.0-rc] Release of `v20.0.0-RC1` [#16137](https://github.com/vitessio/vitess/pull/16137)
+ * [release-20.0-rc] Bump to `v20.0.0-SNAPSHOT` after the `v20.0.0-RC1` release [#16142](https://github.com/vitessio/vitess/pull/16142)
+ * [release-20.0-rc] Bump to `v20.0.0-SNAPSHOT` after the `v20.0.0-RC1` release [#16146](https://github.com/vitessio/vitess/pull/16146)
+ * [release-20.0-rc] Release of `v20.0.0-RC2` [#16225](https://github.com/vitessio/vitess/pull/16225)
+ * [release-20.0-rc] Bump to `v20.0.0-SNAPSHOT` after the `v20.0.0-RC2` release [#16236](https://github.com/vitessio/vitess/pull/16236)
 ### Testing 
 #### Build/CI
  * Rewrite _many_ tests to use vtctldclient invocations, mostly non-output related stuff [#15270](https://github.com/vitessio/vitess/pull/15270)
  * [e2e] vtctld init tablet and some output-based commands [#15297](https://github.com/vitessio/vitess/pull/15297)
  * CI: Address data races on memorytopo Conn.closed [#15365](https://github.com/vitessio/vitess/pull/15365)
  * Reduce excessing logging in CI  [#15462](https://github.com/vitessio/vitess/pull/15462)
  * Split unit test and unit race into 2 components [#15734](https://github.com/vitessio/vitess/pull/15734)
- * CI: Address data race in TestSchemaVersioning [#15998](https://github.com/vitessio/vitess/pull/15998) 
+ * CI: Address data race in TestSchemaVersioning [#15998](https://github.com/vitessio/vitess/pull/15998)
+ * test: Replace t.fatalf with testify require [#16038](https://github.com/vitessio/vitess/pull/16038) 
 #### CLI
  * Add required tests for `internal/flag` [#15220](https://github.com/vitessio/vitess/pull/15220)
  * test: Add missing tests and refactor existing tests for `go/flagutil` [#15789](https://github.com/vitessio/vitess/pull/15789) 
@@ -489,7 +499,7 @@
  * Remove mysql 5.7 tests that are no longer required [#15809](https://github.com/vitessio/vitess/pull/15809)
  * Fix unit-test-runner bug [#15815](https://github.com/vitessio/vitess/pull/15815)
  * test: Add tests for `go/ioutil` and refactor existing [#15885](https://github.com/vitessio/vitess/pull/15885)
- * test: Add required tests for `vt/key`, `timer` and `cache/theine/bf` [#15976](https://github.com/vitessio/vitess/pull/15976) 
+ * test: Add required tests for `vt/key`, `timer` and `cache/theine/bf` [#15976](https://github.com/vitessio/vitess/pull/15976)
 #### Observability
  * VStreamer: add throttled logs when row/result/vstreamers get throttled. [#14936](https://github.com/vitessio/vitess/pull/14936) 
 #### Query Serving