[log backup] abnormal checkpoint TS lag

[apollodafoni]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

• create gcp cluster
• use br restore data from v8.1.2 to master version
• after restore finished，start log backup

2. What did you expect to see? (Required)

checkpoint ts lag is less than 5min

3. What did you see instead (Required)

[2025/03/11 01:28:15.065 +00:00] [WARN] [regioniter.go:141] ["failed with trying to scan regions"] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [start=74] [end=75]

4. What is your TiDB version? (Required)

Release Version: v9.0.0-alpha-381-g739a934
Edition: Community
Git Commit Hash: 739a934
Git Branch: HEAD
UTC Build Time: 2025-03-10 06:19:26
GoVersion: go1.23.7
Race Enabled: false
Check Table Before Drop: false
Store: tikv

[apollodafoni]

/component br
/severity critical
/assign @3pointer

[apollodafoni]

/severity major

[apollodafoni]

/remove-severity critical

[3pointer]

This is a large-scale internal simulation test. The cluster contains tens of millions of regions, hundreds TBs of data, and under this setup, we performed large-scale update operations that generated hundreds of GBs of incremental data every hour.

In this scenario, we observed that accessing the PD to scan regions frequently timed out, which in turn caused the log backup to lag. At the same time, we noticed other queries also timing out when attempting to retrieve regions. This suggests that the test may have pushed the product beyond its current limits, making it difficult to fix in the short term. Therefore, in the near term, it’s advisable to lower the severity of this issue, while viewing it as a potential area for optimization in large-scale clusters over the longer term.