Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv: ability to gracefully recover from large intent buildup #135934

Open
andrewbaptist opened this issue Nov 21, 2024 · 0 comments
Open

kv: ability to gracefully recover from large intent buildup #135934

andrewbaptist opened this issue Nov 21, 2024 · 0 comments
Labels
A-admission-control A-kv-transactions Relating to MVCC and the transactional model. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs P-1 Issues/test failures with a fix SLA of 1 month T-kv KV Team

Comments

@andrewbaptist
Copy link
Collaborator

andrewbaptist commented Nov 21, 2024

Is your feature request related to a problem? Please describe.

In a customer case, we saw almost 10 billion intents created over multiple ranges over a 5 hour window from a single INSERT INTO ... SELECT FROM statement that was ultimately killed. The MVCC GC queue ended up causing severe LSM inversion which ultimately caused a 30+ minute outage on a cluster. We need the ability to recover gracefully when we detect this type of problem.

Describe the solution you'd like
There are two high level approaches to gracefully recovering:

  1. Long term - Admission control automatically pacing the rate of intent resolution to prevent IO overload
  2. Short term - Some knob to manually rate limit intent resolution

Current the mvccGCQueue calls CleanupTxnIntentsOnGCAsync which ends up calling cleanupFinishedTxnIntents.

See #97108 for more details on the current status of AC and intent resolution.

Jira issue: CRDB-44794

@andrewbaptist andrewbaptist added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs A-kv-transactions Relating to MVCC and the transactional model. A-admission-control T-kv KV Team P-1 Issues/test failures with a fix SLA of 1 month labels Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-admission-control A-kv-transactions Relating to MVCC and the transactional model. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs P-1 Issues/test failures with a fix SLA of 1 month T-kv KV Team
Projects
None yet
Development

No branches or pull requests

1 participant