Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv: ability to limit the maximum intent size on a transaction independently of #135935

Open
andrewbaptist opened this issue Nov 21, 2024 · 1 comment
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@andrewbaptist
Copy link
Collaborator

andrewbaptist commented Nov 21, 2024

Is your feature request related to a problem? Please describe.

In a customer case, we saw almost 10 billion intents created over multiple ranges over a 5 hour window from a single INSERT INTO ... SELECT FROM statement that was ultimately killed. We have an ability to cap that today using kv.transaction.reject_over_max_intents_budget.enabled however that setting will not work in general. Specifically it is tied to the setting kv.transaction.max_intents_bytes. The max intents bytes is currently kept under a max limit is by condensing the tracking of intents from point intents to range intents. This reduces the memory footprint but increases the chance of a transaction conflict. In practice clusters typically will not enable kv.transaction.reject_over_max_intents_budget.enabled because it can reject transactions that are expected to complete.

Describe the solution you'd like
Ideally these settings should be separate. The condensing of spans should still occur once a lower memory limit (4 MiB) is passed. There should be a higher memory limit for failing the transaction (something like 256MiB by default). This will allow these larger transactions to run without either using excessive memory or eventually creating too many intents.

Jira issue: CRDB-44795

@andrewbaptist andrewbaptist added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Nov 21, 2024
@arulajmani
Copy link
Collaborator

Hey @andrewbaptist, at first glance, this looks very similar to #135841. Should we coalesce around a single issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
None yet
Development

No branches or pull requests

2 participants