kv: ability to limit the maximum intent size on a transaction independently of #135935
Labels
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Is your feature request related to a problem? Please describe.
In a customer case, we saw almost 10 billion intents created over multiple ranges over a 5 hour window from a single INSERT INTO ... SELECT FROM statement that was ultimately killed. We have an ability to cap that today using
kv.transaction.reject_over_max_intents_budget.enabled
however that setting will not work in general. Specifically it is tied to the settingkv.transaction.max_intents_bytes
. The max intents bytes is currently kept under a max limit is by condensing the tracking of intents from point intents to range intents. This reduces the memory footprint but increases the chance of a transaction conflict. In practice clusters typically will not enablekv.transaction.reject_over_max_intents_budget.enabled
because it can reject transactions that are expected to complete.Describe the solution you'd like
Ideally these settings should be separate. The condensing of spans should still occur once a lower memory limit (4 MiB) is passed. There should be a higher memory limit for failing the transaction (something like 256MiB by default). This will allow these larger transactions to run without either using excessive memory or eventually creating too many intents.
Jira issue: CRDB-44795
The text was updated successfully, but these errors were encountered: