Add an optional limit for concurrent CreateAuditStream operations
#41957
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
espadolini
added
no-changelog
|
How will this work in sync recording mode? Are we okay with preventing sessions from starting due to the limit of audit streams being open? |
If too many session uploads happen with no limit at the same time and all the instances of the Auth Service crash, sync recording mode is still going to be broken, right? If the cluster admin has decided that the control plane can only handle N session uploads at once, a consequence is going to be not running more than N sessions with sync recording mode. |
rosstimothy
approved these changes
May 23, 2024
greedy52
approved these changes
May 24, 2024
public-teleport-github-review-bot
bot
removed the request for review
from hugoShaka
espadolini
force-pushed
the
espadolini/createauditstream-limit
branch
from
013af80 to
dc1c97b
Compare
|
@espadolini See the table below for backport results.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds an unstable envvar (
TELEPORT_UNSTABLE_CREATEAUDITSTREAM_INFLIGHT_LIMIT) to limit the allowed in-flightCreateAuditStreamRPCs on the Auth Service's gRPC server. Rejected streams are immediately closed with atrace.ConnectionProblemError, which results in a gRPCunavailablecode. Current agents attempting to upload sessions will not even notice the error, they'll just get a (short) timeout while waiting for the upload status, and then they'll back off (linearly, from 5 to 500 seconds). The circuit breaker on the clients only checks that the stream is established correctly (which it's going to be, in the case of this sort of manual rejection of the stream), so the rest of Teleport will keep working as usual.This change is motivated by an outage that hit a large cluster with burst activity of hundreds of thousands of SSH sessions at once - the peripheral nodes tried uploading the newly created session recordings all at once, and memory consumption skyrocketed to the point of going out of memory. With a manually configured limit, memory consumption is going to be limited.
This is going to be unsupported and undocumented (thus the
TELEPORT_UNSTABLE_envvar) until we figure out a better strategy to limit concurrent operations in general. Teleport 15+ only, as Teleport 14 control planes have to support v13 agents, which rely on the audit stream to also emit audit log events.