Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't return error when attempt of sending delta fails to allow next attempt #213

Merged
merged 1 commit into from
Jan 20, 2025
Merged

Don't return error when attempt of sending delta fails to allow next attempt #213

merged 1 commit into from
Jan 20, 2025

Conversation

kasia-kujawa
Copy link
Contributor

func ExponentialBackoffWithContext(ctx context.Context, backoff Backoff, condition ConditionWithContextFunc) error repeats a condition check with exponential backoff. It immediately returns an error if the condition returns an error, the context is cancelled or hits the deadline,
or if the maximum attempts defined in backoff is exceeded (ErrWaitTimeout).

Agent logs without this change (only one attempt):

time="2025-01-16T15:58:27Z" level=warning msg="failed sending delta request: Post \"https://api--mac-2.local.cast.ai/v1/kubernetes/clusters/36b93485-f902-41a7-a8bc-cb7494d1c56b/agent-deltas\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" attempts=1 cluster_id=36b93485-f902-41a7-a8bc-cb7494d1c56b component_node_name=gke-kasiak-01-16-default-pool-1c35ddc0-nnn5 component_pod_name=castai-agent-6b6d699ff5-d85w4 full_snapshot=true items=408 provider=gke version=v0.77.0
time="2025-01-16T15:58:27Z" level=error msg="failed sending delta: sending delta request: Post \"https://api--mac-2.local.cast.ai/v1/kubernetes/clusters/36b93485-f902-41a7-a8bc-cb7494d1c56b/agent-deltas\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" cluster_id=36b93485-f902-41a7-a8bc-cb7494d1c56b component_node_name=gke-kasiak-01-16-default-pool-1c35ddc0-nnn5 component_pod_name=castai-agent-6b6d699ff5-d85w4 controller_id=ee9b4c4c-690d-4a9e-b2b7-1c7887ec7a9b k8s_version=1.30.8-gke.1051000 provider=gke version=v0.77.

Agent logs with this change (3 attempts):

time="2025-01-16T16:06:12Z" level=warning msg="failed sending delta request: Post \"https://api--mac-2.local.cast.ai/v1/kubernetes/clusters/36b93485-f902-41a7-a8bc-cb7494d1c56b/agent-deltas\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" attempts=1 cluster_id=36b93485-f902-41a7-a8bc-cb7494d1c56b component_node_name=gke-kasiak-01-16-default-pool-1c35ddc0-nnn5 component_pod_name=castai-agent-879b4c747-w6mqx full_snapshot=true items=405 provider=gke version=local
time="2025-01-16T16:06:12Z" level=warning msg="failed sending delta request: Post \"https://api--mac-2.local.cast.ai/v1/kubernetes/clusters/36b93485-f902-41a7-a8bc-cb7494d1c56b/agent-deltas\": io: read/write on closed pipe" attempts=2 cluster_id=36b93485-f902-41a7-a8bc-cb7494d1c56b component_node_name=gke-kasiak-01-16-default-pool-1c35ddc0-nnn5 component_pod_name=castai-agent-879b4c747-w6mqx full_snapshot=true items=405 provider=gke version=local
time="2025-01-16T16:06:12Z" level=warning msg="failed sending delta request: Post \"https://api--mac-2.local.cast.ai/v1/kubernetes/clusters/36b93485-f902-41a7-a8bc-cb7494d1c56b/agent-deltas\": io: read/write on closed pipe" attempts=3 cluster_id=36b93485-f902-41a7-a8bc-cb7494d1c56b component_node_name=gke-kasiak-01-16-default-pool-1c35ddc0-nnn5 component_pod_name=castai-agent-879b4c747-w6mqx full_snapshot=true items=405 provider=gke version=local
time="2025-01-16T16:06:12Z" level=error msg="failed sending delta: timed out waiting for the condition" cluster_id=36b93485-f902-41a7-a8bc-cb7494d1c56b component_node_name=gke-kasiak-01-16-default-pool-1c35ddc0-nnn5 component_pod_name=castai-agent-879b4c747-w6mqx controller_id=9a9adc26-6e57-45b5-a85e-9c10c376f5df k8s_version=1.30.8-gke.1051000 provider=gke version=local

@kasia-kujawa
Don't return error when attempt of sending delta fails to allow next …
49d672d 
…attempt

func ExponentialBackoffWithContext(ctx context.Context, backoff Backoff, condition ConditionWithContextFunc) error
repeats a condition check with exponential backoff.
It immediately returns an error if the condition returns an error,
the context is cancelled or hits the deadline,
or if the maximum attempts defined in backoff is exceeded (ErrWaitTimeout).
@kasia-kujawa kasia-kujawa marked this pull request as ready for review January 16, 2025 16:25
@kasia-kujawa kasia-kujawa merged commit 7af30a5 into main Jan 20, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pijusn-cast pijusn-cast pijusn-cast approved these changes

@saumas saumas Awaiting requested review from saumas saumas is a code owner

@mindaugasCast mindaugasCast Awaiting requested review from mindaugasCast mindaugasCast is a code owner

@dariuspie dariuspie Awaiting requested review from dariuspie dariuspie is a code owner

@zilvinasu zilvinasu Awaiting requested review from zilvinasu zilvinasu is a code owner

@KarolisL KarolisL Awaiting requested review from KarolisL KarolisL is a code owner

@mikenorgate mikenorgate Awaiting requested review from mikenorgate mikenorgate is a code owner

@jakubsolecki jakubsolecki Awaiting requested review from jakubsolecki jakubsolecki is a code owner

@vladklokun vladklokun Awaiting requested review from vladklokun vladklokun is a code owner

@linkas45 linkas45 Awaiting requested review from linkas45 linkas45 is a code owner

@laimonasr laimonasr Awaiting requested review from laimonasr laimonasr is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kasia-kujawa @pijusn-cast