Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Poor connection management in in-server proxy #2238

Open
jvstme opened this issue Jan 28, 2025 · 0 comments
Open

[Bug]: Poor connection management in in-server proxy #2238

jvstme opened this issue Jan 28, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@jvstme
Copy link
Collaborator

jvstme commented Jan 28, 2025

Problem

The in-server proxy never removes SSH connections to service replicas, even after the replicas were stopped. While the SSH processes can stop automatically, the connection objects are stored in Python memory indefinitely, which can eventually lead to excessive RAM consumption.

Solution

Remove connections to stopped replicas.

Implementation note

The in-server proxy cannot receive events about stopped replicas because they can be stopped by other dstack-server replicas. So the solution may need to introduce a background job to check replica statuses and remove connections accordingly.

Workaround

Restart the dstack server if this ever becomes a problem.

Would you like to help us implement this feature by sending a PR?

Yes

@jvstme jvstme added the bug Something isn't working label Jan 28, 2025
@jvstme jvstme changed the title [Bug]: Resource leak in in-server proxy [Bug]: Ineffective connection management in in-server proxy Jan 28, 2025
@jvstme jvstme changed the title [Bug]: Ineffective connection management in in-server proxy [Bug]: Poor connection management in in-server proxy Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant