You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The in-server proxy never removes SSH connections to service replicas, even after the replicas were stopped. While the SSH processes can stop automatically, the connection objects are stored in Python memory indefinitely, which can eventually lead to excessive RAM consumption.
Solution
Remove connections to stopped replicas.
Implementation note
The in-server proxy cannot receive events about stopped replicas because they can be stopped by other dstack-server replicas. So the solution may need to introduce a background job to check replica statuses and remove connections accordingly.
Workaround
Restart the dstack server if this ever becomes a problem.
Would you like to help us implement this feature by sending a PR?
Yes
The text was updated successfully, but these errors were encountered:
Problem
The in-server proxy never removes SSH connections to service replicas, even after the replicas were stopped. While the SSH processes can stop automatically, the connection objects are stored in Python memory indefinitely, which can eventually lead to excessive RAM consumption.
Solution
Remove connections to stopped replicas.
Implementation note
The in-server proxy cannot receive events about stopped replicas because they can be stopped by other dstack-server replicas. So the solution may need to introduce a background job to check replica statuses and remove connections accordingly.
Workaround
Restart the dstack server if this ever becomes a problem.
Would you like to help us implement this feature by sending a PR?
Yes
The text was updated successfully, but these errors were encountered: