Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #2218
The PR:
process_terminating_job
could be called fromprocess_terminating_run
(e.g. for non-running provisioned jobs) (evidently to safe a processing iteration). Now,process_terminating_run
only marks jobs as TERMINATION, and onlyprocess_termination_jobs
background task performs the jobs termination. This simplifies locking management significantly and overall logic.stop_duration
to run configuration/profile to control max duration a job waits until force terminating.*_duration
parameters parsing. E.g."off"
was handled formax_duration
but notoff
and vice versa foridle_duration
.true
was allowed but parsed as 1.Next:
dstack stop
CLI command to wait until the run enters the terminal status. Currently, the API is async meaning that run may be terminating for some time after the API returns.