run epilog even if job prolog fails or is canceled #6249
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: When a perilog process completes, the same two calls are always made, causing duplicated code. Combine the two calls into a new perilog_proc_finish() call. Update callers.
Problem: In order to trigger an epilog as a prolog is finishing, some extra data will need to be carried in the perilog proc structure. Add userid and R to struct perilog_proc for this purpose.
Problem: When the prolog fails for any reason, including a timeout, cancelation, or a failure on one or more nodes, the associated job is not started and therefore any configured epilog is not started. This is problematic if the prolog did something that is assumed to be undone by the epilog. Start any configured epilog immediately when a prolog fails. The epilog-start event is posted before the prolog-finish in order to prevent the job from reaching INACTIVE state immediately.
Problem: No tests ensure that a job epilog runs even after a prolog fails. Add a test to t2274-manager-perilog-per-rank.t.
Problem: The description of when the job epilog is run is not correct in flux-config-job-manager(5). Amend the description to include that the epilog runs after prolog failure as well as on the finish event.
|
Setting MWP here, Thanks! |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6249 +/- ##
==========================================
- Coverage 83.34% 83.34% -0.01%
==========================================
Files 521 521
Lines 85658 85693 +35
==========================================
+ Hits 71393 71418 +25
- Misses 14265 14275 +10
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses #6055 by triggering the job epilog when the prolog completes if it fails, e.g. due to timeout or if it is canceled due to a job exception.
Fixes #6055