What is the best way to get job exit code from Python API?

[SteVwonder]

@jameshcorbett is attempting to get the exact exit code of a FAILED job via the Python API.

The only way that I'm aware of is to grab the job's eventlog and extract the status value from the finish event payload and then convert it to an exit code with os.WEXITSTATUS (which looks to be the same way that flux job status does it).

Is there a better way to do it? If not, maybe we should add a function/method for that to the python bindings.

[grondo]

Strange, I could have sworn the exitcode was tracked by job-info and thus available via a JobInfo object for the job, but this doesn't appear to be the case. We should open an issue to get this added.

[grondo]

In case it is helpful for now, here's the code to get the status from the eventlog:

import os
import sys
import flux
from flux import job

jobid = job.JobID(sys.argv[1])
status = job.event_wait(flux.Flux(), jobid, "finish").context["status"]

print(os.WEXITSTATUS(status))

ƒ(s=4,d=0,builddir) grondo@asp:~/git/flux-core.git$ flux mini submit sh -c 'exit 22'
ƒ8sd4WgXgj
ƒ(s=4,d=0,builddir) grondo@asp:~/git/flux-core.git$ flux python status.py ƒ8sd4WgXgj
22
ƒ(s=4,d=0,builddir) grondo@asp:~/git/flux-core.git$ flux mini submit true
ƒ8skQHSGkX
ƒ(s=4,d=0,builddir) grondo@asp:~/git/flux-core.git$ flux python status.py ƒ8skQHSGkX
0

[jameshcorbett]

Thanks @SteVwonder for opening this for me and @grondo for providing a workaround. Right now all my Python code uses the wait-any functionality, so it doesn't look like I can leverage the event_wait function since that seems to wait for updates about a specific job. But I can manage without any way to get the return code for now.

[SteVwonder]

Can you use wait-any to wait for a job (any job) to finish, and then use the above snippet to get the exit code? It is a little silly to wait for the same event twice, but IIUC it should work. And the second wait (the one in the snippet that gets the finish event) should be very quick. Just a thought.

[grondo]

Even if we add the exit status to job-info and thereby the JobInfo structure, the wait-any interface doesn't return a JobInfo object, it is a completely separate interface that talks directly to the job-manager. The C API has a flux_job_wait_get_status(3) interface, but unfortunately this just reports a boolean success/fail with an error string containing the actual exit status. I think that should be improved as well:

 {"id":255803252539392,"success":false,"errstr":"task(s) exited with exit code 22"}

Maybe when the job has a finish event, the status can be included directly in this response, since the job-manager is processing it anyway?

@garlick, any thoughts here?

[grondo]

Can you use wait-any to wait for a job (any job) to finish, and then use the above snippet to get the exit code?

I would suggest this for now. You only have to make the extra call to process the eventlog when the job status from wait indicates failure, and until we fix the job-manager wait interface, you'll have to make an extra call to job-info anyway.

[jameshcorbett]

Can you use wait-any to wait for a job (any job) to finish, and then use the above snippet to get the exit code?

I would suggest this for now. You only have to make the extra call to process the eventlog when the job status from wait indicates failure, and until we fix the job-manager wait interface, you'll have to make an extra call to job-info anyway.

Oh, I hadn't thought of that---good point. Yeah, I can do that for now.

[grondo]

This has been partially resolved by #3408. However you will still have to fetch extra information when wait_any indicates that a job was not successful.

You can fetch a JobInfo object for a failed job via something like the following:

job = JobList(h, ids=[jobid]).jobs[0]

#3408 introduced a new property returncode, which will be 0 for "successful" jobs, but will otherwise contain the job exit code (if the job exited), the negative signal number if the job shell was terminated due to a signal, or -128 if the job was cancelled before it ran.

[jameshcorbett]

I think this can be marked as answered. In addition to the event_wait and JobList interfaces @grondo mentioned above, there are two new interfaces:

• flux.job.result which returns a JobInfo object which has the returncode property mentioned above.
• flux.job.FluxExecutor() which returns futures that represent the return code of the job.