Recommended way to interface with Kubernetes from jobtap plugin

[milroy]

Based on our coffee time conversation today, I'd like to start a discussion on interfacing a jobtap plugin with the Kubernetes/OpenShift Python API. I've written a skeleton jobtap plugin based on @grondo's alloc-bypass plugin which looks for a flux-kube key in the system attributes in the canonical jobspec, fetches the jobspec, and puts a dummy R into the KVS. The goal I'm trying to achieve is to pass a canonical jobspec to a Kubernetes API-based plugin which will convert it to kube-YAML and start up corresponding pods/deployments.

The main design question I have is how best to pass the jobspec to a Python service or command (e.g. draft PR #3190). @grondo's initial suggestion was to register the Kubernetes API-based plugin as a service and have it grab the jobspec from the KVS. This should work, except I need to be sure Kube won't need environment variables (i.e. an auth token). The alternative is to pass the jobspec to the Kube plugin "directly". Does this mean a popen call from the jobtap plugin (passing the jobspec as an argument), or is there a more elegant way to achieve this?

[grondo]

To clarify: If you pull the jobspec from the KVS then you get the original submitted jobspec. The jobtap plugin has access to the redacted jobspec which is kept internally in the job manager. You could send this as part of the payload to the Python service, but the environment would be missing.

It might be easiest to prototype with the jobspec in the request message to your Python service.

The alternative is to pass the jobspec to the Kube plugin "directly". Does this mean a popen call from the jobtap plugin (passing the jobspec as an argument), or is there a more elegant way to achieve this?

By "directly" I meant as part of the payload in the request to the Python service. You could just use something like (untested):

flux_rpc_pack (flux_jobtap_get_flux (p),
               "kubeflux.request",
                FLUX_NODEID_ANY,
                0,
                "{s:I s:O}",
                "id", id,
                "jobspec", jobspec);

Where id is the jobid and jobspec is a json_t * which you got from flux_plugin_arg_unpack (args, FLUX_PLUGIN_ARG_IN, "{s:o}", "jobspec", &jobspec);

The big O increments the reference count on the jobspec object so that it shouldn't have to be copied more than necessary to send it along to your service.

BTW, you can create a Python script that registers a new service with the service_register() method of a Flux handle. So at script startup, you can do:

import flux
from flux.constants import FLUX_MSGTYPE_REQUEST

h = flux.Flux()
h.service_register("kubeflux").get()

def callback(f, t, msg, arg):
    print ("got message!", msg.payload_str)
    f.respond(msg)

w = h.msg_watcher_create(callback, FLUX_MSGTYPE_REQUEST, "kubeflux.*")
w.start()

h.reactor_run()

callback will be called for all requests to "kubeflux.*"

[milroy]

This makes sense; thanks for the help. We're still iterating on the architecture, but when we home in on a solution I'll probably ask some follow-up questions here.

[milroy]

After several discussions with @dongahn and @SteVwonder I decided to pursue an architecture where the jobtap plugin with popen flux kube daemonize, where daemonize starts the flux kube CLI command as a service.

I've run into the following error:

flux-kube: ERROR: [Errno 2] Unable to connect to Flux

generated by:

self.flux_h = flux.Flux()

in the daemonize class. Here's the call from the jobtap side:

    pf = popen("flux kube daemonize", "r");
    if (!pf) {
        flux_log_error (flux_jobtap_get_flux (p),
                        "flux-kube: failed to daemonize");
        goto done;
    }

Any idea what I'm doing wrong here? I assume there's something going on in the popen, maybe involving the shell it spawns?

[milroy]

Perhaps the fork/exec route is better here?