add '-priority' flag to 'nomad job dispatch'

[janet]

Proposal

I would like the ability to set the priority in the job stanza using variable interpolation. It looks like the priority field is not yet supported for variable interpolation.

Example of desired nomad job stanza:

job "terraform-execution" {
  type = "batch"

  parameterized {
    payload = "required"
    meta_required = ["priority"]
  }

  priority = "${NOMAD_META_PRIORITY}"
  ...
}

Currently I get the following error when attempting the above:

Error getting job struct: Error parsing job file from /dev/fd/63:
error parsing 'job': 1 error(s) decoding:

* cannot parse 'Priority' as int: strconv.ParseInt: parsing "${NOMAD_META_PRIORITY}": invalid syntax

Use-cases

Currently, I am looking for a way to dispatch a nomad parameterized job from my application code and set its job priority dynamically within the application code.

A future use case could be to use a feature flag for the job priority in the application code to dynamically change the job priority.

Attempted Solutions

A workaround would be to create separate nomad jobs by a predetermined set of priorities and have my application code dispatch jobs to the nomad job with the appropriate priority.

[Fuco1]

Our use-case is as follows: one logical task is split into 1000 nomad batch jobs. To maximally utilize the hardware, when one logical task is finishing up, for example, last 400 batch jobs are running, but we have 300 CPUs, we start another logical task so the resources are fluidly taken over by the new logical task. But sometimes nomad will instead immediately place new batch jobs from task 2 and task 1 is left unfinished in inconsistent state with the last 50 batch jobs queued.

In other words, it seems nomad doesn't schedule in a FIFO manner but randomly. A dynamic priority assigned to each logical task which decreases by 1 with each new task would ensure that each task is fully finished before jobs from new task get placed.

[Fuco1]

Is there any pointers on to how I would go about implementing this in Nomad? If it is something which doesn't require changing half the core logic, I'd have a go at it, since this is really bothering us a lot.

Here @dadgar wrote that scheduling info must be available during job submission. But as I understand dispatched job really is its own entity so when the dispatch is being placed in nomad's queue it could have the priority set from a meta, right? Further

To clarify how the link issue would resolve this, one could set the priority of the job to be a variable supplied as metadata provided when dispatching the parameterized job.

I don't understand how this is possible to do now.

[tgross]

Hi folks! Sorry about the delayed response on this issue. Generally the way you'll want to do something like this is with HCL2 variable interpolation. Here's a demo modified from the default example job:

variable "priority" {
  type    = number
  default = 50
}

job "example" {
  datacenters = ["dc1"]

  priority = var.priority

  group "cache" {
    network {
      port "db" {
        to = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis:3.2"

        ports = ["db"]
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

$ nomad job run -var priority=100 ./example.nomad
...
$ nomad job inspect ex | jq '.Job.Priority'
100

[Fuco1]

Hi @tgross. This is not really the same setup. What OP and I are interested in is to set the priority of the individual dispatches to a parametrized job. Imagine you have one "video encoding" batch job you dispatch against. Sometimes a priority video might come along that you want to place right away, possibly even preempting existing running dispatches.

What your example does is it replaces the parent "template" job, not on the level of individual dispatches. That's why the OP is trying to set priority = "${NOMAD_META_PRIORITY}" by the meta dispatch value.

I don't think this scenario is supported with hcl2 variables.

Trying to do nomad job dispatch -var priority=50 <job> gives back flag provided but not defined: -var

[tgross]

Thanks @Fuco1, that helped me understand the problem much better.

Because variable interpolation happens at the HCL level in the CLI, adding it to nomad job dispatch (which doesn't require the job file to be present) would probably be tricky.

As for metadata, each dispatched version of the job gets copied from the parent job and then has its own evaluation (see job_endpoint.go#L1944-L2005). That evaluation has to pass through the scheduler just like every other evaluation so Nomad knows where to place it. And, as Armon pointed out in that old ticket, the metadata gets interpolated after placement.

That being said, rather than trying to interpolate the priority field, it'd be much easier to implement (and probably nicer to use!) if we could let nomad job dispatch set the priority on its own with a -priority flag. Normally I'd try to avoid threading job spec through CLI flags like that, but priority is an unusual field in that it effects the scheduler and not the job itself.

I'm going to update the issue title to make that the feature request, and get this in front of folks doing roadmapping.

[Fuco1]

Thanks for the explanation Tim, that sounds very reasonable. Dispatches are indeed a bit different. In a traditional work queue we would often use priority queues, so you would declare the priority at the moment of adding the item to the queue (= dispatch). Your proposal then very much resembles something like a "standard" rabbitmq or redis workflow.

[scarbeau]

Hey @tgross or others on the hashi team, can you share any updated thinking regarding this feature request? I'm considering tackling this as a first contribution.

[tgross]

Hi @scarbeau! I don't think it's on our super near-term todo list, so if you're interested in contributing we'd be happy to review that!