Scheduled *Service* jobs

[Daviey]

I'd like to be able to run Service jobs on a schedule. These differ from Periodic Batch jobs as they do not have an inbuilt end-state. A good example of this would be running a webserver (such as Apache) between the hours of 9:00-5:00 Monday-Friday.

The two workarounds I can currently think of are:

• Service task, with 2 paired Periodic Batch job each to perform a 'nomad run' and 'nomad stop'. This feels pretty dirty and prone to issues, and will get complicated quickly with multiple jobs.

• Run the webserver as a Batch job scheduling start, and have an inbuilt script (such as cron) that kills the entrypoint blocking process which terminates the container. This feels dirty because it will not be treated as a service, and the container will not be restarted on an unexpected failure.

I'd like to see a Time Scheduler in the Service job spec, but I i'm struggling to think of a way to express it.. Perhaps following cron standard:

simple start / stop:

job "docs" {
  scheduled {
    # only run from 9:00am Monday through to 17:00 Friday
    start = ["0 9 * * MON"]
    end   = ["0 17 * * FRI"]
  }
}

Specific hours on days:

job "docs" {
  scheduled {
    # only run on 09:00->17:00 between Monday and Friday
    start = ["0 9 * * MON-FRI"]
    end   = ["0 17 * * MON-FRI"]
  }
}

Limited days, fixed hours:

job "docs" {
  scheduled {
    # only run on 09:00->17:00 between Monday, Wednesday and Friday
    start = ["0 9 * * MON,WED,FRI"]
    end   = ["0 17 * * MON,WED,FRI"]
  }
}

multi-day different times (might be a bit of a reach!):

job "docs" {
  scheduled {
    # only run on Monday, Wednesday and Friday between different hours
    start = ["0 9 * * MON", "0 10 * * WED", "0 11 * * FRI"]
    end   = ["0 17 * * MON", "0 16 * * WED", "0 17 * * FRI"]
  }
}

[kak-tus]

Duplicate #1944

[Daviey]

I'm not quite sure I agree that it is a duplicate, but it is certainly linked. The other issue (#1944) is specifically for System jobs.. this is for scheduled Service jobs.

[stevenscg]

@Daviey I don't have this use-case right now, but it seems very interesting. What kinds of problems is it meant to address? I could see a fit for dev or test environments that you want to tear down after-hours or on weekends.

[redi-walkerd]

@stevenscg At the moment (pre-docker and Nomad), I have lots of cron jobs for starting and killing applications outside of business hours.. I'd like to get away from this... but it is currently really hard to model with Nomad without hacking/fudging it with the description I outlined.

In addition, I could see a use-case for different teams (different apps) having reserved resources between certain hours to make better use of resources.

[dadgar]

@redi-walkerd For now I would probably just run a cron that ups/downs the count of the job during the hours you need

[KZachariassen]

We would like this as well. It would be grate if nomad could cron schedule a docker image, so we dont need to build cron into all our images.

[simon-graham]

We also have a large number of services that should only be run during business hours. During business hours we'd want to retain existing nomad service capabilities such as restarting on task failure, but outside of business hours the service should be prevented from running.
Speaking of business hours, a business calendar would also be extremely useful, so we can prevent batch jobs and services from running on holidays. However that's a whole separate issue. I mention it here because it may impact how a feature like this is implemented.

[danbst]

@dadgar

For now I would probably just run a cron that ups/downs the count of the job during the hours you need

When using .hcl Nomadfiles - should I do those changes with sed on Nomadfiles? Or there is better alternative?

[ctc-santanaa]

We're also interested in this functionality. Scheduling business hours would be a huge plus for systems that connect to outside services that have defined maintenance systems, like financial exchange gateways.

[neomantra]

Noting that the Nomad Autoscaler has an issue which would cover this use case:
hashicorp/nomad-autoscaler#448

There's no code in that issue, but there's a reference to a custom plugin repo with a "cron" scheduler:
https://github.com/jsiebens/nomad-autoscaler-plugins

[charnz]

We are also looking for such a solution as it is a potential snag in our migration to Nomad. Our use case is similar to @ctc-santanaa operating in a financial services context where there is a well defined "end of day" which can vary per application and region. I have come across some work-arounds but all create a barrier to entry for us to migrate our existing jobs.

[sebcoetzee]

Can confirm that this would be a very useful feature for our use-case: trading applications that are only required to be up and running during market hours.

[miketzian]

This is for sure a missing feature. Being able to plugin different calendars/periodic configs (ala airflow timetable) would make this a lot more useful for batch use cases vs other legacy competitors

[tgross]

This was implemented via the schedule block in Nomad Enterprise 1.8.0.

[Daviey]

I'm not seeing the fix in the BUSL'd version, can we keep this issue open until that capability is present? Considering I opened this issue nearly 8 years ago, I can keep waiting a bit longer. Thanks

[tgross]

We implemented in Nomad Enterprise intentionally, as this sort of "governance" feature falls on a boundary line of where we decide to build in ENT vs CE editions. That being said, I'm remembering we still have it marked as "experimental", so maybe we'll see the light on that and get it into CE at some point (I can't promise anything on that front though). Re-opening for the time being.