We currently do not offer any abstractions for the Job or CronJobs resources on Kubernetes.
Setting up a Job
or CronJob
on NAIS requires a more hands-on approach compared to other parts of NAIS.
We'll guide you through it below.
Jobs and CronJobs are only allowed to run in your team's own namespace.
variable | description | example |
---|---|---|
${jobname} |
name for the job or cronjob | my-job |
${teamname} |
name of the team | my-team |
${namespace} |
name of the namespace, should match the team name | my-team |
${image} |
image used for the pods running the job | docker.pkg.github.com/navikt/my-app/my-app:1.0.0 |
${schedule} |
the job's schedule in a cron time string format | "*/15 * * * *" |
${secretname} |
name of the [Kubernetes Secret] | my-secret |
${vault_image} |
the latest release of the Vault sidecar | navikt/vault-sidekick:v0.3.10-d122b16 |
${envclass} |
one of prod or preprod |
prod |
${zone} |
one of fss or sbs |
fss |
${vault_login} |
Vault auth path from Kubernetes | auth/kubernetes/${envclass}/${zone}/login |
${vault_kv_path} |
the path to the secret in the Vault kv engine |
kv/${envclass}/${zone}/my-job/my-team |
{% code-tabs %}
{% code-tabs-item title="Basic", type="yaml" %}
apiVersion: batch/v1
kind: Job
metadata:
name: ${jobname}
namespace: ${teamname}
labels:
team: ${teamname}
spec:
ttlSecondsAfterFinished: 300
backoffLimit: 1
template:
spec:
imagePullSecrets:
- name: gpr-credentials
serviceAccount: default
serviceAccountName: default
restartPolicy: Never
containers:
- name: ${jobname}
image: ${image}
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 1024Mi
cpu: 1000m
{% endcode-tabs-item %}
{% code-tabs-item title="w/Kubernetes Secret", type="yaml" %}
apiVersion: batch/v1
kind: Job
metadata:
name: ${jobname}
namespace: ${teamname}
labels:
team: ${teamname}
spec:
ttlSecondsAfterFinished: 300
backoffLimit: 1
template:
spec:
imagePullSecrets:
- name: gpr-credentials
serviceAccount: default
serviceAccountName: default
restartPolicy: Never
containers:
- name: ${jobname}
image: ${image}
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 1024Mi
cpu: 1000m
envFrom:
- secretRef:
name: ${secretname}
{% endcode-tabs-item %}
{% code-tabs-item title="w/CA Bundles", type="yaml" %}
apiVersion: batch/v1
kind: Job
metadata:
name: ${jobname}
namespace: ${teamname}
labels:
team: ${teamname}
spec:
ttlSecondsAfterFinished: 300
backoffLimit: 1
template:
spec:
imagePullSecrets:
- name: gpr-credentials
serviceAccount: default
serviceAccountName: default
restartPolicy: Never
containers:
- name: ${jobname}
image: ${image}
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 1024Mi
cpu: 1000m
volumeMounts:
- mountPath: /etc/ssl/certs/ca-certificates.crt
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/pki/tls/certs/ca-bundle.crt
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/ssl/ca-bundle.pem
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/pki/tls/cacert.pem
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/ssl/certs/java/cacerts
name: ca-bundle-jks
readOnly: true
subPath: ca-bundle.jks
env:
- name: NAV_TRUSTSTORE_PATH
value: /etc/ssl/certs/java/cacerts
- name: NAV_TRUSTSTORE_PASSWORD
value: changeme
volumes:
- configMap:
defaultMode: 420
name: ca-bundle-jks
name: ca-bundle-jks
- configMap:
defaultMode: 420
name: ca-bundle-pem
name: ca-bundle-pem
{% endcode-tabs-item %}
{% code-tabs-item title="w/Vault integration", type="yaml" %}
apiVersion: batch/v1
kind: Job
metadata:
name: ${jobname}
namespace: ${teamname}
labels:
team: ${teamname}
spec:
ttlSecondsAfterFinished: 300
backoffLimit: 1
template:
spec:
imagePullSecrets:
- name: gpr-credentials
serviceAccount: default
serviceAccountName: default
restartPolicy: Never
containers:
- name: ${jobname}
image: ${image}
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 1024Mi
cpu: 1000m
volumeMounts:
- mountPath: /var/run/secrets/nais.io/vault
name: vault-secrets
subPath: vault/var/run/secrets/nais.io/vault
env:
- name: VAULT_TOKEN_PATH
value: /var/run/secrets/nais.io/vault/vault_token
initContainers:
- name: vks-init
image: ${vault_image}
resources:
requests:
memory: "64Mi"
cpu: "100m"
limits:
memory: "128Mi"
cpu: "1000m"
volumeMounts:
- mountPath: /var/run/secrets/nais.io/vault
name: vault-secrets
subPath: vault/var/run/secrets/nais.io/vault
args:
- -v=10
- -logtostderr
- -vault=https://vault.adeo.no
- -one-shot
- -save-token=/var/run/secrets/nais.io/vault/vault_token
- -cn=secret:${vault_kv_path}:dir=/var/run/secrets/nais.io/vault,fmt=flatten,retries=1
env:
- name: VAULT_AUTH_METHOD
value: kubernetes
- name: VAULT_SIDEKICK_ROLE
value: ${teamname}
- name: VAULT_K8S_LOGIN_PATH
value: ${vault_login}
volumes:
- name: vault-secrets
emptyDir:
medium: Memory
{% endcode-tabs-item %}
{% code-tabs-item title="w/CA Bundles and Vault", type="yaml" %}
apiVersion: batch/v1
kind: Job
metadata:
name: ${jobname}
namespace: ${teamname}
labels:
team: ${teamname}
spec:
ttlSecondsAfterFinished: 300
backoffLimit: 1
template:
spec:
imagePullSecrets:
- name: gpr-credentials
serviceAccount: default
serviceAccountName: default
restartPolicy: Never
containers:
- name: ${jobname}
image: ${image}
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 1024Mi
cpu: 1000m
volumeMounts:
- mountPath: /etc/ssl/certs/ca-certificates.crt
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/pki/tls/certs/ca-bundle.crt
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/ssl/ca-bundle.pem
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/pki/tls/cacert.pem
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/ssl/certs/java/cacerts
name: ca-bundle-jks
readOnly: true
subPath: ca-bundle.jks
- mountPath: /var/run/secrets/nais.io/vault
name: vault-secrets
subPath: vault/var/run/secrets/nais.io/vault
env:
- name: NAV_TRUSTSTORE_PATH
value: /etc/ssl/certs/java/cacerts
- name: NAV_TRUSTSTORE_PASSWORD
value: changeme
- name: VAULT_TOKEN_PATH
value: /var/run/secrets/nais.io/vault/vault_token
initContainers:
- name: vks-init
image:
resources:
requests:
memory: "64Mi"
cpu: "100m"
limits:
memory: "128Mi"
cpu: "1000m"
volumeMounts:
- mountPath: /var/run/secrets/nais.io/vault
name: vault-secrets
subPath: vault/var/run/secrets/nais.io/vault
args:
- -v=10
- -logtostderr
- -vault=https://vault.adeo.no
- -one-shot
- -save-token=/var/run/secrets/nais.io/vault/vault_token
- -cn=secret:${vault_kv_path}:dir=/var/run/secrets/nais.io/vault,fmt=flatten,retries=1
env:
- name: VAULT_AUTH_METHOD
value: kubernetes
- name: VAULT_SIDEKICK_ROLE
value: ${teamname}
- name: VAULT_K8S_LOGIN_PATH
value: ${vault_login}
volumes:
- configMap:
defaultMode: 420
name: ca-bundle-jks
name: ca-bundle-jks
- configMap:
defaultMode: 420
name: ca-bundle-pem
name: ca-bundle-pem
- name: vault-secrets
emptyDir:
medium: Memory
{% endcode-tabs-item %}
{% endcode-tabs %}
{% hint style="info" %}
Note that .spec.template.spec.imagePullSecrets
can be removed if your image is not hosted at Github Package Registry
{% endhint %}
A CronJob
runs Jobs
on a time-based schedule, as denoted in .spec.schedule
.
The .spec.jobTemplate
is the template for the job, and has exactly the same schema as the spec
in Job
.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: ${jobname}
namespace: ${teamname}
labels:
team: ${teamname}
spec:
schedule: ${schedule}
jobTemplate:
spec:
<same as .spec from Jobs examples earlier>
Reaching internal services such as Kafka requires the presence of a truststore that includes the certificates for these services.
Looking at the differences between the Basic
and the w/CA Bundles
examples,
we spot the following:
.spec.template.spec.volumes[]
spec:
template:
spec:
...
volumes:
- configMap:
defaultMode: 420
name: ca-bundle-jks
name: ca-bundle-jks
- configMap:
defaultMode: 420
name: ca-bundle-pem
name: ca-bundle-pem
Fetches CA bundles from ConfigMaps
and injects it into the pod.
.spec.template.spec.containers[0].volumeMounts[]
spec:
template:
spec:
containers:
- name: ${jobname}
image: ${image}
...
volumeMounts:
- mountPath: /etc/ssl/certs/ca-certificates.crt
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/pki/tls/certs/ca-bundle.crt
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/ssl/ca-bundle.pem
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/pki/tls/cacert.pem
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
name: ca-bundle-pem
readOnly: true
subPath: ca-bundle.pem
- mountPath: /etc/ssl/certs/java/cacerts
name: ca-bundle-jks
readOnly: true
subPath: ca-bundle.jks
The files from step 1 found in subPath
are mounted to the path specified in mountPath
for the container of your application.
The example covers the common locations for these bundles for most systems.
.spec.template.spec.containers[0].env[]
spec:
template:
spec:
containers:
- name: ${jobname}
image: ${image}
env:
- name: NAV_TRUSTSTORE_PATH
value: /etc/ssl/certs/java/cacerts
- name: NAV_TRUSTSTORE_PASSWORD
value: changeme
Java applications using the NAV baseimages may specify environment variables to automatically inject the bundles/truststore into the JVM.
If your job needs secrets that are not found in Vault, using the native Kubernetes Secrets functionality is much simpler than utilizing Vault.
This assumes that you have already created a Secret.
Looking at the differences between the Basic
and the w/Kubernetes Secret
examples,
we spot the following:
.spec.template.spec.containers[0].envFrom
spec:
template:
spec:
...
containers:
- name: ${jobname}
image: ${image}
...
envFrom:
- secretRef:
name: ${secretname}
.spec.template.spec.containers[0].envFrom[]
exports all the Secrets referenced
(in envFrom[].secretRef.name
) as environment variables for the container.
Each key from each Secret
becomes the environment variable name in the container.
If your job needs to access Vault for fetching secrets or uses PostgreSQL on-premises, you'll also have to set up Vault integration for your job.
To be able to retrieve secrets from Vault, you need to add permissions for the Kubernetes service account.
Go to Vault IaC and add the following changes to your
terraform/teams/${teamname}/apps/${jobname}.yml
file:
name: ${jobname}
cluster:
${cluster}:
serviceaccounts:
- default
Looking at the differences between the Basic
and the w/Vault integration
examples,
we spot the following:
.spec.template.spec.volumes[]
spec:
ttlSecondsAfterFinished: 300
backoffLimit: 1
template:
spec:
...
volumes:
- name: vault-secrets
emptyDir:
medium: Memory
Sets up an empty in-memory volume mount for the pod. This is where the secrets will be stored in your pod. They will not be written to disk.
.spec.template.spec.initContainers[0]
spec:
template:
spec:
...
initContainers:
- name: vks-init
image: ${image}
...
volumeMounts:
- mountPath: /var/run/secrets/nais.io/vault
name: vault-secrets
subPath: vault/var/run/secrets/nais.io/vault
args:
- -v=10
- -logtostderr
- -vault=https://vault.adeo.no
- -one-shot
- -save-token=/var/run/secrets/nais.io/vault/vault_token
- -cn=secret:${vault_kv_path}:dir=/var/run/secrets/nais.io/vault,fmt=flatten,retries=1
env:
- name: VAULT_AUTH_METHOD
value: kubernetes
- name: VAULT_SIDEKICK_ROLE
value: ${teamname}
- name: VAULT_K8S_LOGIN_PATH
value: ${vault_login}
This specifies a container that should start and run before your application container starts up.
In this case, the container is responsible for fetching secrets from Vault before startup, as seen in
.spec.template.spec.initContainers[0].args[]
:
- -save-token=/var/run/secrets/nais.io/vault/vault_token
Saves the Vault token to this path for later usage if desired.
- -cn=secret:${vault_kv_path}:dir=/var/run/secrets/nais.io/vault,fmt=flatten,retries=1
Reads a secret from ${vault_kv_path}
and outputs the files to the directory /var/run/secrets/nais.io/vault/
in the volume mount.
The volume for the Pod described in step 2.1 is mounted to the init container as seen in
.spec.template.spec.initContainers[0].volumeMounts[0]
:
spec:
template:
spec:
...
initContainers:
- name: vks-init
image: ${image}
...
volumeMounts:
- mountPath: /var/run/secrets/nais.io/vault
name: vault-secrets
subPath: vault/var/run/secrets/nais.io/vault
.spec.template.spec.containers[0]
spec:
template:
spec:
containers:
- name: ${jobname}
image: ${image}
...
volumeMounts:
- mountPath: /var/run/secrets/nais.io/vault
name: vault-secrets
subPath: vault/var/run/secrets/nais.io/vault
The files from step 2.3 found in subPath
are mounted to the path specified in mountPath
for the container of your application.
.spec.template.spec.containers[0].env[]
spec:
template:
spec:
containers:
- name: ${jobname}
image: ${image}
...
env:
- name: VAULT_TOKEN_PATH
value: /var/run/secrets/nais.io/vault/vault_token
If your application uses an environment variable to locate the Vault token path.
Repeat the steps below for each secret you want from Vault.
Add a new volumeMount
entry to the list in:
.spec.template.initContainers[0].volumeMounts[]
.spec.template.containers[0].volumeMounts[]
spec:
template:
spec:
...
containers:
- name: ${jobname}
image: ${image}
...
volumeMounts:
- mountPath: ${mount_path}
name: vault-secrets
subPath: vault/${mount_path}
initContainers:
- name: vks-init
image: ${image}
...
volumeMounts:
- mountPath: ${mount_path}
name: vault-secrets
subPath: vault/${mount_path}
where ${mount_path}
is the path to where the secret should be stored and accessed by your application container.
For example, if you want to fetch a serviceuser
:
spec:
template:
spec:
...
containers:
- name: ${jobname}
image: ${image}
...
volumeMounts:
- mountPath: /var/run/secrets/nais.io/${srvUser}
name: vault-secrets
subPath: vault/var/run/secrets/nais.io/${srvUser}
initContainers:
- name: vks-init
image: ${image}
...
volumeMounts:
- mountPath: /var/run/secrets/nais.io/${srvUser}
name: vault-volume
subPath: vault/var/run/secrets/nais.io/${srvUser}
Now that you've specified the mount path for the secret, you'll need to specify where the secret is found in Vault, and
Add a new entry to the list of .spec.template.initContainers[0].args
spec:
template:
spec:
...
initContainers:
- name: vks-init
image: ${image}
...
args:
...
- -cn=secret:${kv_path}:dir=${mount_path},fmt=flatten,retries=1
where
${kv_path}
is the location of the Vault secret${mount_path}
is themountPath
that you specified in step 3.1
For example:
spec:
template:
spec:
...
initContainers:
- name: vks-init
image: ${image}
...
args:
...
- -cn=secret:/serviceuser/data/prod/srv-my-app:dir=/var/run/secrets/nais.io/srv-my-app,fmt=flatten,retries=1
When all the aforementioned steps have been completed, you're finally ready to deploy your Job
or CronJob
to the cluster.
While you may run your job as part of a pipeline with NAIS deploy, sometimes you might just want to run a one-time job.
The easiest way to do so is to simply apply
the Job
or CronJob
to the cluster:
$ kubectl apply -f job.yml