kubeaudit is a command line tool to audit Kubernetes clusters for various
different security concerns: run the container as a non-root user, use a read
only root filesystem, drop scary capabilities, don't add new ones, don't run
privileged, ... You get the gist of it and more on that later. Just know:
Kubeaudit has official releases that are blessed and stable here: Official releases
Master will have newer features than the stable releases. If you need a newer feature not yet included in a release you can do the following to get kubeaudit:
For go 1.12 and higher:
GO111MODULE=on go get -v github.com/Shopify/kubeauditFor older versions of go:
git clone https://github.com/Shopify/kubeaudit.git
cd kubeaudit
make
make installNow you can just call kubeaudit with one of commands from here
Prerequisite: kubectl v1.12.0 or later
With kubectl v1.12.0 introducing easy pluggability of external functions, kubeaudit can be invoked as kubectl audit just by
- running
make pluginand having $GOPATH/bin available in your path.
or
- renaming the binary to
kubectl-auditand having it available in your path.
kubeaudit has three different modes for its audits:
- Cluster mode
If kubeaudit detects that it's running in a container,
kubeaudit cmdwill attempt to audit the cluster it's running in. - Local config mode
If kubeaudit is not running in a container,
kubeaudit cmdwill audit the resources specified by your local kubeconfig ($HOME/.kube/config) file. You can force kubeaudit to use a specific local config file with the switch-c/--kubeconfig /config/path - Manifest mode
If you wish to audit a manifest file, use the command
kubeaudit -f/--manifest /path/to/manifest.yml
kubeaudit supports two different output types:
- just running
kubeauditwill log human readable output - if run with
-j/--jsonit will log output json formatted so that its output can be used by other programs easily
kubeaudit supports using manual audit configuration provided by the user, use the command
kubeaudit -f/--manifest /path/to/manifest.yml -k/--auditconfig /path/to/config.yml
For more details on audit config check out Audit Configuration.
kubeaudit has four different log levels INFO, WARN, ERROR controlled by
-v/--verbose LEVEL and for those who counted and want to work on kubeaudit
DEBUG
- by default the debug level is set to
ERRORand will logINFO,WARNandERROR - if you only care about
ERRORset it toERROR - if you care about
ERRORandWARNset it toWARN
But wait! Which version am I actually running? kubeaudit version will tell you.
I need help! Run kubeaudit help every audit has its own help so you can run
kubeaudit help sc
Last but not least before we look at the audits: kubeaudit -a/--allPods
audits against pods in all the phases (default Running Phase)
As humans we are lazy and kubeaudit knows that so it comes with the functionality to autofix workload manifests. Point it at your workload manifests and it will automagically fix everything so that manifests are as secure as it gets.
kubeaudit autofix -f path/to/manifest.yml
The manifest might end up a little too secure for the work it is supposed to do. If that is the case check out labels to opt out of certain checks.
kubeaudit has multiple checks:
- Audit all
- Audit security context
- Audit image
- Audit Service Accounts
- Audit network policies
- Audit resources
- Audit mounting Docker Socket
- Audit AppArmor
- Audit Seccomp
- Audit namespaces
Runs all the above checks.
kubeaudit all
ERRO[0000] RunAsNonRoot is not set, which results in root user being allowed!
ERRO[0000] Default serviceAccount with token mounted. Please set automountServiceAccountToken to false
WARN[0000] Privileged defaults to false, which results in non privileged, which is okay.
ERRO[0000] Capability not dropped CapName=AUDIT_WRITEThe security context holds a couple of different security related
configurations. For convenience, kubeaudit will always log the following
information when it creates a log:
kubeaudit command
LOG[0000] KubeType=deployment Name=THEdeployment Namespace=deploymentNSAnd for brevity, the information will not be shown in the commands below.
Currently, kubeaudit is able to check for the following fields in the security context:
kubeaudit will detect whether readOnlyRootFilesystem is either not set nil or explicitly set to false
kubeaudit rootfs
ERRO[0000] ReadOnlyRootFilesystem not set which results in a writable rootFS, please set to true
ERRO[0000] ReadOnlyRootFilesystem set to false, please set to truekubeaudit will detect whether the container is to be run as root:
kubeaudit nonroot
ERRO[0000] RunAsNonRoot is set to false (root user allowed), please set to true!
ERRO[0000] RunAsNonRoot is not set, which results in root user being allowed!kubeaudit will detect whether allowPrivilegeEscalation is either set to nil or explicitly set to false
kubeaudit allowpe
ERRO[0000] AllowPrivilegeEscalation set to true, please set to false
ERRO[0000] AllowPrivilegeEscalation not set which allows privilege escalation, please set to falsekubeaudit will detect whether the container is to be run privileged:
kubeaudit priv
ERRO[0000] Privileged set to true! Please change it to false!Since we want to make sure everything is intentionally configured correctly kubeaudit warns about privileged not being set:
kubeaudit priv
WARN[0000] Privileged defaults to false, which results in non privileged, which is okay.Docker comes with a couple of capabilities that shouldn't be needed and
therefore should be dropped. kubeaudit will also complain about added capabilities.
If the capabilities field doesn't exist within the security context:
kubeaudit caps
ERRO[0000] Capabilities field not defined!When capabilities were added:
kubeaudit caps
ERRO[0000] Capability added CapName=NET_ADMINconfig/caps
holds a list of capabilities that we recommend be dropped, change it if you
want to keep some of the capabilities otherwise kubeaudit will complain about
them not being dropped:
kubeaudit caps
ERRO[0000] Capability not dropped CapName=AUDIT_WRITEkubeaudit can check for image names and image tags:
-
If the image tag is incorrect an ERROR will issued
kubeaudit image -i gcr.io/google_containers/echoserver:1.7 ERRO[0000] Image tag was incorrect
-
If the image doesn't have a tag but an image of the name was found a WARNING will be created:
kubeaudit image -i gcr.io/google_containers/echoserver:1.7 WARN[0000] Image tag was missing
-
If the image was found with correct tag
kubeauditnotifies with an INFO message:kubeaudit image -i gcr.io/google_containers/echoserver:1.7 INFO[0000] Image tag was correct
It audits against the following scenarios:
-
A default serviceAccount mounted with a token:
kubeaudit sat ERRO[0000] Default serviceAccount with token mounted. Please set AutomountServiceAccountToken to false
-
A deprecated service account:
kubeaudit sat WARN[0000] serviceAccount is a deprecated alias for ServiceAccountName, use that one instead DSA=DeprecatedServiceAccount
It checks that every namespace should have a default deny network policy installed. See Kubernetes Network Policies for more information:
kubeaudit np
WARN[0000] Default allow mode on test/testingIt checks that every resource has a CPU and memory limit. See Kubernetes Resource Quotas for more information:
kubeaudit limits
WARN[0000] CPU limit not set, please set it!
WARN[0000] Memory limit not set, please set it!With the --cpu and --memory parameters, kubeaudit can check the limits not to be exceeded.
kubeaudit limits --cpu 500m --memory 125Mi
WARN[0000] CPU limit exceeded, it is set to 1 but it must not exceed 500m. Please adjust it! !
WARN[0000] Memory limit exceeded, it is set to 512Mi but it must not exceed 125Mi. Please adjust it!It checks that no container in the pod mounts /var/run/docker.sock, as this can be a very dangerous practice.
If a container does this, it will be indicated as such:
containers:
- image: <image name>
name: <container name>
volumeMounts:
- mountPath: /var/run/docker.sock
name: <volume name>
volumes:
- name: <volume name>
hostPath:
path: /var/run/docker.sock
If /var/run/docker.sock is being mounted by a container:
kubeaudit mountds
WARN[0000] /var/run/docker.sock is being mounted, please avoid this practice. Container=myContainer KubeType=pod Name=myPod Namespace=myNamespaceIt checks that AppArmor is enabled for all containers by making sure the following annotation exists on the pod. There must be an annotation for each container in the pod:
container.apparmor.security.beta.kubernetes.io/<container name>: <profile>
where profile can be "runtime/default" or start with "localhost/" to be considered valid.
If the AppArmor annotation is missing:
kubeaudit apparmor
ERRO[0000] AppArmor annotation missing. Container=myContainer KubeType=pod Name=myPod Namespace=myNamespaceWhen AppArmor annotations are misconfigured:
kubeaudit apparmor
ERRO[0000] AppArmor disabled. Annotation=container.apparmor.security.beta.kubernetes.io/myContainer
Container=myContainer KubeType=pod Name=myPod Namespace=myNamespace Reason=badvalIt checks that Seccomp is enabled for all containers by making sure one or both of the following annotations exists on the pod. If no pod annotation is used, then there must be an annotation for each container. Container annotations override the pod annotation:
# pod annotation
seccomp.security.alpha.kubernetes.io/pod: <profile>
# container annotation
container.seccomp.security.alpha.kubernetes.io/<container name>: <profile>
where profile can be "runtime/default" or start with "localhost/" to be considered valid. "docker/default" is deprecated and will show a warning. It should be replaced with "runtime/default".
If the Seccomp annotation is missing:
kubeaudit seccomp
ERRO[0000] Seccomp annotation missing. Container=myContainer KubeType=pod Name=myPod Namespace=myNamespaceWhen Seccomp annotations are misconfigured for a container:
kubeaudit seccomp
ERRO[0000] Seccomp disabled for container. Annotation=container.seccomp.security.alpha.kubernetes.io/myContainer
Container=myContainer KubeType=pod Name=myPod Namespace=myNamespace Reason=badvalWhen Seccomp annotations are misconfigured for a pod:
kubeaudit seccomp
ERRO[0000] Seccomp disabled for pod. Annotation=seccomp.security.alpha.kubernetes.io/pod Container= KubeType=pod
Name=myPod Namespace=myNamespace Reason=unconfinedkubeaudit will detect whether hostNetwork,hostIPC or hostPID is either set to true in podSpec for Pod workloads
kubeaudit namespaces
ERRO[0000] hostNetwork is set to true in podSpec, please set to false!
ERRO[0000] hostIPC is set to true in podSpec, please set to false!
ERRO[0000] hostPID is set to true in podSpec, please set to false!Override labels give you the ability to have kubeaudit allow certain audits to fail.
For example, if you want kubeaudit to ignore the fact that AllowPrivilegeEscalation was set to true, you can add the following label:
spec:
template:
metadata:
labels:
apps: YourAppNameHere
container.audit.kubernetes.io/<container-name>.allow-privilege-escalation: "YourReasonForOverrideHere"Any label with a non-nil reason string will prevent kubeaudit from throwing the corresponding error and issue a warning instead. Note that the reason may only contain alphanumeric characters and ., _, -, per Kubernetes label specification.
Reasons matching "true" (not case sensitive) will be displayed as Unspecified.
kubeaudit can skip certain audits by applying override labels to containers. If you want skip an audit for a specific container inside a pod, you can add an container override label. For example, if you use kubeaudit to ignore allow-run-as-root check for container "MyContainer1", you can add the following label:
spec:
template:
metadata:
labels:
apps: YourAppNameHere
container.audit.kubernetes.io/MyContainer1.allow-run-as-root: "YourReasonForOverrideHere"Similarly, you can have kubeaudit to skip a specific audit for all containers inside the pod by adding a pod override label. For example, if you use kubeaudit to ignore allow-run-as-root check for all containers inside the pod, you can add the following label:
spec:
template:
metadata:
labels:
apps: YourAppNameHere
audit.kubernetes.io/pod.allow-run-as-root: "YourReasonForOverrideHere"kubeaudit can also skip a specific audit for all network policies associated with a namespace resource
by adding a namespace override label. For example, if you use kubeaudit to ignore the allow-non-default-deny-egress-network-policy check for the namespace namespaceName1 you can add the following label to the namespace:
metadata:
name: namespaceName1
labels:
audit.kubernetes.io/namespaceName1.allow-non-default-deny-egress-network-policy: "YourReasonForOverrideHere"kubeaudit supports many labels on pod, namespace or container level:
- audit.kubernetes.io/pod.allow-privilege-escalation
- container.audit.kubernetes.io/<container-name>.allow-privilege-escalation
- audit.kubernetes.io/pod.allow-privileged
- container.audit.kubernetes.io/<container-name>.allow-privileged
- audit.kubernetes.io/pod.allow-capability
- container.audit.kubernetes.io/<container-name>.allow-capability
- audit.kubernetes.io/pod.allow-run-as-root
- container.audit.kubernetes.io/<container-name>.allow-run-as-root
- audit.kubernetes.io/pod.allow-automount-service-account-token
- audit.kubernetes.io/pod.allow-read-only-root-filesystem-false
- container.audit.kubernetes.io/<container-name>.allow-read-only-root-filesystem-false
- audit.kubernetes.io/<namespace-name>.allow-non-default-deny-egress-network-policy
- audit.kubernetes.io/<namespace-name>.allow-non-default-deny-ingress-network-policy
- audit.kubernetes.io/pod.allow-namespace-host-network
- audit.kubernetes.io/pod.allow-namespace-host-IPC
- audit.kubernetes.io/pod.allow-namespace-host-PID
Allow allowPrivilegeEscalation to be set to true to a specific container.
Allows allowPrivilegeEscalation to be set to true to all the containers in a pod.
kubeaudit.allow.privilegeEscalation: "SuperuserPrivilegesNeeded"
WARN[0000] Allowed setting AllowPrivilegeEscalation to true Reason="SuperuserPrivilegesNeeded"Allow privileged to be set to true to a specific container.
Allows privileged to be set to true to all the containers in a pod.
kubeaudit.allow.privileged: "PrivilegedExecutionRequired"
WARN[0000] Allowed setting privileged to true Reason="PrivilegedExecutionRequired"Allows adding a capability or keeping one that would otherwise be dropped to a specific container.
Allows adding a capability or keeping one that would otherwise be dropped to all the containers in a pod.
kubeaudit.allow.capability.chown: "true"
WARN[0000] Capability allowed CapName=CHOWN Reason=UnspecifiedAllows setting runAsNonRoot to false to a specific container.
Allows setting runAsNonRoot to false to all the containers in a pod.
kubeaudit.allow.runAsRoot: "RootPrivilegesNeeded"
WARN[0000] Allowed setting RunAsNonRoot to false Reason="RootPrivilegesNeeded"Allows setting automountServiceAccountToken to true to a pod.
kubeaudit.allow.autmountServiceAccountToken: "True"
WARN[0000] Allowed setting automountServiceAccountToken to true Reason=UnspecifiedAllows setting readOnlyRootFilesystem to false to a specific container.
Allows setting readOnlyRootFilesystem to false to all containers in a pod.
kubeaudit.allow.readOnlyRootFilesystemFalse: "WritePermissionsNeeded"
WARN[0000] Allowed setting readOnlyRootFilesystem to false Reason="WritePermissionsNeeded"Allows absense of default-deny egress network policy for that specific namespace.
Allows absense of default-deny ingress network policy for that specific namespace.
audit.kubernetes.io/default.allow-non-default-deny-egress-network-policy: "EgressIsAllowed"
WARN[0000] Allowed Namespace without a default deny egress NetworkPolicy KubeType=namespace Name=default Reason="EgressIsAllowed"audit.kubernetes.io/pod.allow-namespace-host-network: "HostNetworkIsAllowed"
WARN[0000] Allowed setting hostNetwork to true KubeType=pod Name=Pod Namespace=PodNamespace Reason="HostNetworkIsAllowed"audit.kubernetes.io/pod.allow-namespace-host-IPC: "HostIPCIsAllowed"
WARN[0000] Allowed setting hostIPC to true KubeType=pod Name=Pod Namespace=PodNamespace Reason="HostIPCIsAllowed"audit.kubernetes.io/pod.allow-namespace-host-PID: "HOSTPIDIsAllowed"
WARN[0000] Allowed setting hostPID to true KubeType=pod Name=Pod Namespace=PodNamespace Reason="HOSTPIDIsAllowed"Allows configuring the audit against drop capabilities. Sane defaults are as follows:
# SANE DEFAULTS:
capabilitiesToBeDropped:
# https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
- SETPCAP #Modify process capabilities.
- MKNOD #Create special files using mknod(2).
- AUDIT_WRITE #Write records to kernel auditing log.
- CHOWN #Make arbitrary changes to file UIDs and GIDs (see chown(2)).
- NET_RAW #Use RAW and PACKET sockets.
- DAC_OVERRIDE #Bypass file read, write, and execute permission checks.
- FOWNER #Bypass permission checks on operations that normally require the file system UID of the process to match the UID of the file.
- FSETID #Don’t clear set-user-ID and set-group-ID permission bits when a file is modified.
- KILL #Bypass permission checks for sending signals.
- SETGID #Make arbitrary manipulations of process GIDs and supplementary GID list.
- SETUID #Make arbitrary manipulations of process UIDs.
- NET_BIND_SERVICE #Bind a socket to internet domain privileged ports (port numbers less than 1024).
- SYS_CHROOT #Use chroot(2), change root directory.
- SETFCAP #Set file capabilities.
This can be overridden by using -k flag and providing your own defaults in the yaml format as shown below.
Allows configuring your own audit settings for kubeaudit. By default following configuration is used:
apiVersion: v1
kind: kubeauditConfig
audit: true # Set to false if you want kubeaudit to not audit your k8s manifests
spec:
capabilities: # List of all supported capabilities
NET_ADMIN: drop # Set to `keep` to keep capability
SETPCAP: drop # Set to `keep` to keep capability
MKNOD: drop # Set to `keep` to keep capability
AUDIT_WRITE: drop # Set to `keep` to keep capability
CHOWN: drop # Set to `keep` to keep capability
NET_RAW: drop # Set to `keep` to keep capability
DAC_OVERRIDE: drop # Set to `keep` to keep capability
FOWNER: drop # Set to `keep` to keep capability
FSETID: drop # Set to `keep` to keep capability
KILL: drop # Set to `keep` to keep capability
SETGID: drop # Set to `keep` to keep capability
SETUID: drop # Set to `keep` to keep capability
NET_BIND_SERVICE: drop # Set to `keep` to keep capability
SYS_CHROOT: drop # Set to `keep` to keep capability
SETFCAP: drop # Set to `keep` to keep capability
overrides: # List of all supported overrides
privilege-escalation: deny # Set to `allow` to skip auditing potential vulnerability
privileged: deny # Set to `allow` to skip auditing potential vulnerability
run-as-root: deny # Set to `allow` to skip auditing potential vulnerability
automount-service-account-token: deny # Set to `allow` to skip auditing potential vulnerability
read-only-root-filesystem-false: deny # Set to `allow` to skip auditing potential vulnerability
non-default-deny-ingress-network-policy: deny # Set to `allow` to skip auditing potential vulnerability
non-default-deny-egress-network-policy: deny # Set to `allow` to skip auditing potential vulnerability
namespace-host-network: deny # Set to `allow` to skip auditing potential vulnerability
namespace-host-IPC: deny # Set to `allow` to skip auditing potential vulnerability
namespace-host-PID: deny # Set to `allow` to skip auditing potential vulnerability
If you'd like to fix a bug, contribute a feature or just correct a typo, please feel free to do so as long as you follow our Code of Conduct.
- Create your own fork!
- Get the source:
go get github.com/Shopify/kubeaudit - Go to the source:
cd $GOPATH/src/github.com/Shopify/kubeaudit - Add your forked repo as a fork:
git remote add fork https://github.com/you-are-awesome/kubeaudit - Create your feature branch:
git checkout -b awesome-new-feature - Run the tests to see everything is working as expected:
make test - Commit your changes:
git commit -am 'Adds awesome feature' - Push to the branch:
git push fork - Sign the Contributor License Agreement
- Submit a PR (All PR must be labeled with 🐛 (Bug fix), ✨ (New feature), 📖 (Documentation update), or
⚠️ (Breaking changes) ) - ???
- Profit
Note that if you didn't sign the CLA before opening your PR, you can re-run the check by adding a comment to the PR that says "I've signed the CLA!"!