Best Practices

Policies for Kubernetes best practices.

Add Network Policy

By default, Kubernetes allows communications across all pods within a cluster. Network policies and, a CNI that supports network policies, must be used to restrict communications. A default NetworkPolicy should be configured for each namespace to default deny all ingress and egress traffic to the pods in the namespace. Application teams can then configure additional NetworkPolicy resources to allow desired traffic to application pods from select sources.

Add Quota

To limit the number of objects, as well as the total amount of compute that may be consumed by a single namespace, create a default resource quota for each namespace.

Add safe-to-evict

The Kubernetes cluster autoscaler does not evict pods that use hostPath or emptyDir volumes. To allow eviction of these pods, the annotation cluster-autoscaler.kubernetes.io/safe-to-evict=true must be added to pods.

Disallow CRI socket mounts

Container daemon socket bind mounts allows access to the container engine on the node. This access can be used for privilege escalation and to manage containers outside of Kubernetes, and hence should not be allowed.

Disallow Default Namespace

Kubernetes namespaces are an optional feature that provide a way to segment and isolate cluster resources across multiple applications and users. As a best practice, workloads should be isolated with namespaces. Namespaces should be required and the default (empty) namespace should not be used.

Disallow Helm Tiller

Tiller has known security challenges. It requires administrative privileges and acts as a shared resource accessible to any authenticated user. Tiller can lead to privilege escalation as restricted users can impact other users.

Disallow Latest Tag

The ‘:latest’ tag is mutable and can lead to unexpected errors if the image changes. A best practice is to use an immutable tag that maps to a specific version of an application pod.

Drop All Capabilities

Capabilities permit privileged actions without giving full root access. All capabilities should be dropped from a pod, with only those required added back.

Require Labels

The ‘:latest’ tag is mutable and can lead to unexpected errors if the image changes. A best practice is to use an immutable tag that maps to a specific version of an application pod.

Require Limits and Requests

As application workloads share cluster resources, it is important to limit resources requested and consumed by each pod. It is recommended to require ‘resources.requests’ and ‘resources.limits.memory’ per pod. If a namespace level request or limit is specified, defaults will automatically be applied to each pod based on the ‘LimitRange’ configuration.

Require Pod Probes

Liveness and readiness probes need to be configured to correctly manage a pods lifecycle during deployments, restarts, and upgrades. For each pod, a periodic livenessProbe is performed by the kubelet to determine if the pod’s containers are running or need to be restarted. A readinessProbe is used by services and deployments to determine if the pod is ready to receive network traffic.

Require Read-Only Root FS

A read-only root file system helps to enforce an immutable infrastructure strategy; the container only needs to write on the mounted volume that persists the state. An immutable root filesystem can also prevent malicious binaries from writing to the host system.

Restrict External IPs

Service externalIPs can be used for a MITM attack (CVE-2020-8554). Restrict externalIPs or limit to a known set of addresses. See: https://github.com/kyverno/kyverno/issues/1367.

Restrict Image Registries

Images from unknown registries may not be scanned and secured. Requiring use of known registries helps reduce threat exposure.

Disallow Node Ports

A Kubernetes service of type NodePort uses a host port to receive traffic from any source. A ‘NetworkPolicy’ resource cannot be used to control traffic to host ports. Although ‘NodePort’ services can be useful, their use must be limited to services with additional upstream security checks.

Last modified December 30, 2020: update sample policies (647553b)