Installation

Install Kyverno using Helm

Add the nirmata Helm repository

helm repo add kyverno https://kyverno.github.io/kyverno/

Create a namespace and then install the kyverno helm chart.

# Create a namespace
kubectl create ns <namespace>

# Install the kyverno helm chart
helm install kyverno --namespace <namespace> kyverno/kyverno

For installing in kyverno namespace:

kubectl create ns kyverno

helm install kyverno --namespace kyverno kyverno/kyverno

Install Kyverno using YAMLs

The Kyverno policy engine runs as an admission webhook and requires a CA-signed certificate and key to setup secure TLS communication with the kube-apiserver (the CA can be self-signed). There are 2 ways to configure the secure communications link between Kyverno and the kube-apiserver.

Option 1: Use kube-controller-manager to generate a CA-signed certificate

Kyverno can request a CA signed certificate-key pair from kube-controller-manager. To install Kyverno in a cluster that supports certificate signing, run the following command on a host with kubectl cluster-admin access:

## Install Kyverno
kubectl create -f https://github.com/kyverno/kyverno/raw/master/definitions/install.yaml

This method requires that the kube-controller-manager is configured to act as a certificate signer. To verify that this option is enabled for your cluster, check the command-line args for the kube-controller-manager. If --cluster-signing-cert-file and --cluster-signing-key-file are passed to the controller manager with paths to your CA’s key-pair, then you can proceed to install Kyverno using this method.

Deploying on EKS requires enabling a command-line argument --fqdn-as-cn in the ‘kyverno’ container in the deployment, due to a current limitation with the certificates returned by EKS for CSR(bug: https://github.com/awslabs/amazon-eks-ami/issues/341)

Note that the above command will install the last released (stable) version of Kyverno. If you want to install the latest version, you can edit the [install.yaml] and update the image tag.

Also, by default kyverno is installed in “kyverno” namespace. To install in different namespace, you can edit the [install.yaml] and update the namespace.

To check the Kyverno controller status, run the command:

## Check pod status
kubectl get pods -n <namespace>

If the Kyverno controller is not running, you can check its status and logs for errors:

kubectl describe pod <kyverno-pod-name> -n <namespace>

kubectl logs <kyverno-pod-name> -n <namespace>

Option 2: Use your own CA-signed certificate

You can install your own CA-signed certificate, or generate a self-signed CA and use it to sign a certifcate. Once you have a CA and X.509 certificate-key pair, you can install these as Kubernetes secrets in your cluster. If Kyverno finds these secrets, it uses them. Otherwise it will request the kube-controller-manager to generate a certificate (see Option 1 above).

2.1. Generate a self-signed CA and signed certificate-key pair

Note: using a separate self-signed root CA is difficult to manage and not recommeded for production use.

If you already have a CA and a signed certificate, you can directly proceed to Step 2.

Here are the commands to create a self-signed root CA, and generate a signed certificate and key using openssl (you can customize the certificate attributes for your deployment):

1. Create a self-signed CA

openssl genrsa -out rootCA.key 4096
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt  -subj "/C=US/ST=test/L=test /O=test /OU=PIB/CN=*.kyverno.svc/emailAddress=test@test.com"

2. Create a keypair

openssl genrsa -out webhook.key 4096
openssl req -new -key webhook.key -out webhook.csr  -subj "/C=US/ST=test /L=test /O=test /OU=PIB/CN=kyverno-svc.kyverno.svc/emailAddress=test@test.com"

3. Create a webhook.ext file with the Subject Alternate Names (SAN) to use. This is required with Kubernetes 1.19+ and Go 1.15+.

subjectAltName = DNS:kyverno-svc,DNS:kyverno-svc.kyverno,DNS:kyverno-svc.kyverno.svc

4. Sign the keypair with the CA passing in the extension

openssl x509 -req -in webhook.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out webhook.crt -days 1024 -sha256 -extfile webhook.ext

5. Verify the contents of the certificate

 openssl x509 -in webhook.crt -text -noout

The certificate must contain the SAN information in the X509v3 extensions section:

X509v3 extensions:                                                                                                                              
    X509v3 Subject Alternative Name:                                                                                                            
        DNS:kyverno-svc, DNS:kyverno-svc.kyverno, DNS:kyverno-svc.kyverno.svc          

2.2. Configure secrets for the CA and TLS certificate-key pair

You can now use the following files to create secrets:

• rootCA.crt
• webhooks.crt
• webhooks.key

To create the required secrets, use the following commands (do not change the secret names):

kubectl create ns <namespace>
kubectl create secret tls kyverno-svc.kyverno.svc.kyverno-tls-pair --cert=webhook.crt --key=webhook.key -n <namespace> 
kubectl annotate secret kyverno-svc.kyverno.svc.kyverno-tls-pair self-signed-cert=true -n <namespace> 
kubectl create secret generic kyverno-svc.kyverno.svc.kyverno-tls-ca --from-file=rootCA.crt -n <namespace> 

NOTE: The annotation on the TLS pair secret is used by Kyverno to identify the use of self-signed certificates and checks for the required root CA secret

Kyverno uses secrets created above to setup TLS communication with the kube-apiserver and specify the CA bundle to be used to validate the webhook server’s certificate in the admission webhook configurations.

2.3. Install Kyverno

You can now install kyverno by downloading and updating the [install.yaml], or using the command below (assumes that the namespace is kyverno):

kubectl create -f https://github.com/kyverno/kyverno/raw/master/definitions/install.yaml

Configure Kyverno permissions

Kyverno, in foreground mode, leverages admission webhooks to manage incoming api-requests, and background mode applies the policies on existing resources. It uses ServiceAccount kyverno-service-account, which is bound to multiple ClusterRole, which defines the default resources and operations that are permitted.

ClusterRoles used by kyverno:

• kyverno:webhook
• kyverno:userinfo
• kyverno:customresources
• kyverno:policycontroller
• kyverno:generatecontroller

The generate rule creates a new resource, and to allow kyverno to create resource kyverno ClusterRole needs permissions to create/update/delete. This can be done by adding the resource to the ClusterRole kyverno:generatecontroller used by kyverno or by creating a new ClusterRole and a ClusterRoleBinding to kyverno’s default ServiceAccount.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: kyverno:generatecontroller
rules:
- apiGroups:
  - "*"
  resources:
  - namespaces
  - networkpolicies
  - secrets
  - configmaps
  - resourcequotas
  - limitranges
  - ResourceA # new Resource to be generated
  - ResourceB
  verbs:
  - create # generate new resources
  - get # check the contents of exiting resources
  - update # update existing resource, if required configuration defined in policy is not present
  - delete # clean-up, if the generate trigger resource is deleted

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: kyverno-admin-generate
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kyverno:generatecontroller # clusterRole defined above, to manage generated resources
subjects:
- kind: ServiceAccount
  name: kyverno-service-account # default kyverno serviceAccount
  namespace: kyverno

Custom installations

To install a specific version, download [install.yaml] and then change the image tag.

e.g., change image tag from latest to the specific tag v1.0.0.

spec:
  containers:
    - name: kyverno
      # image: nirmata/kyverno:latest
      image: nirmata/kyverno:v1.0.0

To install in a specific namespace replace the namespace “kyverno” with your namespace.

Example:

apiVersion: v1
kind: Namespace
metadata:
  name: <namespace>

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kyverno
  name: kyverno-svc
  namespace: <namespace>

and in other places (ServiceAccount, ClusterRoles, ClusterRoleBindings, ConfigMaps, Service, Deployment) where namespace is mentioned.

To run kyverno:

kubectl create -f ./install.yaml

To check the Kyverno controller status, run the command:

kubectl get pods -n <namespace>

If the Kyverno controller is not running, you can check its status and logs for errors:

kubectl describe pod <kyverno-pod-name> -n <namespace>

kubectl logs <kyverno-pod-name> -n <namespace>

Here is a script that generates a self-signed CA, a TLS certificate-key pair, and the corresponding kubernetes secrets: helper script

Configure Kyverno flags

1. excludeGroupRole : excludeGroupRole role expected string with Comma seperated group role. It will exclude all the group role from the user request. Default we are using system:serviceaccounts:kube-system,system:nodes,system:kube-scheduler.
2. excludeUsername : excludeUsername expected string with Comma seperated kubernetes username. In generate request if user enable Synchronize in generate policy then only kyverno can update/delete generated resource but admin can exclude specific username who have access of delete/update generated resource.
3. filterK8Resources: k8s resource in format [kind,namespace,name] where policy is not evaluated by the admission webhook. For example –filterKind “[Deployment, kyverno, kyverno]” –filterKind “[Deployment, kyverno, kyverno],[Events, *, *].

Configure access to policy violations

During Kyverno installation, it creates a ClusterRole kyverno:policyviolations which has the list,get,watch operations on resource policyviolations. To grant access to a namespace admin, configure the following YAML file then apply to the cluster.

• Replace metadata.namespace with namespace of the admin
• Configure subjects field to bind admin’s role to the ClusterRole policyviolation

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: policyviolation
  # change namespace below to create rolebinding for the namespace admin
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: policyviolation
subjects:
# configure below to access policy violation for the namespace admin
- kind: ServiceAccount
  name: default
  namespace: default
# - apiGroup: rbac.authorization.k8s.io
#   kind: User
#   name: 
# - apiGroup: rbac.authorization.k8s.io
#   kind: Group
#   name: 

Filter resources that Kyverno should not process

The admission webhook checks if a policy is applicable on all admission requests. The Kubernetes kinds that are not be processed can be filtered by adding a ConfigMap in namespace kyverno and specifying the resources to be filtered under data.resourceFilters. The default name of this ConfigMap is init-config but can be changed by modifying the value of the environment variable INIT_CONFIG in the kyverno deployment dpec. data.resourceFilters must be a sequence of one or more [<Kind>,<Namespace>,<Name>] entries with * as wildcard. Thus, an item [Node,*,*] means that admissions of Node in any namespace and with any name will be ignored.

By default we have specified Nodes, Events, APIService & SubjectAccessReview as the kinds to be skipped in the default configuration [install.yaml].

apiVersion: v1
kind: ConfigMap
metadata:
  name: init-config
  namespace: kyverno
data:
  # resource types to be skipped by kyverno policy engine
  resourceFilters: "[Event,*,*][*,kube-system,*][*,kube-public,*][*,kube-node-lease,*][Node,*,*][APIService,*,*][TokenReview,*,*][SubjectAccessReview,*,*][*,kyverno,*]"

To modify the ConfigMap, either directly edit the ConfigMap init-config in the default configuration [install.yaml] and redeploy it or modify the ConfigMap use kubectl. Changes to the ConfigMap through kubectl will automatically be picked up at runtime.