Configure Kyverno in HA mode
To install Kyverno in HA using the Helm Chart.
NOTE: For high availability, the only supported replica count is 3.
1helm install kyverno kyverno/kyverno -n kyverno --create-namespace --set=replicaCount=3
How HA works in Kyverno
This section provides details on how Kyverno handles HA scenarios.
The webhook server is where Kyverno receives and processes admission requests. This controller does not require leader election. When Kyverno runs multiple instances, the Kubernetes Service will distribute the admission requests across different Kyverno instances.
validate rules in
Enforce mode, Kyverno returns the decision along with the admission response; it is a synchronous request-response process. However, for
validate rules in
Audit mode, Kyverno pushes these requests to a queue and returns the response immediately, then starts processing the data asynchronously. The queue in the
validate audit handler is used to generate policy reports.
Since the report will be reconciled when Kyverno restarts, there’s no need to drain this queue on shutdown. If the process is terminated, we need to complete pending requests / drain the queue and then shutdown Kyverno gracefully.
The library is used by kube-controller-manager, kube-scheduler, etc. Notice that this library does not guarantee that only one client is acting as a leader (a.k.a. fencing), so we have to design in a way that even if the same process gets executed twice, the results are consistent.
The certificate renewer controller is responsible for monitoring the Secrets Kyverno uses for its webhooks. This controller also uses leader election.
The webhook controller is responsible for configuring the various webhooks. These webhooks, by default managed dynamically, instruct the Kubernetes API server which resources to send to Kyverno. The webhook controller uses leader election as it maintains an internal webhook timestamp to monitor the webhook status. The controller also recreates the webhook configurations if any are missing.
The background controller is responsible for handling UpdateRequest resources (an intermediary resource used by generate and mutate-existing rules) and processing rules in background scanning mode. The background controller does not require leader election.
The report controller is responsible for creation of policy reports from both admission requests and background scans and requires leader election. It track resources that need to be processed in the background and generates background scan reports (when policy/resource change). It also aggregates these and the intermediary admission reports into the final policy report resources PolicyReport and ClusterPolicyReport.
Cleanup - Webhook Server
In the cleanup process, the webhook server reconciles cleanup policies and existing CronJobs. It does not require leader election.
Cleanup - Cleanup controller
The cleanup controller applies cleanup policies to generate CronJobs and reconciles existing CronJobs when they change. This component does require leader election.
Cleanup - Cert Renewer
The cert manager manages the certificates stored as Secrets and does require leader election.
Cleanup - Webhook Controller
The webhook controller updates the webhook used by the cleanup controller when the Secret changes. This component also uses leader election.