External Data Sources
The Variables section discusses how variables can help create smarter and reusable policy definitions and introduced the concept of a rule context that stores all variables.
This section provides details on using ConfigMaps, API calls, service calls, and image registries to reference external data as variables in policies.
Variables from ConfigMaps
A ConfigMap resource in Kubernetes is commonly used as a source of configuration details which can be consumed by applications. This data can be written in multiple formats, stored in a Namespace, and accessed easily. Kyverno supports using a ConfigMap as a data source for variables. When a policy referencing a ConfigMap resource is evaluated, the ConfigMap data is checked at that time ensuring that references to the ConfigMap are always dynamic. Should the ConfigMap be updated, subsequent policy lookups will pick up the latest data at that point.
In order to consume data from a ConfigMap in a rule, a context is required. For each rule you wish to consume data from a ConfigMap, you must define a context. The context data can then be referenced in the policy rule using JMESPath notation.
Looking up ConfigMap values
A ConfigMap that is defined in a rule’s context can be referenced using its unique name within the context. ConfigMap values can be referenced using a JMESPath style expression.
Consider a simple ConfigMap definition like so.
1apiVersion: v1
2kind: ConfigMap
3metadata:
4 name: some-config-map
5 namespace: some-namespace
6data:
7 env: production
To refer to values from a ConfigMap inside a rule, define a context inside the rule with one or more ConfigMap declarations. Using the sample ConfigMap snippet referenced above, the below rule defines a context which references this specific ConfigMap by name.
1rules:
2 - name: example-lookup
3 # Define a context for the rule
4 context:
5 # A unique name for the context variable under which the below contents will later be accessible
6 - name: dictionary
7 configMap:
8 # Name of the ConfigMap which will be looked up
9 name: some-config-map
10 # Namespace in which this ConfigMap is stored
11 namespace: some-namespace
Based on the example above, we can now refer to a ConfigMap value using {{dictionary.data.env}}. The variable will be substituted with the value production during policy execution.
Put into context of a full policy, referencing a ConfigMap as a variable looks like the following.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: cm-variable-example
5 annotations:
6 pod-policies.kyverno.io/autogen-controllers: DaemonSet,Deployment,StatefulSet
7spec:
8 rules:
9 - name: example-configmap-lookup
10 match:
11 any:
12 - resources:
13 kinds:
14 - Pod
15 context:
16 - name: dictionary
17 configMap:
18 name: some-config-map
19 namespace: some-namespace
20 mutate:
21 patchStrategicMerge:
22 metadata:
23 labels:
24 my-environment-name: "{{dictionary.data.env}}"
In the above ClusterPolicy, a mutate rule matches all incoming Pod resources and adds a label to them with the name of my-environment-name. Because we have defined a context which points to our earlier ConfigMap named mycmap, we can reference the value with the expression {{dictionary.data.env}}. A new Pod will then receive the label my-environment-name=production.
Note
ConfigMap names and keys can contain characters that are not supported by JMESPath, such as “-” (minus or dash) or “/” (slash). To evaluate these characters as literals, add double quotes to that part of the JMESPath expression as follows:
See the JMESPath page for more information on formatting.
Kyverno also has the ability to cache ConfigMaps commonly used by policies to reduce the number of API calls made. This both decreases the load on the API server and increases the speed of policy evaluation. Assign the label cache.kyverno.io/enabled: "true" to any ConfigMap and Kyverno will automatically cache it. Policy decisions will fetch the data from cache rather than querying the API server. This feature may be disabled through an optional container flag if desired.
Handling ConfigMap Array Values
In addition to simple string values, Kyverno has the ability to consume array values from a ConfigMap stored as either JSON- or YAML-formatted values. Depending on how you choose to store an array, the policy which consumes the values in a variable context will need to be written accordingly.
For example, let’s say you wanted to define a list of allowed roles in a ConfigMap. A Kyverno policy can refer to this list to deny a request where the role, defined as an annotation, does not match one of the values in the list.
Consider a ConfigMap with the following content written as a JSON array. You may also store array values in a YAML block scalar (in which case the parse_yaml() filter will be necessary in a policy definition).
1apiVersion: v1
2kind: ConfigMap
3metadata:
4 name: roles-dictionary
5 namespace: default
6data:
7 allowed-roles: '["cluster-admin", "cluster-operator", "tenant-admin"]'
Now that the array data is saved in the allowed-roles key, here is a sample policy which blocks a Deployment if the value of the annotation named role is not in the allowed list. Notice how the parse_json() JMESPath filter is used to interpret the value of the ConfigMap’s allowed-roles key into an array of strings.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: cm-array-example
5spec:
6 background: false
7 rules:
8 - name: validate-role-annotation
9 context:
10 - name: roles-dictionary
11 configMap:
12 name: roles-dictionary
13 namespace: default
14 match:
15 any:
16 - resources:
17 kinds:
18 - Deployment
19 validate:
20 failureAction: Enforce
21 message: "The role {{ request.object.metadata.annotations.role }} is not in the allowed list of roles: {{ \"roles-dictionary\".data.\"allowed-roles\" }}."
22 deny:
23 conditions:
24 any:
25 - key: "{{ request.object.metadata.annotations.role }}"
26 operator: AnyNotIn
27 value: "{{ \"roles-dictionary\".data.\"allowed-roles\" | parse_json(@) }}"
This rule denies the request for a new Deployment if the annotation role is not found in the array we defined in the earlier ConfigMap named roles-dictionary.
Note
You may also notice that this sample uses variables from both AdmissionReview and ConfigMap sources in a single rule. This combination can prove to be very powerful and flexible in crafting useful policies.Once creating this sample policy, attempt to create a new Deployment where the annotation role=super-user and test the result.
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: busybox
5 annotations:
6 role: super-user
7 labels:
8 app: busybox
9spec:
10 replicas: 1
11 selector:
12 matchLabels:
13 app: busybox
14 template:
15 metadata:
16 labels:
17 app: busybox
18 spec:
19 containers:
20 - image: busybox:1.28
21 name: busybox
22 command: ["sleep", "9999"]
Submit the manifest and see how Kyverno reacts.
1Error from server: error when creating "deploy.yaml": admission webhook "validate.kyverno.svc" denied the request:
2
3resource Deployment/default/busybox was blocked due to the following policies
4
5cm-array-example:
6 validate-role-annotation: 'The role super-user is not in the allowed list of roles: ["cluster-admin", "cluster-operator", "tenant-admin"].'
Changing the role annotation to one of the values present in the ConfigMap, for example tenant-admin, allows the Deployment resource to be created.
Variables from Kubernetes API Server Calls
Kubernetes is powered by a declarative API that allows querying and manipulating resources. Kyverno policies can use the Kubernetes API to fetch a resource, or even collections of resource types, for use in a policy. Additionally, Kyverno allows applying JMESPath (JSON Match Expression) to the resource data to extract and transform values into a format that is easy to use within a policy.
A Kyverno Kubernetes API call works just as with kubectl and other API clients, and can be tested using existing tools.
For example, this command uses kubectl to fetch the list of Pods in a Namespace and then pipes the output to kyverno jp which counts the number of Pods:
The corresponding API call in Kyverno is defined as below. It uses a variable {{request.namespace}} to use the Namespace of the object being operated on, and then applies the same JMESPath to store the count of Pods in the Namespace in the context as the variable podCount. Variables may be used in both fields. This new resulting variable podCount can then be used in the policy rule.
1rules:
2- name: example-api-call
3 context:
4 - name: podCount
5 apiCall:
6 urlPath: "/api/v1/namespaces/{{request.namespace}}/pods"
7 jmesPath: "items | length(@)"
Calls to the Kubernetes API server may also perform POST operations in addition to GET which is the default method. The returned data from the API server can then be used for further policy decisions.
For example, this snippet below shows making a call to the SubjectAccessReview API to determine if an actor is authorized. Performing a POST operation requires specifying the method field and the data object with the contents of the request given as a list of key/value pairs.
1context:
2 - name: subjectaccessreview
3 apiCall:
4 urlPath: /apis/authorization.k8s.io/v1/subjectaccessreviews
5 method: POST
6 data:
7 - key: kind
8 value: SubjectAccessReview
9 - key: apiVersion
10 value: authorization.k8s.io/v1
11 - key: spec
12 value:
13 resource: "namespace"
14 resourceAttributes:
15 namespace: "{{ request.namespace }}"
16 verb: "delete"
17 group: ""
18 user: "{{ request.userInfo.username }}"
The response from such a request will be the full JSON return and accessible under the variable subjectaccessreview.
URL Paths
The Kubernetes API organizes resources under groups and versions. For example, the resource type Deployment is available in the API Group apps with a version v1.
The HTTP URL paths of the API calls are based on the group, version, and resource type as follows:
/apis/{GROUP}/{VERSION}/{RESOURCETYPE}: get a collection of resources/apis/{GROUP}/{VERSION}/{RESOURCETYPE}/{NAME}: get a resource
For Namespaced resources, to get a specific resource by name or to get all resources in a Namespace, the Namespace name must also be provided as follows:
/apis/{GROUP}/{VERSION}/namespaces/{NAMESPACE}/{RESOURCETYPE}: get a collection of resources in the namespace/apis/{GROUP}/{VERSION}/namespaces/{NAMESPACE}/{RESOURCETYPE}/{NAME}: get a resource in a namespace
For historic resources, the Kubernetes Core API is available under /api/v1. For example, to query all Namespace resources the path /api/v1/namespaces is used.
The Kubernetes API groups are defined in the API reference documentation for v1.25 and can also be retrieved via the kubectl api-resources command shown below:
1$ kubectl api-resources
2NAME SHORTNAMES APIVERSION NAMESPACED KIND
3bindings v1 true Binding
4componentstatuses cs v1 false ComponentStatus
5configmaps cm v1 true ConfigMap
6endpoints ep v1 true Endpoints
7events ev v1 true Event
8limitranges limits v1 true LimitRange
9namespaces ns v1 false Namespace
10nodes no v1 false Node
11persistentvolumeclaims pvc v1 true PersistentVolumeClaim
12persistentvolumes pv v1 false PersistentVolume
13pods po v1 true Pod
14podtemplates v1 true PodTemplate
15replicationcontrollers rc v1 true ReplicationController
16resourcequotas quota v1 true ResourceQuota
17...
The kubectl api-versions command prints out the available versions for each API group. Here is a sample:
1$ kubectl api-versions
2admissionregistration.k8s.io/v1
3admissionregistration.k8s.io/v1alpha1
4apiextensions.k8s.io/v1
5apiregistration.k8s.io/v1
6apps/v1
7authentication.k8s.io/v1
8authorization.k8s.io/v1
9autoscaling/v1
10autoscaling/v2
11batch/v1
12certificates.k8s.io/v1
13coordination.k8s.io/v1
14...
You can use these commands together to find the URL path for resources, as shown below:
Tip
To find the API group and version for a resource use kubectl api-resources to find the group and then kubectl api-versions to find the available versions.
This example finds the group of Deployment resources and then queries the version:
The API group is shown in the third column of the output. You can then use the group name to find the version:
The output of this will be apps/v1.
Kyverno can also fetch data from other API locations such as /version and aggregated APIs.
For example, fetching from /version might return something similar to what is shown below.
1$ kubectl get --raw /version
2{
3 "major": "1",
4 "minor": "23",
5 "gitVersion": "v1.23.8+k3s1",
6 "gitCommit": "53f2d4e7d80c09a7db1858e3f4e7ddfa13256c45",
7 "gitTreeState": "clean",
8 "buildDate": "2022-06-27T21:48:01Z",
9 "goVersion": "go1.17.5",
10 "compiler": "gc",
11 "platform": "linux/amd64"
12}
Fetching from an aggregated API, for example the metrics.k8s.io group, can be done with /apis/metrics.k8s.io/<api_version>/<resource_type> as shown below.
1$ kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq
2{
3 "kind": "NodeMetricsList",
4 "apiVersion": "metrics.k8s.io/v1beta1",
5 "metadata": {},
6 "items": [
7 {
8 "metadata": {
9 "name": "k3d-kyv180rc1-server-0",
10 "creationTimestamp": "2022-09-11T13:37:39Z",
11 "labels": {
12 "beta.kubernetes.io/arch": "amd64",
13 "beta.kubernetes.io/instance-type": "k3s",
14 "beta.kubernetes.io/os": "linux",
15 "egress.k3s.io/cluster": "true",
16 "kubernetes.io/arch": "amd64",
17 "kubernetes.io/hostname": "k3d-kyv180rc1-server-0",
18 "kubernetes.io/os": "linux",
19 "node-role.kubernetes.io/control-plane": "true",
20 "node-role.kubernetes.io/master": "true",
21 "node.kubernetes.io/instance-type": "k3s"
22 }
23 },
24 "timestamp": "2022-09-11T13:37:24Z",
25 "window": "10.059s",
26 "usage": {
27 "cpu": "298952967n",
28 "memory": "1311340Ki"
29 }
30 }
31 ]
32}
Query parameters are also accepted in the urlPath field. This allows, for example, making API calls with a label selector or a return limit which is beneficial in that some of the processing of these API calls may be offloaded to the Kubernetes API server rather than Kyverno having to process them in JMESPath statements. The following shows a context variable being set which uses an API call with label selector and limit queries.
1context:
2- name: serviceCount
3 apiCall:
4 urlPath: "/api/v1/namespaces/{{ request.namespace }}/services?labelSelector=foo=bar?limit=5"
5 jmesPath: "items[?spec.type == 'LoadBalancer'] | length(@)"
Using query parameters has the added benefit that there will always be a response even if the query returns no results. This can be beneficial (and even necessary) in some cases where an API call to fetch an exact resource by name may fail because the resource does not exist.
For example, if the foo Service does not exist, an API call to return that specific resource will fail.
1$ kubectl get --raw /api/v1/namespaces/default/services/foo
2Error from server (NotFound): services "foo" not found
However, an API call with a query parameter against all Services will return successful but with an empty collection.
1$ kubectl get --raw /api/v1/namespaces/default/services?fieldSelector=metadata.name=foo | jq
2{
3 "kind": "ServiceList",
4 "apiVersion": "v1",
5 "metadata": {
6 "resourceVersion": "167567"
7 },
8 "items": []
9}
Further information on handling collections is covered below.
Handling collections
The API server response for a HTTP GET on a URL path that requests collections of resources will be an object with a list of items (resources).
Here is an example that fetches all Namespace resources:
Tip
Usejq to format output for readability.This will return a NamespaceList object with a property items that contains the list of Namespaces:
1{
2 "kind": "NamespaceList",
3 "apiVersion": "v1",
4 "metadata": {
5 "selfLink": "/api/v1/namespaces",
6 "resourceVersion": "2009258"
7 },
8 "items": [
9 {
10 "metadata": {
11 "name": "default",
12 "selfLink": "/api/v1/namespaces/default",
13 "uid": "5011b5d5-abb7-4fef-93f9-8b5fa4b2eba9",
14 "resourceVersion": "155",
15 "creationTimestamp": "2021-01-19T20:20:37Z",
16 "managedFields": [
17 {
18 "manager": "kube-apiserver",
19 "operation": "Update",
20 "apiVersion": "v1",
21 "time": "2021-01-19T20:20:37Z",
22 "fieldsType": "FieldsV1",
23 "fieldsV1": {
24 "f:status": {
25 "f:phase": {}
26 }
27 }
28 }
29 ]
30 },
31 "spec": {
32 "finalizers": [
33 "kubernetes"
34 ]
35 },
36 "status": {
37 "phase": "Active"
38 }
39 },
40 ...
To process this data in JMESPath, reference the items. Here is an example which extracts a few metadata fields across all Namespace resources:
1kubectl get --raw /api/v1/namespaces | kyverno jp query "items[*].{name: metadata.name, creationTime: metadata.creationTimestamp}"
This produces a new JSON list of objects with properties name and creationTime.
1[
2 {
3 "creationTimestamp": "2021-01-19T20:20:37Z",
4 "name": "default"
5 },
6 {
7 "creationTimestamp": "2021-01-19T20:20:36Z",
8 "name": "kube-node-lease"
9 },
10 ...
To find an item in the list you can use JMESPath filters. For example, this command will match a Namespace by its name:
1kubectl get --raw /api/v1/namespaces | kyverno jp query "items[?metadata.name == 'default'].{uid: metadata.uid, creationTimestamp: metadata.creationTimestamp}"
In addition to wildcards and filters, JMESPath has many additional powerful features including several useful functions. Be sure to go through the JMESPath tutorial and try the interactive examples in addition to the Kyverno JMESPath page.
Examples
Here is a complete sample policy that limits each Namespace to a single Service of type LoadBalancer.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: limits
5spec:
6 rules:
7 - name: limit-lb-svc
8 match:
9 any:
10 - resources:
11 kinds:
12 - Service
13 operations:
14 - CREATE
15 context:
16 - name: serviceCount
17 apiCall:
18 urlPath: "/api/v1/namespaces/{{ request.namespace }}/services"
19 jmesPath: "items[?spec.type == 'LoadBalancer'] | length(@)"
20 validate:
21 failureAction: Enforce
22 message: "Only one LoadBalancer service is allowed per namespace"
23 deny:
24 conditions:
25 any:
26 - key: "{{ serviceCount }}"
27 operator: GreaterThan
28 value: 1
This sample policy retrieves the list of Services in the Namespace and stores the count of type LoadBalancer in a variable called serviceCount. A deny rule is used to ensure that the count cannot exceed one.
Variables from Service Calls
Similar to how Kyverno is able to call the Kubernetes API server to both GET and POST data for use in a context variable, Kyverno is also able to call any other service in the cluster. This feature is nested under the apiCall context type and builds upon it. See the section above on Kubernetes API calls for more information.
By using the apiCall.service object, a call may be made to another URL to retrieve and store data. The fields caBundle and url are used to specify the CA bundle and URL, respectively, for the call. The fields apiCall.urlPath and apiCall.service.url are mutually exclusive; a call can only be to either the Kubernetes API or some other service. At this time, authentication as part of these service calls is not supported. The response from a Service call must only be JSON.
For example, the following policy makes a POST request to another Kubernetes Service accessible at http://sample.kyverno-extension/check-namespace and sends it the data {"name":"foonamespace"} when a ConfigMap is created in the foonamespace Namespace. The JSON result Kyverno receives is stored in the context called result. The value of result is JSON where the key allowed is either true or false. The request is blocked if the value is false.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: check-namespaces
5spec:
6 rules:
7 - name: call-extension
8 match:
9 any:
10 - resources:
11 kinds:
12 - ConfigMap
13 context:
14 - name: result
15 apiCall:
16 method: POST
17 data:
18 - key: namespace
19 value: "{{request.namespace}}"
20 service:
21 url: http://sample.kyverno-extension/check-namespace
22 caBundle: |-
23 -----BEGIN CERTIFICATE-----
24 <snip>
25 -----END CERTIFICATE-----
26 validate:
27 failureAction: Enforce
28 message: "namespace {{request.namespace}} is not allowed"
29 deny:
30 conditions:
31 all:
32 - key: "{{ result.allowed }}"
33 operator: Equals
34 value: false
Global Context
Global Context allows users to cache Kubernetes resources or the results of external API calls for later reference within policies. It provides a mechanism to efficiently retrieve and utilize data across policies, enhancing flexibility and performance in policy enforcement. This new Global Context ability joins the existing ConfigMap caching ability.
Global Context Entries are declared globally using a new GlobalContextEntry Custom Resource and referenced as part of a policy context. There are two types of Global Context Entries, a Kubernetes resource entry and an API call entry.
Kubernetes Resource
A Kubernetes resource Global Context allows you to easily reference a specific kind of Kubernetes resource. Only a single kind may be referenced with the option of specifying both Namespaced and global resources.
This GlobalContextEntry caches all Deployment resources in the fitness Namespace.
1apiVersion: kyverno.io/v2alpha1
2kind: GlobalContextEntry
3metadata:
4 name: deployments
5spec:
6 kubernetesResource:
7 group: apps
8 version: v1
9 resource: deployments
10 namespace: fitness
The resource value must be the pluralized, lower-case form (“deployments” and not “Deployment”). Omitting the namespace field for Namespaced resources will result in a cache entry being built for all such resources in the cluster. For cluster-scoped resources, omit the namespace field.
Only a single Kubernetes resource kind may be specified per GlobalContextEntry. Internally, Kyverno uses informers to automatically watch and build an updated cache whenever the target resource kind changes.
Once the deployments GlobalContextEntry has been created, it may be referenced in a Kyverno policy using a context of the type globalReference where the name is the same name as the GlobalContextEntry Custom Resource.
In the case of a Kubernetes resource type of GlobalContextEntry, the value will be an array of objects as shown below.
1[
2 {
3 "apiVersion": "apps/v1",
4 "kind": "Deployment",
5 "metadata": {
6 "annotations": {
7 "deployment.kubernetes.io/revision": "1"
8 },
9 }
10 ...
11 },
12 {
13 "apiVersion": "apps/v1",
14 "kind": "Deployment",
15 "metadata": {
16 "annotations": {
17 "deployment.kubernetes.io/revision": "1"
18 },
19 }
20 ...
21 }
22]
API Call
An API call GlobalContextEntry defines either an API call to an external service or a raw call to the Kubernetes API. The latter is useful over a Kubernetes resource GlobalContextEntry when you wish to reduce the amount of data populated into the cache by using query parameters or when needing to perform a POST rather than GET.
Below shows a similar example from the Kubernetes resource type previously but using a labelSelector to limit the number of resources returned into the cache. The contents will be refreshed according to the value of the refreshInterval field.
1apiVersion: kyverno.io/v2alpha1
2kind: GlobalContextEntry
3metadata:
4 name: deployments
5spec:
6 apiCall:
7 urlPath: "/apis/apps/v1/namespaces/fitness/deployments?labelSelector=app=blue"
8 refreshInterval: 10s
The data returned from an apiCall GlobalContextEntry to the Kubernetes API is the same data structure returned from a context entry of type apiCall referenced above. Note that specifically this means the contents will be wrapped in items[] and not [] as is the case with a Kubernetes resource type.
A cache entry may also be created for external services as well specifying an optional CA bundle to establish trust.
1apiVersion: kyverno.io/v2alpha1
2kind: GlobalContextEntry
3metadata:
4 name: redisdata
5spec:
6 apiCall:
7 method: GET
8 refreshInterval: 1m
9 service:
10 url: https://redis.myns.svc:6379
11 caBundle: |-
12 -----BEGIN CERTIFICATE-----
13 MIIBdjCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy
14 <snip>
15 W/LgVuvZmucCIBcETS4DIw2pWAfeKRDaEOi2YsJoDpWd7lFLQBUbe4G7
16 -----END CERTIFICATE-----
Note
apiCall GlobalContextEntries are implemented by periodically making a call to the specified endpoint every refreshInterval, thus beware of stale data.Reference the GlobalContextEntry in a policy using the context.globalReference type. Shown below is an example referencing the redisdata cache entry and applying a JMESPath filter to its contents. The resulting location variable will be the result of the address.city filter applied to the contents of redisdata.
The data returned by GlobalContextEntries may vary depending on whether it is a Kubernetes resource or an API call. Consequently, the JMESPath expression used to manipulate the data may differ as well. Ensure you use the appropriate JMESPath expression based on the type of data being accessed to ensure accurate processing within policies.
To use Global Contexts with the Kyverno CLI, you can use the Values file to inject these global context entries into your policy evaluation. This allows you to simulate different scenarios and test your policies with various global context values without modifying the actual GlobalContextEntry resources in your cluster. Refer to it here: kyverno apply.
Warning
GlobalContextEntries must be in a healthy state (i.e., there is a response received from the remote endpoint) in order for the policies which reference them to be considered healthy. A GlobalContextEntry which is in anot ready state will cause any/all referenced policies to also be in a similar state and therefore will not be processed. Creation of a policy referencing a GlobalContextEntry which either does not exist or is not ready will print a warning notifying users.Default values for API calls
In the case where the api server returns an error, default can be used to provide a fallback value for the api call context entry. The following example shows how to add default value to context entries:
1...
2 context:
3 - name: currentnamespace
4 apiCall:
5 urlPath: "/api/v1/namespaces/{{ request.namespace }}"
6 jmesPath: metadata.name
7 default: default
8...
Variables from Image Registries
A context can also be used to store metadata on an OCI image by using the imageRegistry context type. By using this external data source, a Kyverno policy can make decisions based on details of the container image that occurs as part of an incoming resource.
For example, if you are using an imageRegistry like shown below:
the output imageData variable will have a structure which looks like the following:
1{
2 "image": "ghcr.io/kyverno/kyverno",
3 "resolvedImage": "ghcr.io/kyverno/kyverno@sha256:17bfcdf276ce2cec0236e069f0ad6b3536c653c73dbeba59405334c0d3b51ecb",
4 "registry": "ghcr.io",
5 "repository": "kyverno/kyverno",
6 "identifier": "latest",
7 "imageIndex": imageIndex,
8 "manifest": manifest,
9 "configData": config,
10}
Note
The imageData variable represents a “normalized” view of an image after any redirects by the registry are performed and internal modifications by Kyverno (Kyverno by default sets an empty registry to docker.io and an empty tag to latest). Most notably, this impacts official images hosted on Docker Hub. Official images on Docker Hub are differentiated from other images in that their repository is prefixed by library/ even if the image being pulled does not contain it. For example, pulling the python official image with python:slim results in the following fields of imageData being set:
The imageIndex, manifest and config keys contain the output from crane manifest <image> and crane config <image> respectively.
For example, one could inspect the labels, entrypoint, volumes, history, layers, etc of a given image. Using the crane tool, show the config of the ghcr.io/kyverno/kyverno:latest image:
1$ crane config ghcr.io/kyverno/kyverno:latest | jq
2{
3 "architecture": "amd64",
4 "author": "github.com/ko-build/ko",
5 "created": "2023-01-08T00:10:08Z",
6 "history": [
7 {
8 "author": "apko",
9 "created": "2023-01-08T00:10:08Z",
10 "created_by": "apko",
11 "comment": "This is an apko single-layer image"
12 },
13 {
14 "author": "ko",
15 "created": "0001-01-01T00:00:00Z",
16 "created_by": "ko build ko://github.com/kyverno/kyverno/cmd/kyverno",
17 "comment": "kodata contents, at $KO_DATA_PATH"
18 },
19 {
20 "author": "ko",
21 "created": "0001-01-01T00:00:00Z",
22 "created_by": "ko build ko://github.com/kyverno/kyverno/cmd/kyverno",
23 "comment": "go build output, at /ko-app/kyverno"
24 }
25 ],
26 "os": "linux",
27 "rootfs": {
28 "type": "layers",
29 "diff_ids": [
30 "sha256:c9770b71bc04d50fb006eaacea8180b5f7c0fc72d16618590ec5231f9cec2525",
31 "sha256:ffe56a1c5f3878e9b5f803842adb9e2ce81584b6bd027e8599582aefe14a975b",
32 "sha256:de3816af2ab66f6b306277c83a7cc9af74e5b0e235021a37f2fc916882751819"
33 ]
34 },
35 "config": {
36 "Entrypoint": [
37 "/ko-app/kyverno"
38 ],
39 "Env": [
40 "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ko-app",
41 "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt",
42 "KO_DATA_PATH=/var/run/ko"
43 ],
44 "User": "65532"
45 }
46}
In the output above, we can see under config.User that the USER Dockerfile statement to run this container is 65532. A Kyverno policy can be written to harness this information and perform, for example, a validation that the USER of an image is non-root.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: imageref-demo
5spec:
6 rules:
7 - name: no-root-images
8 match:
9 any:
10 - resources:
11 kinds:
12 - Pod
13 operations:
14 - CREATE
15 - UPDATE
16 validate:
17 failureAction: Enforce
18 message: "Images run as root are not allowed."
19 foreach:
20 - list: "request.object.spec.containers"
21 context:
22 - name: imageData
23 imageRegistry:
24 reference: "{{ element.image }}"
25 deny:
26 conditions:
27 any:
28 - key: "{{ imageData.configData.config.User || ''}}"
29 operator: Equals
30 value: ""
In the above sample policy, a new context has been written named imageData which uses the imageRegistry type. The reference key is used to instruct Kyverno where the image metadata is stored. In this case, the location is the same as the image itself hence element.image where element is each container image inside of a Pod. The value can then be referenced in an expression, for example in deny.conditions via the key {{ imageData.configData.config.User || ''}}.
Using a sample “bad” resource to test which violates this policy, such as below, the Pod is blocked.
1apiVersion: v1
2kind: Pod
3metadata:
4 name: badpod
5spec:
6 containers:
7 - name: ubuntu
8 image: ubuntu:latest
1$ kubectl apply -f bad.yaml
2Error from server: error when creating "bad.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:
3
4resource Pod/default/badpod was blocked due to the following policies
5
6imageref-demo:
7 no-root-images: 'validation failure: Images run as root are not allowed.'
By contrast, when using a “good” Pod, such as the Kyverno container image referenced above, the resource is allowed.
1apiVersion: v1
2kind: Pod
3metadata:
4 name: goodpod
5spec:
6 containers:
7 - name: kyverno
8 image: ghcr.io/kyverno/kyverno:latest
The imageRegistry context type also has an optional property called jmesPath which can be used to apply a JMESPath expression to contents returned by imageRegistry prior to storing as the context value. For example, the below snippet stores the total size of an image in a context named imageSize by summing up all the constituent layers of the image as reported by its manifest (visible with, for example, crane by using the crane manifest command). The value of the context variable can then be evaluated in a later expression.
1context:
2 - name: imageSize
3 imageRegistry:
4 reference: "{{ element.image }}"
5 # Note that we need to use `to_string` here to allow kyverno to treat it like a resource quantity of type memory
6 # the total size of an image as calculated by docker is the total sum of its layer sizes
7 jmesPath: "to_string(sum(manifest.layers[*].size))"
To access images stored on private registries, see using private registries
For more examples of using an imageRegistry context, see the samples page.
The policy-level setting failurePolicy when set to Ignore additionally means that failing calls to image registries will be ignored. This allows for Pods to not be blocked if the registry is offline, useful in situations where images already exist on the nodes.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.