Kyverno Policy As Code Using CDK8S

Kyverno Policy As Code Using CDK8S


6 min read


  • Kyverno Kyverno is a policy engine designed for Kubernetes, Kyverno policies can validate, mutate, and generate Kubernetes resources plus ensure OCI image supply chain security.

  • This blog, it provides the way to create Kyverno policy as code using CDK8S typescript.

  • By importing Kyverno CRDs and using CDK8S you can create Kyverno policy manifest using your familiar programming languages such as typescript as scale.

Table Of Contents

๐Ÿš€ Pre-requisite

  • Install typescript, node, and cdk8s as well as projen (optional) which is a tool for managing project configuration as code.

  • Getting started with cdk8s

  • EKS/kubernetes cluster to test

๐Ÿš€ Overview of Kyverno

  • The features are

    • Policies as Kubernetes resources in YAML

    • Validate, mutate, or generate any resource using Kustomize overlays

    • Match resources using label selectors and wildcards

    • Block non-conformant resources using admission controls, or report policy violations

    • Test policies and validate resources using the Kyverno CLI, in your CI/CD pipeline, before applying them to your cluster

  • How does it work?

๐Ÿš€ Import Kyverno CRDs

  • Import kyverno CRDs as cdk8s lib
โšก $ cdk8s import --output src/imports/
Importing resources, this may take a few moments...
  • Output of importing

        โšก $ tree src/imports/
        0 directories, 1 file

๐Ÿš€ Write code

  • It's much more convenient to use visual code writing Kyverno policies in typescript language. We can read the document and find all references of construct, objects and properties of Kyverno policies through code descriptions.

  • On top of all policies, there's a simple construct (feel free to implement more the construct) so that each policy just needs to input name, pattern, etc.

    • The interface of kyverno properties

        export interface KyvernoProps {
          name: string;
          message: string;
          namespace?: string;
          action?: ClusterPolicySpecValidationFailureAction;
          kinds?: Array<string>;
          resources?: {};
          exclude?: ClusterPolicySpecRulesExclude;
          deny?: ClusterPolicySpecRulesValidateDeny;
          pattern?: {};
          anyPatterns?: {};
    • The constructed class

        export class KyvernoClusterPolicy extends Chart {
          constructor(scope: Construct, name: string, kyvernoProps: KyvernoProps) {
            super(scope, name);
            new ClusterPolicy(this, `${}`, {
              metadata: {
                namespace: kyvernoProps.namespace || undefined,
                annotations: {
                  '': 'Pod Security Standards',
              spec: {
                validationFailureAction: kyvernoProps.action || ClusterPolicySpecValidationFailureAction.ENFORCE,
                rules: [{
                  match: {
                    any: [{
                      resources: kyvernoProps.resources || { kinds: ['Pod'] },
                  validate: {
                    deny: kyvernoProps.deny || undefined,
                    message: kyvernoProps.message,
                    pattern: kyvernoProps.pattern || undefined,
                    anyPattern: kyvernoProps.anyPatterns || undefined,
                  exclude: kyvernoProps.exclude || undefined,
  • This blog provides examples of 5 use cases

    1. Deny delete objects which have label protected: 'true'

    2. require-app-label

    3. require-request-limit

    4. Require run-as-non-root

    5. [Restart Deployment On Configmap Change]

๐Ÿš€ Build Kyverno policy from code

  • Source code:

      โšก $ tree src/
      โ”œโ”€โ”€ imports
      โ”‚   โ””โ”€โ”€
      โ”œโ”€โ”€ kyverno-policies
      โ”‚   โ”œโ”€โ”€ deny-delete-resources.ts
      โ”‚   โ”œโ”€โ”€ kverno-list.ts
      โ”‚   โ”œโ”€โ”€ kyvernoProps.ts
      โ”‚   โ”œโ”€โ”€ require-app-labels.ts
      โ”‚   โ”œโ”€โ”€ require-requests-limits.ts
      โ”‚   โ””โ”€โ”€ require-runasnonroot.ts
      โ”œโ”€โ”€ main.ts
      โ””โ”€โ”€ test-yaml
          โ”œโ”€โ”€ inflate-negative-test-deployment.yaml
          โ””โ”€โ”€ inflate-positive-test-deployment.yaml
      3 directories, 10 files
  • Build

      โšก $ npx projen build
      ๐Ÿ‘พ build ยป default | ts-node --project .projenrc.ts
      ๐Ÿ‘พ build ยป compile | tsc --build
      ๐Ÿ‘พ build ยป post-compile ยป synth | cdk8s synth
      No manifests synthesized
      ๐Ÿ‘พ build ยป test | jest --passWithNoTests --all --updateSnapshot
      No tests found, exiting with code 0
      File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
      All files |       0 |        0 |       0 |       0 |
      ๐Ÿ‘พ build ยป test ยป eslint | eslint --ext .ts,.tsx --fix --no-error-on-unmatched-pattern src test build-tools projenrc .projenrc.ts
  • Output yaml files

      โšก $ tree dist/
      โ””โ”€โ”€ kyverno
          โ”œโ”€โ”€ require-app-label-kyverno-policy.yaml
          โ”œโ”€โ”€ require-request-limit-kyverno-policy.yaml
          โ””โ”€โ”€ run-as-non-root-kyverno-policy.yaml
      1 directory, 3 files

๐Ÿš€ Apply and test

  • Apply policies and check the result

      โšก $ kubectl apply -f dist/kyverno/ configured configured configured
  • Test negative, the deployment inflate-negative-test-deployment.yaml does not have resource limit and request and enable runAsNonRoot

      โšก $ kubectl apply -f src/test-yaml/inflate-negative-test-deployment.yaml
      Error from server: error when creating "src/test-yaml/inflate-negative-test-deployment.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:
      policy Deployment/default/inflate-negative-test for resource violations:
      require-app-label: {}
        autogen-require-request-limit: 'validation error: All containers must have CPU and
          memory resource requests and limits defined. rule autogen-require-request-limit
          failed at path /spec/template/spec/containers/0/resources/limits/'
  • Test positive

      kubectl apply -f src/test-yaml/inflate-positive-test-deployment.yaml
      deployment.apps/inflate-positive-test created
  • Test without non-root user enabled because the validation failure action is AUDIT so the deployment is applied successfully

      โšก $ kubectl apply -f src/test-yaml/inflate-without-nonroot-test-deployment.yaml
      deployment.apps/inflate-without-nonroot-test created
  • But let's view the policy violations

      โšก $ kubectl describe polr polr-ns-default | grep inflate -A15 -B10| grep "Result: \+fail" -B10
          Seconds:  1661326749
        Category:   Pod Security Standards
        Message:    validation error: Containers must be required to run as non-root users. This policy ensures runAsNonRoot is set to true. rule autogen-run-as-non-root[0] failed at path /spec/template/spec/securityContext/runAsNonRoot/ rule autogen-run-as-non-root[1] failed at path /spec/template/spec/containers/0/securityContext/
        Policy:     run-as-non-root
          API Version:  apps/v1
          Kind:         Deployment
          Name:         inflate-without-nonroot-test
          Namespace:    default
          UID:          b05068c1-425c-41f4-ae0f-c913100a1c9c
        Result:         fail

๐Ÿš€ Test Restart Deployment On Configmap Change

  • Changing configmap requires rollout restart of deployments which reference to that configmap. We can use kyverno to automate this for us.

  • Create kyverno policy to watch a Configmap and if it changes will write an annotation to one or more target Deployments thus triggering a new rollout and thereby refreshing the referred Configmap

  • First we need to grant additional privileges to the Kyverno ServiceAccount for updating apps.deployments resources through Aggregated ClusterRoles

    • Kyverno has clusterrole with aggregationRule which will combine all clusterrole with label app: kyverno into one in aggregation

          - matchLabels:
              app: kyverno
    • Create a new kyverno cluster role to inject into the main one kyverno-clusterrole.ts.

  • Kyverno policy to Restart Deployment On Configmap Change: restart-on-configmap-changes.ts

  • Rebuild the project to generate manifest yaml files. npx projen build

      โšก $ tree dist/
      โ”œโ”€โ”€ kyverno
      โ”‚   โ”œโ”€โ”€ require-app-label-kyverno-policy.yaml
      โ”‚   โ”œโ”€โ”€ require-request-limit-kyverno-policy.yaml
      โ”‚   โ”œโ”€โ”€ restart-on-configmap-change-policy.yaml
      โ”‚   โ””โ”€โ”€ run-as-non-root-kyverno-policy.yaml
      โ””โ”€โ”€ role
          โ””โ”€โ”€ kyverno-create-deployments-clusterrole.yaml
      2 directories, 5 files
  • Apply clusterrole and policy then test using inflate-positive-test-deployment.yaml and inflate-test-configmap.yaml

      โšก $ kv7 get cpol restart-on-configmap-change
      NAME                          BACKGROUND   ACTION    READY
      restart-on-configmap-change   true         audit     true
      โšก $ kv7 get deploy -l app=inflate-positive-test
      NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
      inflate-positive-test   1/1     1            1           62m
      โšก $ kv7 get cm -l app=inflate-test-configmap
      NAME                     DATA   AGE
      inflate-test-configmap   2      64m
  • We now update the configmap to see kyverno rollout restart the deployment

      โšก $ kv7 apply -f inflate-test-configmap.yaml
      configmap/inflate-test-configmap configured
      ~ $ kv7 get pod -l app=inflate-positive-test --watch
      NAME                                     READY   STATUS    RESTARTS   AGE
      inflate-positive-test-668477b686-cdggl   1/1     Running   0          3m3s
      inflate-positive-test-59bb77549c-lxcjx   0/1     Pending   0          0s
      inflate-positive-test-668477b686-cdggl   1/1     Terminating   0          3m9s
      inflate-positive-test-59bb77549c-lxcjx   0/1     Pending       0          0s
      inflate-positive-test-59bb77549c-lxcjx   0/1     ContainerCreating   0          0s
      inflate-positive-test-59bb77549c-lxcjx   1/1     Running             0          1s
      inflate-positive-test-668477b686-cdggl   1/1     Terminating         0          3m11s
      inflate-positive-test-668477b686-cdggl   1/1     Terminating         0          3m11s
  • Reference:

๐Ÿš€ Conclusion

  • Someone said Kyverno policy as code but the code is in yaml language, it's not an actual programming language.

  • Using CDK8S to generate Kyverno policy help to leverage the strong programming skill of the developer and structure project more efficiently.