Tools Resources Plans

Cloud Security K8s

NGINX Custom Snippets CVE-2021-25742

Gafnit Amiga

Wednesday, Oct 27th, 2021

Attackers can gain access to secrets across all namespaces

The high severity alert known otherwise as CVE-2021-25742, was recently brought to the public’s attention and has prompted us to believe that it may be worthwhile to do a deeper dive into what this vulnerability really is and what it means for today’s organizations. Let’s jump right in!

Here’s the CVE itself:

CVE-2021-25742: Ingress-nginx custom snippets allows retrieval of ingress-nginx serviceaccount token and secrets across all namespaces

What is this CVE about?

CVE-2021-25742 is about a low privilege user that is able to gain access to an “ingress-nginx” service account with the permission to list secrets in all namespaces in the cluster. If an organization doesn’t have the right version of nginx-ingress-controller that allows them to change the default value of “allow-snippet-annotations” to “false” this can lead to the risky configuration which may result in a cluster takeover.

As we have written about before, the ability to list all secrets is one of the simplest ways to escalate privileges in Kubernetes creating increased risk to a cloud environment, and allowing access to any service account token in the cluster. This could result in a malicious user changing existing applications, gaining access to sensitive information stored in configuration mapping and secrets, and gaining control over the cluster in its entirety.

Risky even in the latest version, nginx-ingress-controller still uses cluster role and cluster role bindings which allow the “ingress-nginx” service account to list all secrets in the cluster.

Here is the deployment file for the latest nginx-ingress-controller version (v1.0.4) which creates as part of the deployment, a ClusterRole and ClusterRoleBinding named “ingress-nginx”:.

Putting the pieces together

What are Ingress and Ingress controller?

Ingress is a Kubernetes resource, enabling the user to define the rules that route your traffic from outside the cluster to services within the cluster. This can be completed without needing to have a dedicated load balancer for every service. An Ingress controller is responsible for fulfilling the Ingress resource rules. Therefore, having an Ingress without an Ingress controller is completely ineffectual. The user needs the Ingress controller to implement the Ingress rules.

You can learn more about how Ingress controllers work here.

There are many different Ingress controllers that can be deployed in Kubernetes, one of which is the NGINX Ingress Controller. ingress-nginx is an Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer. This is arguably one of the most popular Ingress controllers available, and the open source code of this controller is even managed by Kubernetes itself.

The nginx-ingress-controller resources are in a dedicated namespace called “ingress-nginx”.

kubectl get all -n ingress-nginx

Understanding Custom Snippets

When defining Ingress resource objects you can add annotations to customize their behavior. Some of them are snippet annotations which enable users to insert configuration snippets for the nginx.conf configuration file (located on the nginx-ingress-controller pod).

There are 4 types of snippets that can be defined:

Here’s an example for an Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:  
    nginx.ingress.kubernetes.io/server-snippet: |    
      location / {      
        return 200 'OK';    
      }  
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - http:    
    paths:      
      - path: /exp        
        pathType: Prefix        
        backend:          
          service:            
            name: some-service            
            port:              
              number: 1234

Here is some explanation from NGINX on how to use snippets. “Snippets allow you to insert raw NGINX config into different contexts of the NGINX configurations that the Ingress Controller generates. These should be used as a last-resort solution in cases where annotations and ConfigMap entries cannot help. Snippets are intended for advanced NGINX users who need more control over the generated NGINX configuration.”

As a result of this functionality, a user that has the permissions to create or update Ingress resources can control the content of these snippet annotations and add almost any NGINX configuration section to the nginx.conf file.

Securing Custom Snippets

But what if the cluster administrator does not want to allow this permission? What if the desired state is that only the Ingress controller administrator will be allowed to control the customized parts of the nginx.conf file?

This is why we have the allow-snippet-annotations configuration. The configuration of the nginx-ingress-controller is located on a ConfigMap named “ingress-nginx-controller” in the “ingress-nginx” namespace.

The Pod of the NGINX Ingress controller is in the “ingress-nginx” namespace as well (alongside all the server logic and the nginx.conf file).The nginx Ingress Controller administrator can overwrite nginx-controller configuration values by adding key-val value pairs to the data section of the ConfigMap.

kubectl get configmap ingress-nginx-controller -n ingress-nginx -o yaml

data:
  allow-snippet-annotations: "true"

All configuration options can be found here.

Note that the default value of the “allow-snippet-annotations” is “true”.

If the nginx-ingress-controller administrator wishes to prevent users from adding snippet annotations to their Ingress resources, the administrator needs to explicitly change the value to “false”.

data:
  allow-snippet-annotations: "false"

If the value of the “allow-snippet-annotation” is “false” in the “ingress-nginx-controller” ConfigMap, then even if the user has the permission to create or update an Ingress resource, the user will not be authorized to add a snippet annotation such as “nginx.ingress.kubernetes.io/server-snippet”.

annotations:  
  nginx.ingress.kubernetes.io/server-snippet: |    
    location / {      
      return 200 'OK';    
    }

In such cases, the nginx-ingress-controller will simply drop these configurations.

https://github.com/kubernetes/ingress-nginx/pull/7665/files#diff-4198ec010671801881244a8052177f31bcbc682c99fbd7391bceb136025c0568R515

The recommended mitigation

The bug described earlier in this post affects ingress-nginx. Multi-tenant environments where non-admin users have permissions to create Ingress objects are most impacted by this issue.

In the CVE, the mitigation recommended is in two parts. First, you will need to upgrade to a version that allows for mitigation, which is (>= v0.49.1 or >= v1.0.1). Then, once this is complete, you will need to set the “allow-snippet-annotations” configuration to false.

Check your nginx-ingress-controller version

POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -l app.kubernetes.io/component=controller -n ingress-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -n ingress-nginx -- /nginx-ingress-controller –version

If your version is v.1.0.0 or <=v.0.49.1 please update to a version that allows mitigation. We recommend upgrading to the latest version.
Check the “allow-snippet-annotations” configuration even if you just upgraded to the latest version (as the default value is “true”).
```
kubectl get configmap ingress-nginx-controller -n ingress-nginx -o jsonpath='{.data}'
```
Follow the instructions to change the value of “allow-snippet-annotations” to “false”

Like with certain other CVEs, the mitigation offered is far from fool proof. It’s clear that many organizations will remain unprotected, either because they have not upgraded to the right version, or because they do not change the risky default configuration.

Even for those who would like to change the configuration, there will be many cases where snippets have already been used to create custom code that meets a genuine business need. It’s simply not realistic to expect these users to look for alternate solutions or pull apart their existing configurations and stop using the feature.

Merging graph observability and security in Kubernetes for real mitigation

A major capability to handling such cases is having a full view of your entire cloud infrastructure, including all permissions and user roles, to ensure that only trusted users can create and update Ingress resources in the first place.

See which ClusterRoles allow the creation of or updates to Ingress resources:

kubectl get clusterroles -o json | jq -r '.items[] | select(.rules[] | select(.resources != null) | select(.resources[] == "ingresses") | select(.verbs[] | (.== "create" or . == "update" or . == "*"))) | .metadata.name' | uniq

See which Roles allow the creation of or updates to Ingress resources:

kubectl get ns | awk {'print $1'} | tail -n +2 | xargs -I {} kubectl get roles -n {} -o json | jq -r '.items[] | select(.rules[] | select(.resources != null) | select(.resources[] == "ingresses") | select(.verbs[] | (.== "create" or . == "update" or . == "*"))) | .metadata |"RoleName: \(.name); RoleNamespace: \(.namespace)"' | uniq

Using these ClusterRoles and Roles bindings, it is possible to know exactly who can exploit this vulnerability in your cluster.