Kubernetes Cluster: Attack and Defense Perspective Part-3

Hello all,

Welcome to the third post on the series of Kubernetes Cluster from attackers and defensers perspective

In kubernetes when we create secrets, it get stored in etcd database which in general it stored in base64 which can easily get decoded. You created secrets for PODS, actually sometimes POD needs some credentials to process or run the server for ex, MYSQL server, so you can either mount the secrets as volume inside POD or you can expose secrets as Environment Variables. In this post we will focus on how we can securely expose secrets inside POD by using Environment variables. We will see how an attacker can access the secrets and how we can defend from an attacker to potentially access the secrets.

So this is our secret that we created in a cluster


apiVersion: v1

kind: Secret

metadata:

  name: db-creds

type: Opaque

data:

  password: d2ViYWRtaW4xMjMK

Now we can expose secrets inside POD as Environment Variables like this below:-


env:

- name: MYSQL_ROOT_PASSWORD

  valueFrom:

   secretKeyRef:

    name: db-creds

    key: password

And when we run `env` command inside the POD we can clearly see the secret were successfully exposed to POD env variables.

Also I am running a Deployment where I have a simple application running with privileged service account named utk2 and the application is vulnerable to SSRF issue.

Where I am able to access the token file from /var/run/.../ as shown in below image

So upon getting the service account token, the next step is to find the Kubernetes API Port, so we can use nmap on that host to check the Port Numbers and Services running on Ports.

So our target port is 55683

Now we can check the secrets if this user account have access by using the API that is exposed on port 55683

And we can easily get access to the secrets

Now lets understand how we can secure secrets even when the service account have higher privileges.

Vault Secrets

Now we will create vault and we will store secrets inside vault. Here you can see I created a vault secrets with a password. There are tons of documents where we can find how to store secrets inside vault.

For injecting secrets from vault into pod we have to create a sidecar annotation. Here is the annotation for injecting secrets as env variables inside POD.


template:

  metadata:

    annotations:

        vault.hashicorp.com/agent-inject: 'true'

        vault.hashicorp.com/role: 'policy'

        vault.hashicorp.com/agent-inject-secret-config: 'internal/data/database/config'

        vault.hashicorp.com/agent-inject-template-config: |

          

            export MYSQL_ROOT_PASSWORD_COMING_FROM_VAULT="\{\{.Data.data.password\}\}"

Also, we need to create a policy which has the capability to read the secrets from vault


command: vault policy write policy 



path "internal/data/database/config" {

  capabilities = ["read"]

}

Also we need to create a authentication role for kubernetes which will bind with the capability we defined above with the service account. The role will help in cross check the access when request received by Vault.


vault write auth/kubernetes/role/policy \

    bound_service_account_names=utk2 \

    bound_service_account_namespaces=default \

    policies=policy \

    ttl=24h

So once everything sets up, we can apply deployment and check if we are able to inject secrets as ENV variables. Check last 3rd Line.

When the pod is up and running, we can see the file config get stored inside /vault/secrets/ directory -

There will be two containers run in a POD. One of them is our container defined in deployment and other one is the vault agent sidecar which is used to verify the kubernetes authentication when the client or our container defined in deployment requests vault secrets.

So when we tried to access the secret from the service account, there is no secrets available even we successfully injected secret inside POD.

Hence we secured our secret being exploited, even when privileged service account were in use.

Wrap Up

Security learning from above environment -

- Do not give excessive Permission to any service account
- Do not create create instead use Vault and expose secrets securely.
- Do not expose Kubernetes API publicly, if exposed, it should be authenticated.

That's all for this blog post, We will talk more about kubernetes security in future posts

Contact me on Twitter at agrawalsmart7x

Hope you found this blog useful

Thanks for your time

← Previous Post Next Post →