EKS Error — CSI Driver Failing to Sync Secrets

In EKS, secrets can be synchronized directly from AWS Secrets Manager, eliminating the need to hardcode them as base64-encoded Kubernetes secrets.

Kubernetes utilizes the Secrets Store CSI (Container Storage Interface) Driver. This driver allows Kubernetes pods to dynamically access secrets stored in external secret management systems like AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault.

The CSI driver works by mounting a volume into the pod, where the secrets are exposed as files. It integrates with Kubernetes’ native Secret API, enabling automatic secret injection during pod creation. The driver continuously syncs updates from the external secret manager, ensuring that pods always have the latest secret values without manual intervention.

However, there are times when the secret sync simply does not work.

The Problem

I came to this issue recently when I tried to sync a secret from the AWS Secrets Manager, it gave me a very misleading error:

Failed to fetch secret from all regions

What does that mean? Well this is actually only part of the full problem. It gives you this message after trying to sync the secret, but it can be many things:

secret name is wrong, or the key could not be found, or even JMESPath query is invalid. Let me explain:

JMESPath is a query language designed for filtering, transforming, and extracting specific data from JSON documents. It provides a simple yet powerful syntax to navigate through JSON structures and retrieve the information you need

So, you can have something like:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: config-manager-secrets-csi
  namespace: my-app
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "dev/my-app/secrets"
        objectType: "secretsmanager"
        jmesPath:
          - path: JWT_PUBLIC_KEY
            objectAlias: JWT_PUBLIC_KEY

This is going to query JWT_PUBLIC_KEY in the secrets when it fetches from secrets manager.

If it cannot find this query, it will throw an error similar to the first screenshot.

But how do we find that this is actually the issue?

The Solution

There is a daemonset in the kubesystem that syncs the secrets, you can find it with the following command:

kubectl get ds -n kube-system

More details about this csi driver you can find here:

GitHub - aws/secrets-store-csi-driver-provider-aws: The AWS provider for the Secrets Store CSI…

You can check its logs, and you can find not only this, but also any other thing related to the sync:

kubectl logs -f --selector app=secrets-store-csi-driver-provider-aws -n kube-system --max-log-requests=10

In my case, a variable was missing, and this is because my app was not using it anymore, so I removed it from the secrets in the secrets manager:

Notice that the first error is also there in red. So basically, the pod log printed out the last line only.

You can find other problems, such as a wrong service account or invalid permissions to fetch secrets with the used service account. Just keep digging these logs, and you’ll eventually find out.

Conclusion

To investigate issues with secret synchronization, it can be helpful to review the logs of the components responsible for managing the process, such as the secrets-store-csi-driver-provider-aws and the secrets-store-csi-driver. These logs can provide useful information about potential errors or configuration issues affecting the synchronization.

Follow me on Linkedin

Clap if you liked this content!