Integrating Existing Helm Charts with Terraform for Kubernetes Management

In this blog post, we’re going to explore how to add existing Helm charts to your Terraform setup. Whether you’re looking to streamline your deployments or just curious about how to integrate these tools effectively. Let’s get started!

The Terraform Code

We need to first have the helm provider, Please adjust it so that you can connect to your own K8s cluster:

terraform.tf

provider "helm" {
  kubernetes {
    host = eks_cluster_endpoint
    cluster_ca_certificate = base64decode(eks_cluster_certificate_authority_data)
    config_path            = ""
    config_context         = ""

    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", eks_cluster_name]
    }
  }
}

In this post, we will be using metrics server as an example. You can set up its configuration like the following:

main.tf

resource "helm_release" "metrics_server" {
  name       = "metrics-server"
  namespace  = "kube-system"
  repository = "https://kubernetes-sigs.github.io/metrics-server"
  chart      = "metrics-server"
  version    = "3.12.1"
}

The chart repository can be found here:

https://kubernetes-sigs.github.io/metrics-server/

Assuming that you have deployed the resource before, as soon as you run terraform plan, you immediately get the following issue:

This will not only happen with the deployment but with all the resources deployed by the helm chart.

If you are migrating deployment away from ArgoCD back to helm, this will also throw the same error!

So, How do I Fix That?

Terraform is complaining that 2 annotations are missing in that specific resource, but it will complain about all resources if you fix them one by one. There is no straightforward way to find all the resources deployed by a helm chart using Kubectl only, but here is what has worked for me:

kubectl get sa,role,rolebinding,clusterrole,clusterrolebindings,deployments,daemonsets -n kube-system | grep -i metrics-server

• The first part of the pipe lists resources such as service accounts, roles, role bindings, and deployments in the kube-system namespace (where I deployed this metric-server chart before)
• The second part filters the results to show only those related to metrics-server, it will be something like this:

From this output, we know that we need to update:

• Service Account: 1
• RoleBinding: 1
• ClusterRole: 2
• ClusterRoleBinding: 2
• Deployment: 1

You need to patch all of these, otherwise they will keep throwing that error:

For each service, edit them and add the annotations required in the error message:

kubectl edit sa metrics-server -n kube-system

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    meta.helm.sh/release-name: metrics-server   #<---ADD THIS
    meta.helm.sh/release-namespace: kube-system #<---ADD THIS

After patching all the resources and run terraform apply again, there is one more that you can’t easily find with GUI tools like Lens or k8s dashboard, which is the API service:

This happens because your v1beta.metrics.k8s.io API created by metrics-server chart has none of these requested annotations as well, let’s first find them:

kubectl api-resources

If you see the output, we can actually confirm that the resource is there:

So, let’s edit the resource and add the annotations:

kubectl edit apiservice v1beta1.metrics.k8s.io

Again, the same two lines:

    meta.helm.sh/release-name: metrics-server
    meta.helm.sh/release-namespace: kube-system

After that, you can finally run terraform apply and enjoy a happy ending:

Conclusion

With the steps we’ve covered, you should now have a good understanding on how to integrate Helm charts into your Terraform projects.

Happy Terraforming and keep studying!
Follow me on Linkedin

Update:

I came up with the following script that helped me to fill in the annotations quickly:

#!/bin/bash


NAMESPACE=kube-system


# Define the annotations
ANNOTATIONS=(
  "meta.helm.sh/release-name=metrics-server"
  "meta.helm.sh/release-namespace=kube-system"
)

# Define the resources to annotate
RESOURCES=(
  "serviceaccount/metrics-server"
  "rolebinding/metrics-server-auth-reader"
  "clusterrole/system:metrics-server"
  "clusterrole/system:metrics-server-aggregated-reader"
  "clusterrolebinding/metrics-server:system:auth-delegator"
  "clusterrolebinding/system:metrics-server"
  "deployment/metrics-server"
)

# Annotate each resource
for RESOURCE in "${RESOURCES[@]}"; do
  kubectl annotate $RESOURCE ${ANNOTATIONS[@]} -n $NAMESPACE
done

So, basically you get the output from:

kubectl get csidriver,sa,role,rolebinding,clusterrole,clusterrolebindings,deployments,daemonsets -n kube-system | grep -i metrics-server

and add to the resources block:

Finally, you get your resources annotated:

Much better, eh?

Want more insights on DevOps, security, and automation? Don’t miss out — Follow me!

Connect with me on Linkedin!