AWS — Configuring a Private NLB with Nginx Ingress on Kubernetes Using IP Target Type

In this guide, we’ll walk through configuring a private Network Load Balancer (NLB) with Nginx Ingress on Kubernetes, leveraging IP target type for efficient traffic management. This setup is particularly useful for routing traffic from API Gateway (APIGW) REST API to your Kubernetes pods via an internal NLB with VPC Link

If you are using REST API gateway, which currently supports only NLB as a VPC link target, this tutorial is for you.

To overcome this issue and still be able to operate at layer 7 using paths, we introduced an internal network load balancer that uses NGINX, which is controlled by a load balancer controller within the cluster.

The final result should look like this:

With NGINX Ingress, even though it’s a network load balancer, we can leverage the usage of paths, instead of having to add an application load balancer, as suggested by AWS here: AWS Blog Post

Prerequisites

1. Amazon EKS Cluster: Ensure you have an EKS cluster up and running.
2. Helm: Used for deploying Nginx Ingress and AWS Load Balancer Controller.

Reasons to use Target Type as IP

1. The target you’re aiming for doesn’t have to be a single server; anything with a private IP address will do. This includes internal load balancers, VPC private services, Fargate containers, databases, and on-premise servers accessed through a VPN.
2. It’s also possible to target resources in different regions, as long as you have cross-region peering set up between your Virtual Private Clouds (VPCs).
3. Instead of sending requests to the server itself, the request will go directly to the relevant pod without needing to search across multiple servers to find the right one.
4. For dynamic environments in Kubernetes, target-type: ip is often the better choice due to its direct routing and efficiency.

Step-by-Step Guide

1. Install Amazon VPC CNI Plugin

The Amazon VPC CNI plugin is necessary for enabling pod networking within your cluster. This is crucial as the plugin supports native AWS VPC networking configuration for pod IP addresses.

• Install the Amazon VPC CNI plugin using the provided EKS add-ons or via Helm.
• You can also install it using the AWS Portal by just going to your kubernetes cluster Addon tab:

2. Install Nginx Ingress Controller

We’ll use the Nginx Ingress Controller to manage the routing of external traffic to internal services.

• Repository: Nginx Ingress Controller

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install [RELEASE_NAME] ingress-nginx/ingress-nginx

• Configuration Parameters:

controller:
  ingressClassByName: true
  ingressClassResource:
    name: nginx-ingress-controller
    enabled: true
    default: false
    controllerValue: "k8s.io/ingress-nginx-internal"
  kind: DaemonSet
  service:
    type: LoadBalancer
    external:
      enabled: false
    externalTrafficPolicy: Local
    internal:
      enabled: true
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip" #<- IMPORTANT
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: yoursslcertarn
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
        service.beta.kubernetes.io/aws-load-balancer-target-type: ip
        service.beta.kubernetes.io/aws-load-balancer-type: nlb #<- IMPORTANT
        service.beta.kubernetes.io/aws-load-balancer-name: "k8s-nlb"
        service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
        service.beta.kubernetes.io/aws-load-balancer-subnets: "sub1,sub2"

This configuration ensures the Nginx Ingress Controller uses an internal NLB with IP target type.

3. Install AWS Load Balancer Controller

The AWS Load Balancer Controller is responsible for managing AWS load balancers.

• Repository: AWS Load Balancer Controller

helm repo add eks https://aws.github.io/eks-charts
# If using IAM Roles for service account install as follows -  NOTE: you need to specify both of the chart values `serviceAccount.create=false` and `serviceAccount.name=aws-load-balancer-controller`
helm install aws-load-balancer-controller eks/aws-load-balancer-controller --set clusterName=my-cluster -n kube-system --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller
# If not using IAM Roles for service account
helm install aws-load-balancer-controller eks/aws-load-balancer-controller --set clusterName=my-cluster -n kube-system

• Configuration Parameters:

serviceAccount:
  create: true
  name: "aws-load-balancer-controller"
clusterName: "clustername"
image:
  repository: "public.ecr.aws/eks/aws-load-balancer-controller"
region: ap-east-1
defaultTargetType: "ip"

4. Create the Ingress Resource

Define your Ingress resource to route traffic to your services.

• Ingress Manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test
  annotations:
    ingressclass.kubernetes.io/is-default-class: 'true' #<- VERY IMPORTANT
    kubernetes.io/ingress.class: nginx-ingress-controller #<- VERY IMPORTANT
spec:
  ingressClassName: nginx-ingress-controller #<- VERY IMPORTANT
  rules:
    - host: myowndomain.com # replace with your domain
      http:
        paths:
          - pathType: Prefix
            path: "/internal/api/v1.0/your/api/path"
            backend:
              service:
                name: your-pod-servicename
                port:
                  number: 80

Troubleshooting

If you encounter issues, such as old DNS names being cached or configurations not updating correctly:

1. Recreate the Resources: Delete and recreate the Nginx Ingress Controller and associated services.
2. Clear Caches: Ensure no old configurations are cached by DNS or load balancer services.

Summary

1. Nginx Load Balancer Controller: Configured to use an internal NLB with IP target type.
2. AWS Load Balancer Controller: Manages AWS load balancers with the necessary configurations.
3. Amazon VPC CNI Plugin: Ensures pod networking is configured properly.
4. Ingress Resource: Directs traffic to your pods using Nginx Ingress.

By following these steps, you should have a robust setup for routing traffic through a private NLB with Nginx Ingress on Kubernetes, optimizing for internal traffic management.