How do we install dynamic modules (non support officially) on Nginx Ingress Controller? (terraform, helm chart)

Question

I'm managing Kubernetes + nginx.

I'd like to install dynamic modules on nginx that are provided by Nginx Ingress Controller. Those dynamic modules are not offered by Nginx Ingress Controller official configmap (https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/)

So I believe, I need to build my own Docker container of Nginx Ingress Controller. (Could be added at this? https://github.com/kubernetes/ingress-nginx/blob/8951b7e22ad3952c549150f61d7346f272c563e1/images/nginx/rootfs/build.sh#L618-L632 )

Do you know how we can customize the controller and manage it by helm chart? I'm thinking about making a Fork branch from the controller master repo on Github. But I don't have any idea on how we install a customized version of the controller on terraform + helm chart.

However, I would prefer to use a non-customizable solution (because of some annotation settings)

Environment: Kubernetes Nginx Ingress Controller is installed by helm chart + terraform Nginx Ingress Controller -> https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx

Terraform:

resource "helm_release" "nginx-ingress-controller" {
  name      = "nginx-ingress-controller"
  chart     = "ingress-nginx/ingress-nginx"
  namespace = "kube-system"
  version   = "3.34.0"
}

dynamic modules https://docs.nginx.com/nginx/admin-guide/dynamic-modules/dynamic-modules/ (install process might be using --add-dynamic-module option, and set load_module modules/something.so on nginx.conf via ingress.yaml)

Thank you.

Answer 1

TL;DR

Extend the official image with the dynamic modules, and update the helm_release terraform resource to set the controller.image.registry, controller.image.image, controller.image.tag, controller.image.digest, and controller.image.digestChroot for your custom image along with a controller.config.main-snippet to load the dynamic module(s) in the main context.

---

This is similar to my previous answer for building modules using the official nginx image. You can extend the ingress-nginx/controller image, build the modules in one stage, extend the official image with the dynamic modules in another stage, and use the image in your helm_release. An example for extending the ingress-nginx/controller with the echo-nginx-module e.g.:

Docker

ARG INGRESS_NGINX_CONTROLLER_VERSION
FROM registry.k8s.io/ingress-nginx/controller:${INGRESS_NGINX_CONTROLLER_VERSION} as build

ARG INGRESS_NGINX_CONTROLLER_VERSION
ENV INGRESS_NGINX_CONTROLLER_VERSION=${INGRESS_NGINX_CONTROLLER_VERSION}

USER root
RUN apk add \
        automake \
        ca-certificates \
        curl \
        gcc \
        g++ \
        make \
        pcre-dev \
        zlib-dev

RUN NGINX_VERSION=$(nginx -V 2>&1 |sed -n -e 's/nginx version: //p' |cut -d'/' -f2); \
    curl -L "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" | tar -C /tmp/nginx --strip-components=1 -xz

WORKDIR /src/echo-nginx-module
RUN curl -L https://github.com/openresty/echo-nginx-module/archive/refs/tags/v0.63.tar.gz | tar --strip-components=1 -xz

WORKDIR /tmp/nginx
RUN ./configure --with-compat --add-dynamic-module=/src/echo-nginx-module && \
    make modules

FROM registry.k8s.io/ingress-nginx/controller:${INGRESS_NGINX_CONTROLLER_VERSION}

COPY --from=build /tmp/nginx/objs/ngx_http_echo_module.so /etc/nginx/modules/

... build and push the image e.g.: docker build --rm -t myrepo/ingress-nginx/controller:v1.5.1-echo --build-arg INGRESS_NGINX_CONTROLLER_VERSION=v1.5.1 . && docker push myrepo/ingress-nginx/controller:v1.5.1-echo

Terraform

Update the terraform helm_release resource to install the charts using the custom image and adding a main-snippet to set the load_module directive in the main context:

resource "helm_release" "ingress-nginx" {
  name       = "ingress-nginx"
  namespace  = "kube-system"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  version    = "3.34.0"
  set {
    name = "controller.image.registry"
    value = "myrepo"
  }
  set {
    name = "controller.image.image"
    value = "ingress-nginx/controller"
  }
  set {
    name = "controller.image.tag"
    value = "v1.5.1-echo"
  }
  set {
    name = "controller.image.digest"
    value = "sha256:1b32b3e8c983ef4a32d87dead51fbbf2a2c085f1deff6aa27a212ca6beefcb72"
  }
  set {
    name = "controller.image.digestChroot"
    value = "sha256:f2e1146adeadac8eebb251284f45f8569beef9c6ec834ae1335d26617da6af2d"
  }
  set {
    name = "controller.config.main-snippet"
    value = <<EOF
load_module /etc/nginx/modules/ngx_http_echo_module.so;
EOF
  }
}

The controller.image.digest is the image RepoDigest: docker inspect myrepo/ingress-nginx/controller:v1.5.1-echo --format '{{range .RepoDigests}}{{println .}}{{end}}' |cut -d'@' -f2

The controller.image.digestChroot is the Parent sha: docker inspect myrepo/ingress-nginx/controller:v1.5.1-echo --format {{.Parent}}

Test

1. Create a nginx pod: kubectl run nginx --image=nginx
2. Expose the pod: kubectl expose pod nginx --port 80 --target-port 80
3. Create an ingress with a server-snippet:

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/server-snippet: |
      location /hello {
        echo "hello, world!";
      }
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: echo.example.com
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: nginx
            port:
              number: 80
  tls:
  - hosts:
    - echo.example.com
    secretName: tls-echo
EOF

Using cert-manager for TLS certificates issuance and external-dns for DNS management.

4. Test using curl: