Nginx `proxy_ssl_trusted_certificate` with letsencrypt upstream

Question

I'm trying to use a proxy_pass with nginx where the connection to the upstream server is encrypted. The certificate of the upstream server has been created by a letsencrypt certbot.

# upstream server: nginx.conf

stream {
  server {
    listen 636 ssl;

    ssl_certificate /etc/letsencrypt/live/upstream.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/upstream.example.com/privkey.pem; # managed by Certbot

    # ...
  }
}

When not verifying the proxy certificate in the downstream server, everything works fine.

# downstream server: nginx.conf

stream {
  server {
    listen 636 ssl;

    ssl_certificate /etc/letsencrypt/live/downstream.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/downstream.example.com/privkey.pem; # managed by Certbot

    proxy_ssl on;
    proxy_ssl_verify off;

    proxy_pass upstream.example.com:636;

    # ...

  }
}

However, if I'm trying to verify the upstream certificate on the downstream server, I'm getting an upstream SSL certificate verify error: (2:unable to get issuer certificate) while SSL handshaking to upstream in the nginx error log.

# downstream server: nginx.conf

stream {
  server {
    listen 636 ssl;

    ssl_certificate /etc/letsencrypt/live/downstream.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/downstream.example.com/privkey.pem; # managed by Certbot

    proxy_ssl on;
    proxy_ssl_verify on;
    proxy_ssl_trusted_certificate /etc/nginx/ssl/upstream.example.com/chain.pem;
    proxy_ssl_verify_depth 2;

    proxy_pass upstream.example.com:636;

    # ...

  }
}

Which settings for proxy_ssl_trusted_certificate and proxy_ssl_verify_depth do I need if the upstream server I'm trying to connect to has a letsencrypt certificate?

I've varied proxy_ssl_verify_depth from 0 to 5, and I've used the upstream server's fullchain.pem, chain.pem, and cert.pem for proxy_ssl_trusted_certificate, but each was unsuccessful.

Additional info

Verifying the CA certificate with openssl works:

# openssl verify -verify_depth 2 chain.pem
chain.pem: OK

Verifying the certificate from the upstream server, fullchain.pem, against the CA certificate works:

# openssl verify -CAfile chain.pem fullchain.pem
fullchain.pem: OK

Further references

• https://docs.nginx.com/nginx/admin-guide/security-controls/securing-http-traffic-upstream/
• https://nginx.org/en/docs/http/ngx_http_proxy_module.html

Answer 1

The CA certificate required for proxy_ssl_trusted_certificate is not provided by letsencrypt or the upstream server. It is already installed on the downstream server.

On Ubuntu, the location of the CA certificate(s) is /etc/ssl/certs/ca-certificates.crt.

# downstream server: nginx.conf

stream {
  server {
    listen 636 ssl;

    ssl_certificate /etc/letsencrypt/live/downstream.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/downstream.example.com/privkey.pem; # managed by Certbot

    proxy_ssl on;
    proxy_ssl_verify on;
    proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;

    proxy_pass upstream.example.com:636;

    # ...

  }
}

Documentation

https://nginx.org/en/docs/http/ngx_http_proxy_module.html

proxy_ssl_verify on | off;

Enables or disables verification of the proxied HTTPS server certificate.

proxy_ssl_verify_depth number;

Sets the verification depth in the proxied HTTPS server certificates chain.

proxy_ssl_trusted_certificate file;

pecifies a file with trusted CA certificates in the PEM format used to verify the certificate of the proxied HTTPS server.

See also

• https://unix.stackexchange.com/a/484938/91720
• https://serverfault.com/a/1052976/167331