Cannot login to minio behind nginx proxy

Question

I cannot login to minio console behind nginx proxy. Both are started as docker container with docker compose. I can login to minio without proxy by accessing localhost:9001, but I cannot when behind proxy. I got response 401 invalid login, though I am using the same login as without proxy. Nginx configuration is mostly taken from minio docs. Can any one see the reason or know what can I check to login?

Part of docker compose:

  nginx:
    image: nginx:alpine
    restart: unless-stopped
    ports:
      - ${FORWARD_NGINX_HTTP_PORT:-80}:80
      - ${FORWARD_NGINX_HTTPS_PORT:-443}:443
    volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d
      - ./nginx/ssl:/etc/nginx/ssl
    networks:
      - default
  minio:
    image: minio/minio
    command: minio server /data/minio --console-address ":9001"
    restart: unless-stopped
    healthcheck:
      test: [ "CMD", "mc", "ready", "local" ]
      interval: 30s
      timeout: 5s
      retries: 3
    expose:
      - 9000
      - 9001
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minio}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-password}
      MINIO_SERVER_URL: https://s3.localhost
      MINIO_BROWSER_REDIRECT_URL: https://s3.localhost/minio/ui
    ports:
      - ${FORWARD_MINIO_API_PORT:-9000}:9000
      - ${FORWARD_MINIO_CONSOLE_PORT:-9001}:9001
    volumes:
      - minio-data:/data
    networks:
      - default

Part of nginx configuration:

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name s3.localhost;

    ssl_certificate /etc/nginx/ssl/localhost.crt;
    ssl_certificate_key /etc/nginx/ssl/localhost.key;

    ignore_invalid_headers off;
    client_max_body_size 0;
    proxy_buffering off;
    proxy_request_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://minio:9000;
    }

    location /minio/ui {
        rewrite ^/minio/ui/(.*) /$1 break;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;

        real_ip_header X-Real-IP;

        proxy_connect_timeout 300;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Origin '';

        chunked_transfer_encoding off;

        proxy_pass http://minio:9001;
    }
}

Certs generated with mkcert.

Answer 1

The Reason of the Problem

Minio Console will connect to Minio Server's storage for accessing its management data (e.g., user data at login time). Setting MINIO_SERVER_URL to https://s3.localhost will let the console access the server (nginx proxy) via HTTPS.

The problem is related to the SSL certificate (seems to be self-signed) which you used for nginx to server HTTPS traffic.

ssl_certificate /etc/nginx/ssl/localhost.crt;
ssl_certificate_key /etc/nginx/ssl/localhost.key;

Due to the server's certificate not being trusted by the console, access data from the server will fail from TLS verification, and thus result in 401 invalid login from the console.

The Solutions

Solution 1: Using Internal Address to Access Minio Server

This is what this answer does, but it modified your nginx configuration, which is not necessary.

As the console and server are collocated in the same container (this is true if you just launched minio with console on single node), the console will automatically pick the container's non-loopback address to access the storage, if the MINIO_SERVER_URL is not explicitly set.

You can also see the address from the startup logs of the container, like the following line (the first address will be used by the console to contact the server):

API: http://192.168.106.5:9000  http://127.0.0.1:9000

Also note that MINIO_SERVER_URL is only used by the console, setting it or not will not affect our access to the storage service.

Solution 2: Let the Console/Server Trust Your Own Certificate

When you need load balancing via a reverse proxy in multiple nodes deployment scenario, you need to set MINIO_SERVER_URL with the domain corresponding to the address of the reverse proxy. In case HTTPS/TLS is enabled for the reverse proxy with self-signed certificate, you should add the certificate (or the CA certificate that signed it) to the MinIO/System's certificate trust store.

Two methods are discussed in the following sections, where the first method is recommended.

2.1 Adding certificate to MinIO's trust store

The instruction can be found at: Self-signed, Internal, Private Certificates, and Public CAs with Intermediate Certificates.

You can also see the GitHub issue Minio console gives "Invalid login" if internal CA cert is not imported when using reverse proxying inside docker and the discussion on it.

Mount your certificate (or the CA certificate that signed it) for nginx virtual server to the home of the MinIO service user,

services:
  minio:
    command: ["server", "/data/minio", "--console-address", ":9001"]
    volumes:
      - ./path/to/myCA.crt:${MINIO_HOME}/.minio/certs/CAs/myCA.crt

or some other customized path with --certs-dir /opt/minio/certs,

services:
  minio:
    command: 
      - server
      - /data/minio
      - --console-address
      - ":9001"
      - --certs-dir
      - /opt/minio/certs
    volumes:
      - ./path/to/myCA.crt:/opt/minio/certs/CAs/myCA.crt

2.2 Adding certificate to system's trust store

Note: This method should work in theory but not verified by me, as I prefer serving Minio with plain HTTP behind reverse proxy for a test environment. This method is mainly for your information.

To add extra certificates to container's system trust store, we need to mount certificate files to the container or extend the image by adding your certificate file to the trusted certificate store of the image.

1. adding the localhost.crt or the root-ca.crt (if you sign localhost.crt with your own CA) to your local system hosting docker.

   sudo apt install ca-certificates  # or yum, dnf,... depends on your OS.
   sudo cp localhost.crt /usr/local/share/ca-certificates/
   sudo update-ca-certificates  # for Debian-based OS
   sudo update-ca-trust extract # for RHEL-based OS
   

2. If not rebuild the image, directly bind mount the /etc/ssl/certs/ca-certificates.crt of the local system to the container in your compose file.

   services:
     minio:
       image: minio/minio
       volumes:
         - minio-data:/data
         - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt
   

   similar to (3), if mount the single file is not enough, mount more directories.

3. alternative to (2), extend the official image:

   # custom.Dockerfile
   FROM minio/minio
   COPY /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
   

   This is learned from the official image. If not enough, copy the whole directory of /etc/ssl/certs/, /usr/share/ca-certificates, and /usr/local/share/ca-certificates to the image.

   docker build -f custom.Dockerfile -t minio:${TAG:-latest}
   

Note: the minio/minio image which is based on RedHat's ubi9:micro, does not include a package manager. So, we cannot install ca-certificates package. This is why we need to copy the CA certificate files from the host system.