Promtail basic auth using kubernetes secret in helm values.yaml

Question

I am using the promtail helm chart to connect to a Loki server running on a different stack. I have Loki behind an Nginx ingress secured with basic auth.

I can't find any documentation on this, and it's very possible it's just my admittedly limited understanding of helm.

I'm simply trying to use basic auth to connect to the Loki instance while using a Kubernetes secret instead of plaintext credentials in the helm values

This works perfect:

  snippets:
    extraClientConfigs: |
      basic_auth:
        username: myusername
        password: mypassword

I created a secret like this:

kubectl create secret generic loki-credentials -n monitoring --from-literal=password="mypassword" --from-literal=username="myusername"

and now I want to use that in the values.yaml file.

This is what I got so far:

extraEnv:
  - name: LOKI_USERNAME
    valueFrom:
      secretKeyRef:
        name: loki-credentials
        key: username
  - name: LOKI_PASSWORD
    valueFrom:
      secretKeyRef:
        name: loki-credentials
        key: password

extraArgs:
  - -client.external-labels=stack=development
  - -config.expand-env

config:
  serverPort: 3101
  lokiAddress: myurl
  snippets:
    extraClientConfigs: |
      basic_auth:
        username: ${LOKI_USERNAME}
        password: ${LOKI_PASSWORD}

I just get a 401 response.

 Chart version: 3.1.0
 Promtail version: 2.1.0

EDIT

Here is my ingress yaml:

controller:
  replicaCount: 1

  config:
    force-ssl-redirect: "true"
    use-forwarded-headers: "true"

  service:
    targetPorts:
      http: http
      https: http
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-2:123456:certificate/123456"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

    enableHttp: false
    enableHttps: true
    type: LoadBalancer
    loadBalancerSourceRanges:
      - "0.0.0.0/0"

   ## Name of the ingress class to route through this controller
  ingressClass: nginx-external

...and my loki values.yaml

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: "nginx-external"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: loki-ingress-auth
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
  hosts:
    - host: loki.mydomain.com
      paths: 
        - "/"
  tls: []

config:
  auth_enabled: false
  ingester:
    chunk_idle_period: 3m
    chunk_block_size: 262144
    chunk_retain_period: 1m
    max_transfer_retries: 3
    lifecycler:
      ring:
        kvstore:
          store: inmemory
        replication_factor: 1
  
  schema_config:
    configs:
    - from: 2021-03-06
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: loki_index_
        period: 24h
        
  server:
    http_listen_port: 3100

  storage_config:
    aws:
      bucketnames: my-bucket-name
      region: us-west-2
      s3forcepathstyle: true

    boltdb_shipper:
      active_index_directory: /data/loki/boltdb-shipper-active
      cache_location: /data/loki/boltdb-shipper-cache
      shared_store: s3

  chunk_store_config:
    max_look_back_period: 0s
  
  table_manager:
    retention_deletes_enabled: false
    retention_period: 0s

  compactor:
    working_directory: /data/loki/boltdb-shipper-compactor
    shared_store: aws

replicas: 1

podAnnotations:
  iam.amazonaws.com/role: "arn:aws:iam::123456:role/my-loki-role"

resources:
  limits:
    cpu: 500m
    memory: 2G
  requests:
    cpu: 250m
    memory: 1G

# The values to set in the PodDisruptionBudget spec
# If not set then a PodDisruptionBudget will not be created
podDisruptionBudget:
  minAvailable: 1

My logs from the nginx pod that Loki is sitting behind:

2021/03/09 04:23:44 [error] 37#37: *925 user "myusername": password mismatch, client: xxx.xx.xxx.xxx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.mydomain.com"
2021/03/09 04:23:44 [error] 37#37: *921 user "myusername": password mismatch, client: xxx.xx.xxx.xxx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.mydomain.com"
xx.xxx.xxx.xx - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 401 172 "-" "promtail/2.1.0" 326 0.000 [monitoring-loki-3100] [] - - - - 63294b16fe010a8c9ec1d4684f0472f5
xxx.xx.xxx.xxx: - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 204 0 "-" "promtail/2.1.0" 2744 0.003 [monitoring-loki-3100] [] xxx.xx.xxx.xxx:3100 0 0.004 204 029e0a9d1ee6242cad8b9a6d2ee50940
2021/03/09 04:23:44 [error] 37#37: *925 user "myusername": password mismatch, client: xx.xxx.xxx.xx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.mydomain.com"
xxx.xx.xxx.xxx - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 401 172 "-" "promtail/2.1.0" 325 0.000 [monitoring-loki-3100] [] - - - - b75a2cfcf6c62b81953dd4fb26f1a844
xxx.xx.xxx.xxx - myusername [09/Mar/2021:04:23:44 +0000] "POST /loki/api/v1/push HTTP/1.1" 204 0 "-" "promtail/2.1.0" 1513 0.014 [monitoring-loki-3100] [] xxx.xx.xxx.xxx:3100 0 0.016 204 0049965a49877cb5d336ac6ec869feb4
2021/03/09 04:23:45 [error] 36#36: *941 user "myusername": password mismatch, client: xxx.xx.xxx.xxx, server: loki.mydomain.com, request: "POST /loki/api/v1/push HTTP/1.1", host: "loki.mydomain.com"
xxx.xx.xxx.xxx - myusername [09/Mar/2021:04:23:45 +0000] "POST /loki/api/v1/push HTTP/1.1" 401 172 "-" "promtail/2.1.0" 326 0.000 [monitoring-loki-3100] [] - - - - e5954bd055db5b3e9bd3227f57651847

Answer 1

I came across the same issue and solved it with the original approach

• Separate secret (via sealed-secret)
• Env-variables.

Versions

Chart-Version	App-Version
6.15.1	2.8.4

My values.yaml

config:
  clients:
    - url: http://loki-gateway/loki/api/v1/push
      basic_auth:
        password: "${LOKI_PASSWORD}"
        username: "${LOKI_USERNAME}"

extraArgs:
  - -config.expand-env=true

extraEnv:
  - name: LOKI_USERNAME
    valueFrom:
      secretKeyRef:
        name: promtail-credentials
        key: username
  - name: LOKI_PASSWORD
    valueFrom:
      secretKeyRef:
        name: promtail-credentials
        key: password

extraObjects:
  - apiVersion: bitnami.com/v1alpha1
    kind: SealedSecret
    metadata:
      annotations:
        sealedsecrets.bitnami.com/cluster-wide: "true"
      creationTimestamp: null
      name: promtail-credentials
      namespace: loki
    spec:
      encryptedData:
        username: <SEALED-VALUE-FOR-USERNAME-HERE>
        password: <SEALED-VALUE-FOR-PASSWORD-HERE>
      template:
        data: null
        metadata:
          annotations:
            sealedsecrets.bitnami.com/cluster-wide: "true"
          creationTimestamp: null
          name: promtail-credentials
          namespace: loki'