Access Forbidden while accessing log in airflow with CeleryExecutor

Question

I'm trying to deploy Airflow on kubernetes (on Azure Kubernetes Service) with the celery Executor. However, once a task is done, I get the following error while trying to access its logs:

*** Log file does not exist: /opt/airflow/logs/maintenance/clean_events/2021-08-23T14:46:18.953030+00:00/1.log
*** Fetching from: http://airflow-worker-0.airflow-worker.airflow.svc.cluster.local:8793/log/maintenance/clean_events/2021-08-23T14:46:18.953030+00:00/1.log
*** Failed to fetch log file from worker. 403 Client Error: FORBIDDEN for url: http://airflow-worker-0.airflow-worker.airflow.svc.cluster.local:8793/log/maintenance/clean_events/2021-08-23T14:46:18.953030+00:00/1.log
For more information check: https://httpstatuses.com/403

my charts.yaml is pretty simple

---
airflow:
   image:
      repository: myrepo.azurecr.io/maintenance-scripts
      tag: latest
      pullPolicy: Always
      pullSecret: "secret"
   executor: CeleryExecutor
   config:
      AIRFLOW__CORE__LOAD_EXAMPLES: "True"
      AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "False"
   users:
    - username: admin
      password: password
      role: Admin
      email: [email protected]
      firstName: admin
      lastName: admin
rbac:
   create: true
serviceAccount:
    create: true
#postgresql:
#    enabled: true
workers:
    enabled: true
redis:
    enabled: true
flower:
    enabled: false
global:
  postgresql: {
      storageClass: managed
  }
persistence:
  fixPermissions: true
  storageClassName: managed

I have not been able to fix this, and it seems to be the most basic conf you can use on airflow, anyone knows where this could come from ?

Thanks a lot

Answer 1

You need to have the same webserver secret configured for both webserver and workers: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key

It's been recently fixed as potential security vulnerability - now you need to know the secret key to be able to retrieve logs (it was unauthenticated before).