How to read airflow variables into kubernetes pod instance

Question

I am trying to read airflow variables into my ETL job to populate variables in the curation script. I am using the KubernetesPodOperator. How do I access the metadata database from my k8's pod?

Error I am getting from airflow: ERROR - Unable to retrieve variable from secrets backend (MetastoreBackend). Checking subsequent secrets backend.

This is what I have in main.py for outputting into the console log. I have a key in airflow variables named "AIRFLOW_URL".

  from airflow.models import Variable
    AIRFLOW_VAR_AIRFLOW_URL = Variable.get("AIRFLOW_URL")

    logger.info("AIRFLOW_VAR_AIRFLOW_URL: %s", AIRFLOW_VAR_AIRFLOW_URL)

Can anyone point me in the right direction?

Answer 1

Your DAG can pass them as environment variables to your Pod, using a template (e.g. KubernetesPodOperator(... env_vars={"MY_VAR": "{{var.value.my_var}}"}, ...)).