How to decode airflow table dag_run column conf value

Question

How do i decode the value of airflow table dag_run column conf value so I can read the value of conf column from the table

Thanks

Answer 1

I recently ran into the same thing, and it turns out the conf column is a binary Pickle string.

I found the answer in the Airflow internal module documentation https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/models/dagrun.html#DagRun.conf

import pickle

# function to return SQLAlchemy Engine connected to Airflow
engine = get_airflow_db_engine()

# query and convert conf
query = """
    select dag_id, execution_date, state, run_id, conf
    from dag_run
    limit 25
"""
df = pd.read_sql(query, engine)
df['conf'] = df['conf'].apply(lambda x: pickle.loads(x))