Airflow - BashOperator: Getting "Dag runs are deadlocked for DAG:.." error

Question

Really liking the Airflow workflow scheduler, but stumped on an error running a simple DAG: "{jobs.py:538} ERROR - Dag runs are deadlocked for DAG: TEST_SCHEDULER_DAG".

This is a new airflow install (v1.7.1.3) and I've been able to run other scheduled dag files just fine. My environment is Linux (ubuntu 16.04), python 2.7.12 (anaconda), postgresql 9.5.5, and using LocalExecutor.

The DAG I'm getting the deadlock error on is:

from airflow import DAG
from airflow.operators import BashOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'owner.name',
    'depends_on_past': True,
    'start_date': datetime(2016, 11, 30, 8, 0, 0),
    'retries': 0,
    'retry_delay': timedelta(seconds=60),
}

tst_dag = DAG(dag_id='TEST_SCHEDULER_DAG',
              default_args=default_args,
              schedule_interval='10 * * * *')

t1 = BashOperator(
    task_id='task_1',
    bash_command='sleep 10',
    dag=tst_dag)

t2 = BashOperator(
    task_id='task_2',
    bash_command='sleep 10',                  
    dag=tst_dag)

t2.set_upstream(t1)

Again, the first execution runs fine, but all the subsequent executions (DagRun) show as 'failed' and I see the 'deadlock' error on the console.

Thank you!

Answer 1

@Gergely's response helped me. I was trying to run airflow backfill for a date that was past the end_date specified on the DAG. Once I changed the end_date on the DAG object to include the date I was backfilling, it worked.

Answer 2

Try to delete dagrun table's row which have state 'running' as value in the database.

And then execute the command airflow backfil <dag_id> -s <start_date> -e <end_date> --reset-dagruns