How can I pass the yes flag to backfill in Google Composer?

Question

I am trying to do a backfill in Google Composer using a gcloud composer command but I am struggling to pass a -y or --yes for a corresponding --reset_dagruns argument.

The response I get is airflow: error: unrecognized arguments: -y.

Command:

gcloud composer environments run my_env --project my_project --location us-east1 backfill -- --reset_dagruns -y -s 2020-01-01 -e 2020-01-31 my_dag

How can I supply this argument?

Answer 1

Before answering your specific question I have a suggestion:
If your DAG (my_dag) already specifies the start & end date over which you need backfill, then just clear the status of runs you need backfilled/redone. The scheduler then picks them up again.

For your question specifically:
The AirFlow CLI documentation has unfortunately not proved directly usable with Google Cloud Composer, this is what works on Composer version 1.10.2

gcloud --project=my_project composer environments run my_env --location us-east1 backfill -- my_dag --reset_dagruns -s 2020-01-01 -e 2020-01-31

Note that this will hold on to your CLI session and will stop if you ctrl-c out, if your backfill is long you are better off defining a start date on the dag and setting catchup=True