Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What is context variable in Airflow operators

Tags:

python

airflow

I'm trying to understand what is this variable called context in Airflow operators. as example:

def execute(self, **context**).

Where it comes from? where can I set it? when and how can I use it inside my function? Another question is What is *context and **context? I saw few examples that uses this variable like this:

def execute(self, *context) / def execute(self, **context). 

What is the difference and when should I use *context and **context

like image 993
Alan Mil Avatar asked Dec 06 '25 21:12

Alan Mil


1 Answers

Documentation on the nature of context is pretty sparse at the moment. (There is a long discussion in the Github repo about "making the concept less nebulous".)

In a few places in the documentation it's referred to as a "context dictionary" or even an "execution context dictionary", but never really spelled out what that is.

Apparently, the Templates Reference is considered to be documentation for the context dictionary, although that's not actually mentioned on the page.

As is often the case with Airflow, a look at the source code is sometimes our best bet. One contributor has pointed to the following code block to describe the context dict:

        return {
            'conf': conf,
            'dag': task.dag,
            'dag_run': dag_run,
            'ds': ds,
            'ds_nodash': ds_nodash,
            'execution_date': pendulum.instance(self.execution_date),
            'inlets': task.inlets,
            'macros': macros,
            'next_ds': next_ds,
            'next_ds_nodash': next_ds_nodash,
            'next_execution_date': next_execution_date,
            'outlets': task.outlets,
            'params': params,
            'prev_ds': prev_ds,
            'prev_ds_nodash': prev_ds_nodash,
            'prev_execution_date': prev_execution_date,
            'prev_execution_date_success': lazy_object_proxy.Proxy(
                lambda: self.get_previous_execution_date(state=State.SUCCESS)
            ),
            'prev_start_date_success': lazy_object_proxy.Proxy(
                lambda: self.get_previous_start_date(state=State.SUCCESS)
            ),
            'run_id': run_id,
            'task': task,
            'task_instance': self,
            'task_instance_key_str': ti_key_str,
            'test_mode': self.test_mode,
            'ti': self,
            'tomorrow_ds': tomorrow_ds,
            'tomorrow_ds_nodash': tomorrow_ds_nodash,
            'ts': ts,
            'ts_nodash': ts_nodash,
            'ts_nodash_with_tz': ts_nodash_with_tz,
            'var': {
                'json': VariableJsonAccessor(),
                'value': VariableAccessor(),
            },
            'yesterday_ds': yesterday_ds,
            'yesterday_ds_nodash': yesterday_ds_nodash,
        }

Update

I've since found a mention of the context dictionary in the documentation! If anyone can find any more, I'll be happy to link them here.

When running your callable, Airflow will pass a set of keyword arguments that can be used in your function. This set of kwargs correspond exactly to the context variables you can use in your Jinja templates.

like image 188
LondonRob Avatar answered Dec 08 '25 09:12

LondonRob



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!