Airflow worker - Connection broken: IncompleteRead(0 bytes read)

Question

Using Airflow worker and webserver/scheduler as a Docker images running on Kubernetes Engine on EC2

We have a task which has KubernetesPodOperator which is resource intensive and runs every 15min.

Got these error as email in airflow-worker

Try 2 out of 3
Exception:
('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
Log: Link
Host: airflow-worker-deployment-123456789
Log file: /usr/local/airflow/logs/DAG_NAME/TASK_NAME/2019-03-14T10:50:00+00:00.log
Mark success: Link

Any idea what It can be ?

Answer 1

so, better late than never

it is because of known bug in KubernetesPodOperator. to avoid this behavior you have to set operators get_logs parameter to False. default value is True.

details here https://issues.apache.org/jira/browse/AIRFLOW-3534 https://issues.apache.org/jira/browse/AIRFLOW-5571