Error "maximum recursion depth exceeded" when starting docker compose

Question

I am trying to spin up the official docker-compose stack for airflow v2.

My only changes have to do with the fact that

a) I have removed the postgres service since I am using an RDS

b) the sqlaclhemy / celery endpoints have been adjusted accordingly.

Here is the file

version: '3'
x-airflow-common:
  &airflow-common
  image: apache/airflow:2.0.1
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: "postgresql+psycopg2://rds_username:rds_pass@rds_address/postgres"
    AIRFLOW__CELERY__RESULT_BACKEND: "db+postgresql://rds_username:rds_pass@wrds_address/postgres"
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
  volumes:
    - ./dags:/opt/airflow/dags:rw
    - ./logs:/opt/airflow/logs:rw
    - ./plugins:/opt/airflow/plugins:rw
  user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
  depends_on:
    redis:
      condition: service_healthy

  redis:
    image: redis:latest
    ports:
      - 6379:6379
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 30s
      retries: 50
    restart: always

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 8080:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    restart: always

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    restart: always

  airflow-init:
    <<: *airflow-common
    command: version
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}

  flower:
    <<: *airflow-common
    command: celery flower
    ports:
      - 5555:5555
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

However I am getting this very crypting - extremely non-helpful error when trying to run any docker-compose command

$ docker-compose ps
Traceback (most recent call last):
  File "bin/docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 67, in main
  File "compose/cli/main.py", line 123, in perform_command
  File "compose/cli/command.py", line 69, in project_from_options
  File "compose/cli/command.py", line 127, in get_project
  File "compose/config/config.py", line 389, in load
  File "compose/config/config.py", line 389, in <listcomp>
  File "compose/config/config.py", line 589, in process_config_file
  File "compose/config/validation.py", line 467, in validate_against_config_schema
  File "compose/config/validation.py", line 461, in keys_to_str
  File "compose/config/validation.py", line 461, in keys_to_str
  File "compose/config/validation.py", line 461, in keys_to_str
  [Previous line repeated 986 more times]
  File "compose/config/validation.py", line 458, in keys_to_str
RecursionError: maximum recursion depth exceeded while calling a Python object
[1318740] Failed to execute script docker-compose

I have tried to use the rds connection strings both with and without quotes.

Does anyone have any idea what is going on?

Answer 1

I ended up here with the same error, the problem is the missing services key declaration, in my case I deleted it accidentally. Just add it back:

...
      condition: service_healthy

services:
  redis:
    image: redis:latest
...