Why I can't start celery with a path to an app

Question

I have a tasks.py file I want to feed to Celery so it can run in the background. When I cdinto the directory, everything works properly

celery -A tasks worker --loglevel=info

However I'd like to run this in a bash script, so I have to specify the full path to the app. But it doesn't seem to work, even though the path is correct

(tf_tasks) arthur@debian:~$ ls /data/tera_1/partage/Biomass_ML/
LICENSE  model.hdf5  __pycache__  server.py  tasks.py
(tf_tasks) arthur@debian:~$ celery -A "/data/tera_1/partage/Biomass_ML/tasks" worker --loglevel=info
Traceback (most recent call last):
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/bin/celery", line 10, in <module>
    sys.exit(main())
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/site-packages/celery/__main__.py", line 16, in main
    _main()
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/site-packages/celery/bin/celery.py", line 322, in main
    cmd.execute_from_commandline(argv)
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/site-packages/celery/bin/base.py", line 273, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/site-packages/celery/bin/base.py", line 479, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/site-packages/celery/bin/base.py", line 501, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/site-packages/celery/app/utils.py", line 359, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/site-packages/celery/bin/base.py", line 504, in symbol_by_name
    return imports.symbol_by_name(name, imp=imp)
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/site-packages/celery/utils/imports.py", line 104, in import_from_cwd
    return imp(module, package=package)
  File "/data/tera_1/partage/anaconda/envs/tf_tasks/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named '/data/tera_1/partage/Biomass_ML/tasks'

I tried with double quotes, simple quotes, specifying --app= instead of -A, specifying the file extension but nothing works.

Answer 1

When starting celery worker with -A(--app), we should pass the python path to the app instance. For example

From /data/tera_1/partage/Biomass_ML/ directory, we can run

celery -A tasks worker --loglevel=info

When running it from a shell script, you can cd into the directory and run it

cd /data/tera_1/partage/Biomass_ML/
celery -A tasks worker --loglevel=info

If you have to run from other directories, add it to your python path

export PYTHONPATH="/data/tera_1/partage/Biomass_ML/:$PYTHONPATH"
celery -A tasks worker --loglevel=info

Alternatively, if your entire path is a valid python path with modules, then we can run it with

celery -A data.tera_1.partage.Biomass_ML.tasks worker --loglevel=info