It is possible to run celery on gpu?

Question

The question is - It is possible to run celery on gpu?

Currently in my project I have settings like below:

celery -A projectname worker -l error --concurrency=8 --autoscale=16,8 
--max-tasks-per-child=1 --prefetch-multiplier=1 --without-gossip --without-mingle --without-heartbeat

This is django project.

One single task take around 12 s to execute (large insert to postgresql).

What I want to achive is multiply workers as many as possible.

Answer 1

No. Celery offers no way to run anything on GPU. However, nothing prevents you to use Keras, TensorFlow, or PyTorch in your Celery tasks (as a matter of fact I see many questions here about these projects and Celery).

Since your tasks are mostly IO-bound, you can simply overutilise. If machine has N cores you may set concurrency to N*2.

Another possibility is to change concurrency type to eventlet/gevent.

Answer 2

you can specify the number of workers like this...

celery -A projectname --workers=3