How to execute tasks in different server using Python Celery?

Question

Let's say I have three servers A, B & C. Server C will have the celery task code, and I need to execute them from servers A & B.

From the celery documentation, I see that there's a task.py file which is run as a celery worker

from celery import Celery

app = Celery('tasks', broker='pyamqp://guest@localhost//')

@app.task
def add(x, y):
    return x + y

And then we have another python file (let's say client.py) which calls these tasks.

from tasks import add

add.delay(4, 4)

Here I can see that the client.py file is dependent on the tasks.py file as it's importing the task from the tasks.py file. If we are to run these two files in separate servers, we need to decouple them and somehow call the tasks without having to import the code. I am not able to figure out how to achieve that. So, how can it be done?

Answer 1

In general you do not do that. You deploy the same code (containing tasks) to both producers (clients) and consumers (workers). However, Celery is a cool piece of software and it allows you to actually schedule task without the need to distribute the code on the producer side. For that you have to use the send_task() method. You have to configure producer with same parameters as your workers (same broker naturally, same serialization) and you must know the calling task name and its parameters in order to schedule its execution correctly.