Directly running a task on a dedicated dask worker

Question

A simple code-snippet is as follows: comment followed by ### is important..

from dask.distributed import Client

### this code-piece will get executed on a dask worker.
def task_to_perform():
    print("task in progress.")
    ## do something here..
    print("task is over.!")

### whereas the below code will run on client side, 
### assume on a different node than worker
client= Client("127.0.01:8786")
future = client.submit(task_to_perform)
print("task results::", future.result())

so the control flow for the execution would be like this: the dask-client will submit the task to dask-scheduler and the scheduler will take call on which worker it has to submit to the task based on the available workers.

but do we have any mechanism in dask through which I can submit my task on a dedicated/particular worker bypassing the dask-scheduler ?

Answer 1

You can choose a particular worker to run a task by using the workers= keyword to submit

client.submit(func, *args, workers='tcp://worker-address:port')

You can get the address by looking at the logs of your worker, or calling client.scheduler_info(). More information about the workers= keyword is available here: http://distributed.readthedocs.io/en/latest/locality.html#user-control

Note that this still routes the task through the scheduler. The client just tells the scheduler where to schedule the task.