Using Dask from script

Question

Is it possible to run dask from a python script?

In interactive session I can just write

from dask.distributed import Client
client = Client()

as described in all tutorials. If I write these lines however in a script.py file and execute it python script.py, it immediately crashes.

I found another option I found, is to use MPI:

# script.py
from dask_mpi import initialize
initialize()

from dask.distributed import Client
client = Client()  # Connect this local process to remote workers

And then run the script with mpirun -n 4 python script.py. This doesn't crash, however if you print the client

print(client)
# <Client: scheduler='tcp://137.250.37.84:35145' processes=0 cores=0> 

you see that no cores are used, accordingly scripts run forever without doing anything.

How do I set my scripts up correctly?

Answer 1

If you want to create processes from within a Python script you need to protect that code in an if __name__ == "__main__": block

from dask.distributed import Client

if __name__ == "__main__":
    client = Client()

If you want to use dask-mpi then you need to run it with mpirun or mpiexec with a suitable number of processes.