GridSearchCV and Google colab: n_jobs=-1 does not work

Question

The problem is the following: when running these lines of code

grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1)
grid_result = grid.fit(X, y)

on Google Colab, it returns the error

PicklingError: Could not pickle the task to send it to the workers.

However, if I set n_jobs = 1, it perfectly works.

How can I use multiple processors on Google Colab?

Answer 1

There are some possible problems for your issue:

• One problem may be your param_grid=param_grid, sometimes a change of grid search parameters solve the issue.

• Another potential issue may be with your model and the parameters that are used there. See this issue as an example.

• You can try any other numbers for example n_jobs=4 instead of n_jobs=-1. Also, you can use:

n_cpus = multiprocessing.cpu_count()

to get the number of CPUs and then

n_jobs=n_cpus 

or

n_jobs=n_cpus - 1