valueError when using multi_gpu_model in keras

Question

I am using google cloud VM with 4 Tesla K80 GPU's.

I am running a keras model using multi_gpu_model with gpus=4(since i have 4 gpu's). But, i am getting the following error

ValueError: To call multi_gpu_model with gpus=4, we expect the following devices to be available: ['/cpu:0', '/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3']. However this machine only has: ['/cpu:0', '/xla_cpu:0', '/xla_gpu:0', '/gpu:0']. Try reducing gpus.

I can see that there are only two gpu's here namely '/xla_gpu:0', '/gpu:0'. so, i tried with gpus = 2 and again got the following error

ValueError: To call multi_gpu_model with gpus=2, we expect the following devices to be available: ['/cpu:0', '/gpu:0', '/gpu:1']. However this machine only has: ['/cpu:0', '/xla_cpu:0', '/xla_gpu:0', '/gpu:0']. Try reducing gpus.

can anyone help me out with the error. Thanks!

Answer 1

It looks like Keras only sees one of the GPUs.

Make sure that all 4 GPUs are accessible, you can use device_lib with TensorFlow.

from tensorflow.python.client import device_lib

def get_available_gpus():
    local_device_protos = device_lib.list_local_devices()
    return [x.name for x in local_device_protos if x.device_type == 'GPU']

You might need to manually install or update GPU drivers on your instance. Consult here.