Can pre trained Keras model on GPU be used for predictions on a CPU only PC?

Question

I am currently learning Keras. My question is if I train a model on a high end GPU like RTX Titan, can I export the model, load it into a new program on another low end machine and then do prediction using only the CPU. Is that possible?

In theory, I think that's how machine learning works. The model is trained on a high end GPU and once exported it can be loaded on any machine regardless of whether it has a GPU or not and can be used to do predictions.

If not, then are Machine Learning models deployed on high end servers with many GPUs?

Answer 1

Yes in keras it will work seamlessly. Keras using tensorflow back will check if the GPUs are available and if so the model will be trained on GPU.

Similarly while inference when you load the model, if no GPU is available it will use the CPU.

Experiment using google colab

Lets start a google collab using "GPU" runtime

import numpy as np
from keras.models import Sequential
from keras.layers import Dense
import tensorflow as tf 

tf.compat.v1.debugging.set_log_device_placement(True)
print(tf.config.list_physical_devices('GPU'))

model = Sequential()
model.add(Dense(1024, input_dim=8, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])

X = np.random.randn(10,8) 
y = np.random.randn(10) 

model.fit(X, y, epochs=2)
model.save("model.h5")

Output

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
Epoch 1/2
1/1 [==============================] - 0s 1ms/step - loss: 0.6570 - accuracy: 0.0000e+00
Epoch 2/2
1/1 [==============================] - 0s 983us/step - loss: 0.6242 - accuracy: 0.0000e+00
<tensorflow.python.keras.callbacks.History at 0x7fcad09366a0>

So the model is trained on the available GPU in this case. You can see that it ocupies GPU using command !nvidia-smi. We have saved the model as model.h5. Lets download it and make a local copy

Now lets change the runtime of the colab to "CPU". lets upload our model.h5 to the collab and make a prediction.

import numpy as np
from keras.models import Sequential
from keras.layers import Dense
import tensorflow as tf 


from keras.models import load_model
tf.compat.v1.debugging.set_log_device_placement(True)
print(tf.config.list_physical_devices('GPU'))

model = load_model('model.h5')
model.predict(X)

Oputput:

[]
array([[0.4464949 ],
       [0.43229908],
       [0.49823508],
       [0.4367126 ],
       [0.47648385],
       [0.48096564],
       [0.47863394],
       [0.5031184 ],
       [0.45698297],
       [0.45885688]], dtype=float32)

As you can see as expected there is no GPU available and the model was loaded and prediction ran on CPU.

Keras its seamless. But in pytorch it is different where we have to manually move the model from GPU to CPU.