How to predict in multiple and simultanous Keras classifier sessions in Django?

Question

I know similar questions have been asked before and I have read all of them but none solved my problem.

I have a Django project in which I am using CNNSequenceClassifier from sequence_classifiers which is a Keras model. The model files have been fit before and saved to particular destinations from which I again load them and predict what I want.

clf = CNNSequenceClassifier(epochs=2)

When I load the model I do this, due to the suggestions I found in searches, which is using the global model before loading the model, and then the other two lines:

global clf
clf = pickle.load(open(modelfilenameandpath, "rb"))
global graph
graph = tf.get_default_graph()

and before predicting I use graph.as_default()

with graph.as_default():
    probs = clf.predict_proba(the_new_vecs)
K.clear_session()

I put K.clear_session() because I predict in a for loop and sometimes the next item of the for loop's prediction gets mixed up with the last one and raises tensorflow errors. But clearly K.clear_session() clears the session and makes it easy for the new item's prediction to work fine.

The problem is in my views.py I have two functions which trigger prediction. And sometimes I need to use both simultanously. But since the probject is using Tensorflow Backend, only one session is defined and the predictions of these two functions get mixed up together. K.clear_session() does not help here because it was only for that particular session and it does not expect new stuff coming in the same session.

I really do not know how to make the functions understand whenever they start loading the model and/or want to predict it, start a new and independent session so that nothing gets mixed up. I have seen codes like:

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

but I don't know where to put the probs = clf.predict_proba(the_new_vecs). It seems all of them require the action you want to be done in the session to be passed as an argument to sess.run() which does not work for me or at least I do not know how to make it work. I want the predict to be done in a new session and after that the values of probs are really important. Also global_variables_initializer() seems to change the value of all my variables.

I have also tried K.set_session() or K.get_session() but did not work.

To sum up, I think the reason I am stuck is that I do not know how to use sessions for my purpose. I do not know where to use what code?!

Please help!

Answer 1

My understanding of tensorflow graphs and sessions is:

A tensorflow graph hosts operations, placeholders and Variables. A tensorflow graph lives inside a tensorflow session (that's why to save a trained model using tensorflow.train.Saver, you need something like Saver.save(sess, the graph)).

Here is a simple example to help you make sense of the relation between a keras model and a tensorflow graph:

import tensorflow as tf 
from keras.layers import Input, Dense
from keras.models import Model 

tf.reset_default_graph()
graph_1, graph_2 = tf.Graph(), tf.Graph()
with graph_1.as_default():
    x_in = Input(shape=(1, ), name='in_graph1')
    pred = Dense(5, name='dense1_graph1')(x_in)
    pred = Dense(1, name='dense2_graph1')(pred)
    model = Model(input=x_in, output=pred)
with graph_2.as_default():
    x_in = Input(shape=(1, ), name='in_graph2')
    pred = Dense(10, name='dense1_graph2')(x_in)
    pred = Dense(1, name='dense2_graph2')(pred)
    model = Model(input=x_in, output=pred)


with tf.Session() as sess:
    default_ops = sess.graph.get_operations()

with graph_1.as_default():
    with tf.Session() as sess:
        one_ops = sess.graph.get_operations()

with graph_2.as_default():
    with tf.Session() as sess:      
        two_ops = sess.graph.get_operations()

As you can see by running the code, default_ops is an empty list which means there is no operation in the default graph. one_ops is a list of operations of the first keras model and two_ops is a list of operations of the second keras model.

So, by using with graph.as_default(), a keras model can be exclusively embedded in a tensorflow graph.

With this in mind, it becomes easy to load several keras models in a single script. I think the example script below will address your confusions:

import numpy as np
import tensorflow as tf 
from keras.layers import Input, Dense
from keras.models import Model
from Keras.models import model_from_json

tf.reset_default_graph()
x = np.linspace(1, 4, 4)
y = np.random.rand(4)

models = {}
graph_1, graph_2 = tf.Graph(), tf.Graph()

# graph_1 
with graph_1.as_default():
    x_in = Input(shape=(1, ), name='in_graph1')
    pred = Dense(5, name='dense1_graph1')(x_in)
    pred = Dense(1, name='dense2_graph1')(pred)
    model = Model(input=x_in, output=pred)
    models['graph_1'] = model
# graph_2
with graph_2.as_default():
    x_in = Input(shape=(1, ), name='in_graph2')
    pred = Dense(10, name='dense1_graph2')(x_in)
    pred = Dense(1, name='dense2_graph2')(pred)
    model = Model(input=x_in, output=pred)
    models['graph_2'] = model

# save the two models 
with tf.Session(graph=graph_1) as sess:
    with open("model_1.json", "w") as source:
        source.write(models['graph_1'].to_json())
    models['graph_1'].save_weights("weights_1.h5")
with tf.Session(graph=graph_2) as sess:
    with open("model_2.json", "w") as source:
        source.write(models['graph_2'].to_json())
    models['graph_2'].save_weights("weights_2.h5")

####################################################
# play with the model 
pred_one, pred_one_reloaded = [], []
pred_two, pred_two_reloaded = [], []
for _ in range(10):
    print(_)
    if _ % 2 == 0:

        with graph_1.as_default():
            with tf.Session() as sess:
                sess.run(tf.global_variables_initializer())
                pred_one.append(models['graph_1'].predict(x).ravel())
        with tf.Session() as sess:
            with open("model_1.json", "r") as f:
                model = model_from_json(f.read())
            model.load_weights("weights_1.h5")
            pred_one_reloaded.append(model.predict(x).ravel())

    else:
        with graph_2.as_default():
            with tf.Session() as sess:
                sess.run(tf.global_variables_initializer())
                pred_two.append(models['graph_2'].predict(x).ravel())
        with tf.Session() as sess:
            with open("model_2.json", "r") as f:
                model = model_from_json(f.read())
            model.load_weights("weights_2.h5")
            pred_two_reloaded.append(model.predict(x).ravel())

pred_one = np.array(pred_one)
pred_one_reloaded = np.array(pred_one_reloaded)
pred_two = np.array(pred_two)
pred_two_reloaded = np.array(pred_two_reloaded)

print(pred_one)
print(pred_one_reloaded)
print(pred_two)
print(pred_two_reloaded)