keras prediction gets stuck when deployed using uwsgi in a flask app

Question

I have a keras model that works perfectly in unit tests and in local flask app (flask run). However, the moment I launch the flask app in uwsgi, it gets stuck from the second request on, killing the entire app. Is this because uwsgi spawns multiple processes? How do I get around this problem? Thanks.

Answer 1

I am having the same issue, this may not be ideal work around. But I believe this is an Keras issue running in multiple processes.

I am running uWSGI with lazy-apps = true.

uwsgi --http 0.0.0.0:5000 --wsgi-file your_flask_server.py --callable app --processes 2 --threads 2 --stats 127.0.0.1:9191 --lazy-apps

Note: By lazy loading apps this will consume more memory like double the memory since it is loading the whole app again in each thread/process

here are some use full links: Similar Issue, Similar Issue

Example deployment guide

Answer 2

I had a similar issue: in a flask app inside a docker environment, we would find that the neural network would hang on predict after the first prediction. to get around this I created a neural network class which had

def __init__(self):
    self.session = tf.Session()
    self.graph = tf.get_default_graph()
    self.model = self.__load_model()
    with self.graph.as_default():
        with self.session.as_default():
            logging.info("neural network initialised")

The last 3 lines seemed to properly initialise the graph and the session which for some reason wasn't happening at the correct place. My predict function was then simply:

def predict(self, x):
    with self.graph.as_default():
        with self.session.as_default():
            y = self.model.predict(x)
    return y

This seems to have stopped the hanging (I don't know why I need a session and a graph, but I added both while I was debugging this and now I'm to afraid to remove either)