Multiprocessing with flask sqlalchemy - psycopg2.DatabaseError: error with status PGRES_TUPLES_OK and no message from the libpq

Question

am getting below exception while trying to use multiprocessing with flask sqlalchemy.

sqlalchemy.exc.ResourceClosedError: This result object does not return rows. It has been closed automatically.
[12/Aug/2019 18:09:52] "GET /api/resources HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/SQLAlchemy-1.3.6-py3.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/SQLAlchemy-1.3.6-py3.7-linux-x86_64.egg/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
psycopg2.DatabaseError: error with status PGRES_TUPLES_OK and no message from the libpq

Without multiprocessing the code works perfect, but when i add the multiprocessing as below, am running into this issue.

worker = multiprocessing.Process(target=<target_method_which_has_business_logic_with_DB>, args=(data,), name='PROCESS_ID', daemon=False)
worker.start()
return Response("Request Accepted", status=202)

I see an answer to similar question in SO (https://stackoverflow.com/a/33331954/8085047), which suggests to use engine.dispose(), but in my case am using db.session directly, not creating the engine and scope manually.

Please help to resolve the issue. Thanks!

Answer 1

I had the same issue. Following Sam's link helped me solve it.

Before I had (not working):

from multiprocessing import Pool
with Pool() as pool:
    pool.map(f, [arg1, arg2, ...])

This works for me:

from multiprocessing import get_context
with get_context("spawn").Pool() as pool:
    pool.map(f, [arg1, arg2, ...])