py-postgresql multithread problems

Question

I discovered that under heavy load my pyramid web app throws py-postgresql exceptions like postgresql.exceptions.ProtocolError. Some search revealed, that py-postgresql is not thread-safe and one connection can not be used by several threads concurrently.

I tried to make some sort of pooling mechanism, but I still get ProtocolErrors :(

What am I doing wrong?

First I create number of connection objects:

    for x in range(num_db_connections):
        self.pool.append(Connection(conn_string,x))

Each object in pool contains db_lock = threading.Lock() and a connection to database self.conn = postgresql.open( conn_string )

Then I try to acquire lock on a connection and do some work with it. This code can be executed by many threads concurrently, but i think no two threads can run work on one connection concurrently because of lock.

    time_start = time.time()
    while time.time() - time_start < self.max_db_lock_wait_time:
        for conn in self.pool:
            acquired = conn.db_lock.acquire(False)
            if acquired:
                try:
                        lst = conn.work()
                finally:
                    conn.db_lock.release()
                return lst
        time.sleep(0.05)
    raise Exception('Could not get connection lock in time')

Maybe there is flaw in my code, or I misunderstood the nature of "thread unsafety" of py-postgresql? Please, help me!

Answer 1

Are you sure that you are not using cursor objects outside your lock?

Just a suggestion: instead of using time.sleep() and "trying" to lock, use a Queue to pop/push connection objects from/into the pool. That is already thread safe, and it has a timeout parameter. Much more effective. Especially if you have many threads and just a few connections. (Those tiny sleep instructions add up, when you need to run 100 000 queries. All of them will increase your response times.)